MobileRead Forums - View Single Post - Squeezing as much performance as possible out (container)

Adoby · 12-12-2018, 06:07 PM

Most (all?) of your operations worked only on/in the DATABASE. And your set of books were most likely small enough to fit in normal RAM disk caches. This is one thing that is great with calibre. Lazy update of ebook files. Most operations on metadata never reach the actual ebook files. They only reach the database.

If you update/convert/send/save changed metadata to ebooks then you'll benefit much more from putting tmp in RAM. Then book files will have to be unzipped, metadata updated and then zipped again. That takes a lot of processing power and a lot of disk read/writes.

I just use tmpfs for tmp. And have my calibre libraries, databases and ebooks on a big SSD. I used to have the database on a small local SSD and the book files on a NAS. Worked fine. Now I use the NAS only for automatic versioned rsync backups of the calibre libraries.

Plenty of RAM benefits database access almost as much as putting the database in RAM. Not using RAM to put the database in means more RAM for disk caches that makes for faster database access.

I would suggest that you may benefit from partitioning the calibre library into several smaller libraries. Then caches will be utilized better while updating.

Also calibre use an early simple form of container. Almost all dependencies are fully contained inside the calibre install folder. So there are no problems with clashing python version.

12-12-2018, 06:07 PM	#2
Adoby Handy Elephant Posts: 1,737 Karma: 26785684 Join Date: Dec 2009 Location: Southern Sweden, far out in the quiet woods Device: Samsung Galaxy Tab S8 Ultra	Most (all?) of your operations worked only on/in the DATABASE. And your set of books were most likely small enough to fit in normal RAM disk caches. This is one thing that is great with calibre. Lazy update of ebook files. Most operations on metadata never reach the actual ebook files. They only reach the database. If you update/convert/send/save changed metadata to ebooks then you'll benefit much more from putting tmp in RAM. Then book files will have to be unzipped, metadata updated and then zipped again. That takes a lot of processing power and a lot of disk read/writes. I just use tmpfs for tmp. And have my calibre libraries, databases and ebooks on a big SSD. I used to have the database on a small local SSD and the book files on a NAS. Worked fine. Now I use the NAS only for automatic versioned rsync backups of the calibre libraries. Plenty of RAM benefits database access almost as much as putting the database in RAM. Not using RAM to put the database in means more RAM for disk caches that makes for faster database access. I would suggest that you may benefit from partitioning the calibre library into several smaller libraries. Then caches will be utilized better while updating. Also calibre use an early simple form of container. Almost all dependencies are fully contained inside the calibre install folder. So there are no problems with clashing python version. Last edited by Adoby; 12-12-2018 at 06:28 PM.