Performance hit due to Calibre control of file/folder names

[Strange Quark]

Hello,

I am a book lover who spent a lot of money on paper books but who now has a large e-book file collection. For many years (almost 15!) I was collecting my e-book collection and have been looking for a good e-book management SW. But before I even tried to find one I asked myself what are the essential features I would insist on.

After a lot of thinking I concluded that due to lack of dedicated ISBN number filed in metadata in official standards for all the popular e-book file formats (PDF, CHM, DJV(U), EPUB) I am unfortunately forced to use file name as a ISBN storage. If popular e-book file formats did have dedicated ISBN number in their official specifications then ISBN numbers for my e-books would be inserted into book files once I find these numbers either manually for each book or automatically/semi-automatically with help from a SW, and then then these numbers would be extracted easily with any e-book management software like Calibre. In that case I wouldn't care if Calibre or any other e-book management software insisted on its own naming scheme because I would know that ISBN numbers are safe and saved inside dedicated ISBN fields inside files themselves. But unfortunately this is not the case so I am forced to use file name as the most reliable ISBN storage. All my e-book files have the following name structure: (ISBN 978xxxxxxxx).ext and they are all in one folder named Book Library.

Although I am fully aware that any e-book management SW has a connection between ISBN numbers and file names for all files in its library due to the fact that I spent MONTHS on manual and semi-automatic check on reliable ISBN number detection for my e-book files I insist on having these file names "cemented". Although many e-book management applications (including Calibre) allow export of book collection in any file name format I cannot trust any SW (especially not a buggy one) to control the connection between file names and ISBN numbers for my large e-book collection since its database might be corrupted, and especially when I spent a lot of time to find and check these ISBN numbers for each file separately.

Many years ago I found a good e-book management SW (lets call it "Unknown") that allows me to keep my e-book file names and the folders they are in whatever I like. It also has a tremendously efficient and reliable semi-automatic ISBN number extraction from file content (from text). It also stores all its info in one large database file which is completely separated from my e-book files what makes archiving both management database file and e-book files very easy. But then I realized it has extremely poor search capabilities. What good is a e-book management SW if you cannot search your collection??? It is also not an open source but commercial application with extremely slow feature acceptance rate. You have to beg them to add a good search mechanism with no result.

Then, some years ago, I found Calibre. Its open source characteristics and extremely good search capabilities delighted me. But then I realized it insists on its own file/folder naming scheme which collided with my principle above.

I also found that there is a performance penalty in Calibre for using operating system's file system for controlling file/folder names. Unlike application "Unknown" which I currently use which relies on the fact that e-book files it imports in its database already exist somewhere and it does not care where and how they are named, Calibre has to create files and folders separately and since file system in OS has to check for collisions it is slow process for a large number of books being imported, especially when the existing collection is already large. When I import 100 books in Unknown application it takes a few seconds. But when I import them in Calibre it takes 20 minutes even when there are no actual files to import (copy) but only ISBN numbers! Unlike Calibre, application Unknown does not have to check with file subsystem in OS are there any collisions and if the files/folders could be created. Download of metadata from Internet is also a lot quicker in Unknown than in Calibre.

For some time I even used both Unknown and Calibre simultaneously! Calibre didn't have any files but only ISBN numbers imported, metadata downloaded, and then I used it for its good search capabilities.



Now, I am wondering is there a possibility to get rid of Calibre's file/folder naming reliance with a new plug-in? That plug-in would basically allow Calibre to import e-book files with their existing names and keep them in their existing folders. Is existing Calibre's architecture flexible enough to allow such a plug-in? Does Calibre's plug-in API allows such a plug-in? If Calibre's existing architecture does not allow such a plug-in today is there a chance to accommodate it in the future? I am afraid the answer is no but I want to check with Calibre experts anyway.

Thank you for your consideration and sorry for a long post.

Reader from Croatia

[jhowell]

I am sure others will have things to say about this, but here are a few points.

Calibre managing its own e-book files is a fundamental design decision that isn't going to change. There is no way that a plugin can affect this. If you insist on having the book files be under your control then calibre isn't going to work well for you.

One way to work around it is to have the files that you import just be pointers to the actual files, something like m3u playlists for music files. But by doing this you will lose most of the capabilities of calibre. It needs the actual book content for most functions.

If calibre takes 20 minutes to import 100 books then something is wrong. It works far faster for me.

[pdurrant]

Originally Posted by Strange Quark

Now, I am wondering is there a possibility to get rid of Calibre's file/folder naming reliance with a new plug-in?

None at all.

But (IMO) what you should do is add a custom column to calibre's database for ISBN, and tweak the import parameters to auto-fill that from your file names as they're imported. (I'm sure there's a way to do that, although I don't know it off-hand.)

What do you do for ebooks that do not have an ISBN?

[JSWolf]

100 books in 20 minutes means you have a rather slow computer. Either that or your computer has something running that slows it down.

[dwig]

Originally Posted by JSWolf

100 books in 20 minutes means you have a rather slow computer. Either that or your computer has something running that slows it down.

+1 - Any anti-Malware software that actively scans files is a prime suspect.

When calibre imports a new book by a new author, it creates two folders and 3 new files (copy of the book, an extracted or generated cover image, and a small database file). It also adds a new entry into is primary database file.

With a reasonably decent computer the disk processes should be reasonably quick, but not instantaneous. The computer's disk performance is usually the primary limiting factor. 100 books in 20 minutes is 5 books per minute, or 12 seconds per book, and is in line with what my Windows 10 tablet achieves with large ePubs when the library is stored on the microSD expansion card. My desktop machine (i7 processor and 7200rpm SATA HD) is considerably faster.

Calibre's file system and database design is not optimal for fast import, but is excellent in terms of robustness and speed of the library manager. Searches are fast as they use the database info and pre-generated cover image rather than reading filenames and extracting covers every time you search or scroll the display. The extra small db file saved with the book provides a backup of data to help with repairing a damaged library should disk problems occur.

[theducks]

Moderator Notice
This is not a help withdevelopment question. Please read the stickies at the top of each forum before ever posting. Moved.

I agree with the others. It is your computer (as configured).

Are you trying to keep your library on a USB stick/SDcard? Those are slow write devices compared to even a (s)low end HD

Exclude your library folder from ACTIVE malware/AV scans (it was probably scanned wherever it IS before the add)

[JSWolf]

Not all AV will slow down the computer that much. But some like Norton or AVG will slow down the computer too much.

[BetterRed]

Background content indexers can also effect bulk add performance. If you use any, make sure they're set to run as a low priority task - preferably such that the indexer only operates when system is otherwise idle.

FWIW I run MS Defender, Malwarebytes Pro, the Windows Indexer, and X1's Search indexer. My calibre libraries are not excluded from Defender or MWB, and they're explicitly included in the scope of the Windows and X1 indexers.

Right now the libraries are on a SATA III drive, and the indexes are on an NVMe SSD drive. But until recently both were on the same SATA II drive. I add 10-20 books a day, some of which are video clips. The AV and Indexing services have never had any noticeable effect on calibre's performance on either rig.

BR