Import large quantities of epubs to calibre

[Sanchezct89]

Greetings to all friends.

I am new to the experience and management of Calibre as my online particle library.

I have a large collection of epubs built over many years (more than 25,000 epubs).

Is there a way to add them to the library quickly? Because when I try to add the folder where I have them, it locates them but it takes an average of 10-15 seconds per book and of course, an average of 13 seconds for the 25,000 epubs, we are talking about 325,000 seconds = 5416.6 hours = 90.2 hours = 3.76 days. Almost 4 days to add the library! Do you know any way to add it faster?

Thank you all very much and happy reading!

[BetterRed]

You could try the command line calibredb add ==>> https://manual.calibre-ebook.com/gen...bredb.html#add

But I don't know that it would be substantially faster.

BR

[theducks]

And Faster is not always better (or safer ).
Do chunks (~25 or less), This stores everything at the end of the task. A crashed task has no guarantee what the state will be.

fix those issues (normalize Authors, titles as you go. Quality check Plugin ) ) as you find. There will be issues

It took me a long time to document my 900+ Paperbacks into Calibre (some did not have ISBN they were so old )

(I move the Processed Source folder to an 'Archive' drive, which also frees up drive space for the Library )

[un_pogaz]

A realy quick way is to use the "auto-add from folder" feature.

Go to Preferences>Adding book>Automating adding and set a folder to target (need a restart)
After that, Calibre will be automaticly read and add any eBook in your target folder to the current Calibre library, then delete the original file from the folder. If your books is splited on several folder, move then to the target folder (your OS will be move file faster than Calibre read/add/delete the eBook).
Once you've added all your books, don't forget to disable the "auto-add" by clear the target folder field.

This solution are 2 major flaw: The book are added in mass without checking, so that can product a lot of wrong metadata that need to be correted AND the original book are permanently deleted by Calibre after being added in the library, so be undo available.
So that not the safer way, but probably the most faster.
The user interface of Calibre is so practical, convenient and efficient, that if your are well prepared, with a good metodologie, is can be a realy good solution.

[Sarmat89]

Adding EPUBs is so slow due to excessive disk I/O. Putting the TEMP directory on a RAM disk should help.

[DNSB]

When I compared adding 1000 ePubs from a spinning rust drive compared to adding them from a NVMe PCIe 4.0 drive, the speed difference was negligible.

[kjdavies]

Originally Posted by BetterRed

You could try the command line calibredb add ==>> https://manual.calibre-ebook.com/gen...bredb.html#add

But I don't know that it would be substantially faster.

BR

It is unlikely to be be faster. Each time you call calibredb, it loads the library. The GUI loads the library once and processes each file in turn.

I am a little startled at the long load times, though. I have quite-large libraries (some number in the hundreds of thousands -- I use it to manage publisher and game dev resources) and even then it doesn't take so long... even running off an external drive.

edit: That assumes you're loading one file at a time. I don't do that and had forgotten you can have calibre add by directory (even recursively). That would not have the same overhead per file, it should behave closer to the GUI upload.

[kjdavies]

Originally Posted by Sanchezct89

Greetings to all friends.

I am new to the experience and management of Calibre as my online particle library.

I have a large collection of epubs built over many years (more than 25,000 epubs).

Is there a way to add them to the library quickly? Because when I try to add the folder where I have them, it locates them but it takes an average of 10-15 seconds per book and of course, an average of 13 seconds for the 25,000 epubs, we are talking about 325,000 seconds = 5416.6 hours = 90.2 hours = 3.76 days. Almost 4 days to add the library! Do you know any way to add it faster?

Thank you all very much and happy reading!

Are all the files in a single folder? You don't say what OS you're using, but Windows does not like having more than about a thousand files in a directory. I've built systems around breaking that up for performance reasons.

What happens if you try loading from a folder with, say, a hundred files in it?

If that makes a difference and you're storing everything in one folder, perhaps break it up into folders of 500-1000 files instead.

[kjdavies]

Originally Posted by un_pogaz

A realy quick way is to use the "auto-add from folder" feature.

Go to Preferences>Adding book>Automating adding and set a folder to target (need a restart)
After that, Calibre will be automaticly read and add any eBook in your target folder to the current Calibre library, then delete the original file from the folder. If your books is splited on several folder, move then to the target folder (your OS will be move file faster than Calibre read/add/delete the eBook).
Once you've added all your books, don't forget to disable the "auto-add" by clear the target folder field.

This solution are 2 major flaw: The book are added in mass without checking, so that can product a lot of wrong metadata that need to be correted AND the original book are permanently deleted by Calibre after being added in the library, so be undo available.
So that not the safer way, but probably the most faster.
The user interface of Calibre is so practical, convenient and efficient, that if your are well prepared, with a good metodologie, is can be a realy good solution.

A third limitation of this approach is that calibre loads the files into whatever library is open. Unless things have changed since I last did this, you can't have different libraries monitoring different folders.

Also, I found it can affect system performance, if there are a lot of files to be processed. I ended up writing a script that would copy files to be loaded into the intake folder (and move the originals to a 'these have been done' folder) to help limit this. And to make it easier to interrupt (stop the manager script so it no longer moves files, and the autoload would run out of files to upload).