Adding Books to Empty Library

Tanjamuse · 10-10-2018, 01:46 PM

Hi.

Is there any way to speed up the process of adding new books to an empty library?

Unfortunately I only have epubs and opf-files and no database.

Would it be quicker if I create zip or rar files?

They are in folders by authors with multiple epubs & opf-files in each folder.

Thanks

ilovejedd · 10-10-2018, 03:45 PM

If the files were in Calibre folder structure, you can do a restore database. Alas, that doesn't seem to be the case.

Unfortunately, I think this task is pretty disk bound and I don't believe zip/rar would help. Perhaps don't use the computer and disable antivirus during import?

theducks · 10-10-2018, 05:47 PM

I see no real reason to have 'real time scans' on a Library folder.

I agree with ilovejedd, your pretty much IO bound adding.
1)get rid of the A/V scan
2)Have the source on a different drive spindle, to reduce head position latency
Is the OPF from a Calibre Library (one opf + title per folder)? That might help.

Normal Add, is ~5+ per minute, on basic desktop hardware (no SSD or I7 )

ilovejedd · 10-10-2018, 05:59 PM

Quote:

Originally Posted by theducks

Normal Add, is ~5+ per minute, on basic desktop hardware (no SSD or I7 )

Yikes, that bad? I've been spoiled by SSDs.

Tanjamuse · 10-11-2018, 01:47 AM

Quote:

Originally Posted by theducks

I see no real reason to have 'real time scans' on a Library folder.

I agree with ilovejedd, your pretty much IO bound adding.
1)get rid of the A/V scan
2)Have the source on a different drive spindle, to reduce head position latency
Is the OPF from a Calibre Library (one opf + title per folder)? That might help.

Normal Add, is ~5+ per minute, on basic desktop hardware (no SSD or I7 )

I'll see if I can remove the A/V scan from those folders and move the stories to another hdd.

The OPF are from Calibre already, but it's one folder per author and in that folder the epubs and opf's are all together, to avoid the limit on characters in windows 10.

If I were to split everything into separate folders, I guess it would have to be manual and considering how many there are a very large task.

mbovenka · 10-11-2018, 03:45 AM

Quote:

Originally Posted by ilovejedd

Yikes, that bad? I've been spoiled by SSDs.

No, it's not *that* bad. With Calibre itself and the source on an SSD and the library on a dedicated 5400 RPM drive, I get about 25 adds/min (older mobile Core i7).

Library size also plays a role, as Calibre does basic dup checking when adding.

Tanjamuse · 10-11-2018, 03:47 AM

Could I avoid the duplicates check? Would that make it go faster?

theducks · 10-11-2018, 05:04 AM

Quote:

Originally Posted by Tanjamuse

Could I avoid the duplicates check? Would that make it go faster?

dup check (and the resolution) is an option in Preferences: Adding Books

I think some of my speed issue is this 'refurbished Core Duo PC' came with 8G of slow end RAM (according to UserBenchmark) , DDR3-1100 instead of 1600 (listed by Crucial). I put in a WD Black drive thinking the drive was the slow part, but saw no obvious improvement.

Tanjamuse · 10-11-2018, 05:06 AM

It's an empty library I add the books to.

stumped · 10-11-2018, 05:57 AM

just let it run overnight, there is no need to sit and watch it .

that is a old, slow CPU by modern standards.

are we talking hundreds of books, or thousands, or tens of thousands ???

Tanjamuse · 10-11-2018, 06:05 AM

I think the biggest library is 60.000 ...

ilovejedd · 10-11-2018, 09:50 AM

Quote:

Originally Posted by mbovenka

No, it's not *that* bad. With Calibre itself and the source on an SSD and the library on a dedicated 5400 RPM drive, I get about 25 adds/min (older mobile Core i7).

Library size also plays a role, as Calibre does basic dup checking when adding.

I meant when using solely HDD (particularly if there's just one for both source and destination). I reckon having the source on a different drive from the target is already a big help.

Mind, on a single "slow" SSD (Samsung 840 Basic) and i5-3450S, I get 1 book/second and that's probably only because I call calibredb add --library-path per book which I expect is far less efficient than a normal batch import. No auto-merge/dupe check during import. I just use the Find Duplicates plugin after.

Quote:

Originally Posted by theducks

I think some of my speed issue is this 'refurbished Core Duo PC' came with 8G of slow end RAM (according to UserBenchmark) , DDR3-1100 instead of 1600 (listed by Crucial). I put in a WD Black drive thinking the drive was the slow part, but saw no obvious improvement.

A "Black" HDD isn't really notably faster than a "Green" one in this case.

Quote:

Originally Posted by Tanjamuse

I think the biggest library is 60.000 ...

Yeah, just leave it running overnight. Maybe run them in batches of 5-10K?

Tanjamuse · 10-13-2018, 03:52 AM

Is it possible to avoid the duplicate check when adding books to empty library?

I don't have a checkmark in the duplicates check but when I add new books to an empty library it asks what to do about a bunch of duplicates that aren't really duplicates.

It only checks the title and not both title and author or any other columns.

Could it be possible to either override this duplicate check or make sure that it compares more than one column to figure out if it's a duplicate.

ilovejedd · 10-13-2018, 05:19 PM

Quote:

Originally Posted by Tanjamuse

Is it possible to avoid the duplicate check when adding books to empty library?

It only checks the title and not both title and author or any other columns.

Could it be possible to either override this duplicate check or make sure that it compares more than one column to figure out if it's a duplicate.

I just set the ignore via command line. Not sure how to do the same in GUI.

I use the Find Duplicates plugin to check for duplicates after import. I've got plenty of fanfics that have changed author pseudonym or title so the url identifier (e.g. https://archiveofourown.org/works/1234567) is the most reliable method of catching duplicates.

I was actually quite curious about performance so I ran some import tests using part of my AO3 fanfic library.

SSD used is a 500GB Samsung 840 with planar TLC NAND. It's 5-6 years old and 90% full so probably quite slow by modern SSD standards as well as due to normal performance degradation.

HDD used is a brand new, empty 1TB 7200RPM Seagate Barracuda (found it in my box of spare parts).

Flash drive used is a 128GB Samsung Bar USB 3.1 (connected to USB 3.0 port).

Code:

calibredb add --duplicates --recurse --library-path "X:\Calibre Portable\TestLibrary"
"X:\ebooks\import"


Import Stats
               mm:ss.00  MB/min  books/min
SSD to SSD     13:54.10    178      355
SSD to HDD     16:55.52    147      291
HDD to HDD     20:19.20    122      243
Flash to HDD   17:37.79    141      280


Import Structure:
\Fandom\Authors\Authors - Title (id).ext

2.422 GB, 979 folders, 24,650 files

4,930 "unique" books (based on checksum)
* each book has epub, mobi, txt, opf & cover

2,411 unique titles (based on url identifier)


Options:
  -d, --duplicates      Add books to database even if they already exist.
                        Comparison is done based on book titles.

  -r, --recurse         Process directories recursively

ilovejedd · 10-13-2018, 05:43 PM

P.S.

System runs a Core i5-3450S 2.80 GHz (Ivy Bridge) and 16GB RAM. Operating system (Windows 7 64-bit), pagefile, etc. are all on a separate SSD.

Import process was definitely I/O bound. The only time the import managed to max out one core (25% CPU utilization) was during the SSD to SSD import. On all imports to HDD, CPU usage was typically 5-15%.

10-10-2018, 01:46 PM	#1
Tanjamuse Wizard Posts: 1,327 Karma: 5306 Join Date: Jan 2014 Device: none	Adding Books to Empty Library Hi. Is there any way to speed up the process of adding new books to an empty library? Unfortunately I only have epubs and opf-files and no database. Would it be quicker if I create zip or rar files? They are in folders by authors with multiple epubs & opf-files in each folder. Thanks

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Empty file window when adding books...?	patrik	Calibre	2	06-19-2018 12:46 PM
Empty folders when adding books	LucyOne	Calibre	0	07-15-2017 06:27 AM
Adding Books to Library	MikeHackney	Calibre	11	10-16-2015 10:47 AM
Adding books already in library directory but not library database	sfguy5874	Library Management	1	07-07-2012 04:33 AM
Adding empty books	vbussell	Calibre	4	12-10-2009 05:42 PM

10-10-2018, 03:45 PM	#2
ilovejedd hopeless n00b Posts: 5,110 Karma: 19597086 Join Date: Jan 2009 Location: in the middle of nowhere Device: PW4, PW3, Libra H2O, iPad 10.5, iPad 11, iPad 12.9	If the files were in Calibre folder structure, you can do a restore database. Alas, that doesn't seem to be the case. Unfortunately, I think this task is pretty disk bound and I don't believe zip/rar would help. Perhaps don't use the computer and disable antivirus during import?

10-10-2018, 05:47 PM	#3
theducks Well trained by Cats Posts: 31,048 Karma: 60358908 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	I see no real reason to have 'real time scans' on a Library folder. I agree with ilovejedd, your pretty much IO bound adding. 1)get rid of the A/V scan 2)Have the source on a different drive spindle, to reduce head position latency Is the OPF from a Calibre Library (one opf + title per folder)? That might help. Normal Add, is ~5+ per minute, on basic desktop hardware (no SSD or I7 )

10-11-2018, 03:47 AM	#7
Tanjamuse Wizard Posts: 1,327 Karma: 5306 Join Date: Jan 2014 Device: none	Could I avoid the duplicates check? Would that make it go faster?

10-11-2018, 05:06 AM	#9
Tanjamuse Wizard Posts: 1,327 Karma: 5306 Join Date: Jan 2014 Device: none	It's an empty library I add the books to.

10-11-2018, 05:57 AM	#10
stumped Wizard Posts: 3,305 Karma: 10259306 Join Date: May 2016 Device: kobo forma, Kobo Libra, Huawei media Tab, fire HD10, PW3 HDX8.9,	just let it run overnight, there is no need to sit and watch it . that is a old, slow CPU by modern standards. are we talking hundreds of books, or thousands, or tens of thousands ???

10-11-2018, 06:05 AM	#11
Tanjamuse Wizard Posts: 1,327 Karma: 5306 Join Date: Jan 2014 Device: none	I think the biggest library is 60.000 ...

10-13-2018, 03:52 AM	#13
Tanjamuse Wizard Posts: 1,327 Karma: 5306 Join Date: Jan 2014 Device: none	Is it possible to avoid the duplicate check when adding books to empty library? I don't have a checkmark in the duplicates check but when I add new books to an empty library it asks what to do about a bunch of duplicates that aren't really duplicates. It only checks the title and not both title and author or any other columns. Could it be possible to either override this duplicate check or make sure that it compares more than one column to figure out if it's a duplicate.

10-13-2018, 05:43 PM	#15
ilovejedd hopeless n00b Posts: 5,110 Karma: 19597086 Join Date: Jan 2009 Location: in the middle of nowhere Device: PW4, PW3, Libra H2O, iPad 10.5, iPad 11, iPad 12.9	P.S. System runs a Core i5-3450S 2.80 GHz (Ivy Bridge) and 16GB RAM. Operating system (Windows 7 64-bit), pagefile, etc. are all on a separate SSD. Import process was definitely I/O bound. The only time the import managed to max out one core (25% CPU utilization) was during the SSD to SSD import. On all imports to HDD, CPU usage was typically 5-15%.

Advert

Advert