Library Merging

Krazykiwi · 11-21-2015, 10:26 AM

Not the usual question, I know how to merge two libraries, but I'm wondering if anyone can give me a guesstimate how long this will take, so I can figure out if I should set this going overnight, or wait until I have a whole weekend.

I have two libraries, one consisting of 69k "books" and one consisting of 27k. I'd guesstimate there's about 20% duplication (that is, about 20% of the books in the smaller library are in the larger already).

I did a trial run of about 1000 books, and... well I got sick of waiting, but I think it took most of the night.

Is that the kind of speed I can expect (in which case, I will need to do this piecemeal over a few days so as to not interrupt the work too much). Or does that indicate something's dodgy with my databases and I need to do some cleanup?

Will it be faster to try to pull the dupes out first with the duplication plugin, in order to make the merge run faster, or would the duplicates

As an aside, Yes I know these are enormous numbers for Calibre libraries, no they're not pirated commercial books, this is not my personal Calibre library it's a text corpus for a research project. BTW Calibre's been doing a fabulous job managing this mountain of data. Ironically the much maligned "black box" nature, once I got it through everyone's heads, has been a godsend, being able to search effectively and dump out subsets of the texts to disk for data mining purposes works beautifully, and we can safely use destructive tools to chop 'em up and do stuff(TM) to them.

kovidgoyal · 11-21-2015, 10:45 AM

I suggest doing it in chunks and make sure you hide the Tag browser while you are doing this, that should help.

As for expected times, it is rather hard to say, it depends on a lot of variables. As a simple datapoint, copying 400 books to a 40,000 book library with tag browser hidden and both libraries on the same SSD and /tmp on a RAM disk, took ~10mins on my desktop.

Krazykiwi · 11-21-2015, 11:09 AM

Thanks Kovid!

Turning off the tag browser helped ridiculously much on a test, I don't know why I didn't think of that.

It's on a RAID 5 array, so write performance pretty much bites compared to your SSDs, but I can do a /tmp ramdisk (I didn't think of that either) and move it in chunks over a few nights.

Thanks again, and I do love this program so much!

yonkyunior · 12-11-2015, 12:56 PM

try Find Duplicates Plugin,
or using Copy to LIbrary (choose delete/not after copy)

Krazykiwi · 12-11-2015, 08:13 PM

Thanks, but, not necessary, this was completed a couple of weeks ago now. I know how to merge libraries, as I said, and I also know how to find and handle duplicates, fwiw, although my question got cut off.

For the benefit of anyone else interested/future searches, I was asking if it was faster to strip them out first using the Find Duplicates plugin to find library duplicates, bearing in mind that would take some time to do and need some hand management vs. to just let them merge during the copy.

Turned out when testing, it really didn't slow the copy down noticeably enough to be worth spending the time dealing with it for this many items.

technomom · 12-20-2015, 01:34 AM

I'm curious - how would one merge two libraries?

Thank you!

eschwartz · 12-20-2015, 01:37 AM

Select books ==> Right-click ==> Copy to library

technomom · 12-20-2015, 02:30 AM

Thank you!

Rellwood · 12-20-2015, 03:43 PM

I have copied libraries so much that I worry that the books will lose their data by copying a copy of a copy......

Copy to library is the greatest feature, but causes me the most headaches when I end up with a mass of books and metadata that are inconsistent.

However....the suggestion to hide the tags browser is a new one to me, and will definitely come in handy.

ms233 · 01-05-2016, 12:49 PM

Quote:

Originally Posted by kovidgoyal

I suggest doing it in chunks and make sure you hide the Tag browser while you are doing this, that should help.

As for expected times, it is rather hard to say, it depends on a lot of variables. As a simple datapoint, copying 400 books to a 40,000 book library with tag browser hidden and both libraries on the same SSD and /tmp on a RAM disk, took ~10mins on my desktop.

Wow, that's incredibly fast. I have an SSD arriving tomorrow that will hopefully help, but I've found that the speed of the copy function falls through the floor at about 50 books. It clips along at about a book per second until around 40-50 books have been processed, and then it falls off dramatically to about one book every 30 seconds or so. This is copying into a library of about 160,000 books. It doesn't seem to matter how many books are being copied. I've been looking at the _doit() method in class Worker to see if I can spot the issue but no luck so far (sadly I'm no programmer).

kovidgoyal · 01-05-2016, 01:00 PM

You can always stick print statements in the code to see where it is slowing down for you. http://manual.calibre-ebook.com/develop.html

11-21-2015, 10:26 AM	#1
Krazykiwi Zealot Posts: 137 Karma: 2156958 Join Date: Jan 2013 Device: Too many random androids to list	Library Merging Not the usual question, I know how to merge two libraries, but I'm wondering if anyone can give me a guesstimate how long this will take, so I can figure out if I should set this going overnight, or wait until I have a whole weekend. I have two libraries, one consisting of 69k "books" and one consisting of 27k. I'd guesstimate there's about 20% duplication (that is, about 20% of the books in the smaller library are in the larger already). I did a trial run of about 1000 books, and... well I got sick of waiting, but I think it took most of the night. Is that the kind of speed I can expect (in which case, I will need to do this piecemeal over a few days so as to not interrupt the work too much). Or does that indicate something's dodgy with my databases and I need to do some cleanup? Will it be faster to try to pull the dupes out first with the duplication plugin, in order to make the merge run faster, or would the duplicates As an aside, Yes I know these are enormous numbers for Calibre libraries, no they're not pirated commercial books, this is not my personal Calibre library it's a text corpus for a research project. BTW Calibre's been doing a fabulous job managing this mountain of data. Ironically the much maligned "black box" nature, once I got it through everyone's heads, has been a godsend, being able to search effectively and dump out subsets of the texts to disk for data mining purposes works beautifully, and we can safely use destructive tools to chop 'em up and do stuff(TM) to them.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Advice on merging a LARGE library	toomuchreading	Library Management	4	03-28-2014 11:40 AM
Merging 2 Library's	logs	Calibre	8	05-25-2011 09:37 AM
Merging ebooks	jmjacobs	Library Management	7	01-29-2011 07:01 AM
Help with PDF merging	E.M.	iRex	2	03-21-2009 03:19 PM
help: merging	scarlett_yan	iRex	5	09-22-2007 04:18 PM

11-21-2015, 10:45 AM	#2
kovidgoyal creator of calibre Posts: 45,149 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	I suggest doing it in chunks and make sure you hide the Tag browser while you are doing this, that should help. As for expected times, it is rather hard to say, it depends on a lot of variables. As a simple datapoint, copying 400 books to a 40,000 book library with tag browser hidden and both libraries on the same SSD and /tmp on a RAM disk, took ~10mins on my desktop.

11-21-2015, 11:09 AM	#3
Krazykiwi Zealot Posts: 137 Karma: 2156958 Join Date: Jan 2013 Device: Too many random androids to list	Thanks Kovid! Turning off the tag browser helped ridiculously much on a test, I don't know why I didn't think of that. It's on a RAID 5 array, so write performance pretty much bites compared to your SSDs, but I can do a /tmp ramdisk (I didn't think of that either) and move it in chunks over a few nights. Thanks again, and I do love this program so much!

12-11-2015, 12:56 PM	#4
yonkyunior Cultivator Posts: 94 Karma: 216 Join Date: Feb 2015 Device: PRST2	try Find Duplicates Plugin, or using Copy to LIbrary (choose delete/not after copy)

12-11-2015, 08:13 PM	#5
Krazykiwi Zealot Posts: 137 Karma: 2156958 Join Date: Jan 2013 Device: Too many random androids to list	Thanks, but, not necessary, this was completed a couple of weeks ago now. I know how to merge libraries, as I said, and I also know how to find and handle duplicates, fwiw, although my question got cut off. For the benefit of anyone else interested/future searches, I was asking if it was faster to strip them out first using the Find Duplicates plugin to find library duplicates, bearing in mind that would take some time to do and need some hand management vs. to just let them merge during the copy. Turned out when testing, it really didn't slow the copy down noticeably enough to be worth spending the time dealing with it for this many items.

12-20-2015, 01:34 AM	#6
technomom Enthusiast Posts: 44 Karma: 10 Join Date: Sep 2005 Location: Atlanta, GA, US Device: Kindle Colorsoft & Paperwhite, iPad Pro, iPhone 15 Pro Max	I'm curious - how would one merge two libraries? Thank you!

12-20-2015, 01:37 AM	#7
eschwartz Ex-Helpdesk Junkie Posts: 19,421 Karma: 85400180 Join Date: Nov 2012 Location: The Beaten Path, USA, Roundworld, This Side of Infinity Device: Kindle Touch fw5.3.7 (Wifi only)	Select books ==> Right-click ==> Copy to library

12-20-2015, 02:30 AM	#8
technomom Enthusiast Posts: 44 Karma: 10 Join Date: Sep 2005 Location: Atlanta, GA, US Device: Kindle Colorsoft & Paperwhite, iPad Pro, iPhone 15 Pro Max	Thank you!

12-20-2015, 03:43 PM	#9
Rellwood Library Breeder (She/Her) Posts: 1,221 Karma: 1937891 Join Date: Apr 2015 Location: Fullerton, California Device: Paperwhite 2015 (2), PW 2024 (12 GEN), PW 2023 (11 GEN), Scribe (1st)	I have copied libraries so much that I worry that the books will lose their data by copying a copy of a copy...... Copy to library is the greatest feature, but causes me the most headaches when I end up with a mass of books and metadata that are inconsistent. However....the suggestion to hide the tags browser is a new one to me, and will definitely come in handy.

01-05-2016, 01:00 PM	#11
kovidgoyal creator of calibre Posts: 45,149 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	You can always stick print statements in the code to see where it is slowing down for you. http://manual.calibre-ebook.com/develop.html

Advert

Advert