Calibre Organization Question

Hawkeye666+ · 09-15-2018, 08:44 AM

I have a very large database of Ebooks (>350K) now under Calibre across multiple libraries named according to genre but this has some disadvantages. I am considering reorganizing my collection according to the author and using the tag field to indicate the genre or genres involved. The new libraries would of course be broken down alphabetically by author name rather than by genre.

Does anyone have any experience in doing this and is aware of the downsides I have missed? I really don't want to screw this up beyond all redemption - I have been known to do things like that a time or two in the past and I don't think going back is an option once I commit.

theducks · 09-15-2018, 01:04 PM

IMHO One Library if the FIELD structure is the same is best to avoid duplicates.

A custom column or Tags works fine. Create Virtual Libraries (derived from standard searches), based upon your desired criteria for each
Now you hav the choice of the Full Library, or a restricted view.

One reason AGAINST: You want to maintain a solid access restriction (Age...)
This also requires that you NOT have a quick switch and The Content server also uses that list for the web based view

Hawkeye666+ · 09-15-2018, 04:26 PM

Thanks for the input. I can't use just one library because there are too many Ebooks involved. I prefer to hold the number under 40,000 per library to prevent processing impact.

Many times deciding what genre to use is pure guesswork, and some authors write in multiple genres. I spend too much time deciding where a book should go once I decide I wish to keep it and rearranging and expanding genre libraries when necessary is not a trivial task.

It seems to me that using tags (which is what all virtual libraries do anyway) would allow any genre to be accessed and isolated. The time saved in allocating new books and any resizing of libraries should be worthwhile but I don't want to do this if the result is purely cosmetic and there is no gain in efficiency.

I am looking for anything that would negatively impact keeping my collection in multiple libraries by author name rather than by genre as I do now.

One problem would be anthologies, but there is not a good solution for those no matter how they are organized. I keep them in a separate library now but they could be located just as easily with an Anthology tag.

I use Everything as a cross-library search utility for author and title but I wish there was a way to cross-library search for tags.

Adoby · 09-15-2018, 05:20 PM

I suggest that you instead try to make calibre faster. I would hate having my books organized in different libraries based on author. Then you loose the whole point of using calibre: To be able browse your books at random and find something to read that you didn't know you were looking for.

Instead try to consolidate into fewer libraries.

I have my titles in only three main libraries. Fiction, non-fiction and periodicals. Also I have one for incoming and one for data sheets, documentation and manuals.

Get a faster computer. Compact the database. More memory. Use a RAM-disk. Remove all calculated or dynamic fields. Disable antivirus. An SSD.

Hawkeye666+ · 09-15-2018, 06:26 PM

I agree the proper hardware upgrade would solve a lot of problems but this is a hobby. I don't make a penny from it and I am not willing to replace perfectly satisfy hardware with several thousand dollars worth of new equipment. When W7 dies in 2020 or 2021 that will be soon enough for an upgrade.

I do admit I miss my big iron days with VM and REXX that would laugh at what I work with now.

I currently have about 350,000 titles under Calibre with another 150,000 or so that are not yet researched and loaded. That kind of volume will choke just about anything but an bleeding edge I7 with a 3T SSD minimum, which is not cheap. Even then I am not sure it would handle everything in one library.

Plus I am at an age where I look at an investment and decide whether I will live long enough to get my money's worth. Would you put tires with a 100,000 warranty on a car with 100,000 miles already on it?

Thanks for your input - opinions based on experience are exactly what I am looking for.

BetterRed · 09-15-2018, 08:06 PM

@Adoby - ↑ ↑ ↑

Similarly, I have three 'real' libraries - Texts [150,000+], Media [8,000] (audio and video broad/pod-casts), and Periodicals [6000]. They have different custom columns, and the Media and Periodicals libraries use the Title and Author columns 'perversely'. I also have an Intake, and a Resolution (a parking bay for duplicates etc) library. The numbers in [] are approximate number of books.

Memory seems to be what calibre likes most. 64bit Win 7 Pro can address up to 192GB, but I have a gut feeling there's a practical limit of 32GB, assuming you can even physically cram that much onto your mobo.

My rig is an i7 with 16GB of RAM, a 256GB NVMe SSD system drive, running Windows 10 Pro 1803. My libraries are on an internal 4TB SATA3 drive. Calibre starts in < 10 seconds irrespective of which library it's opening.

I got this system in August 2017, prior to that I had an i5 with 6GB of RAM, 128GB SATA2 SSD system drive, and 2TB SATA2 data drive, with Win7 Pro. At that time my Texts library had about 120,000 books. And calibre start-up times were more or less the same as they are now.

I'm with you on the 'will I live that long' question. I had to get a new system because my old one had persistent problems that seemed to emanate from the mobo, three video cards and a memory stick 'blew up' over a couple of years.

You could try using the Consolidate All Library Metadata plugin to test out using a single library.

BTW I run 2AVs, Defender and MWB; two content indexers Windows (docx, epub, pdf) and X1 (docx and pdf); and Windows File History. IMO they have no discernible affect on calibre performance, on this rig or the previous one.

BR

theducks · 09-15-2018, 10:35 PM

I really don't think Calibre needs a Game quality machine. It is DATA intensive.

I had this rammed home. The Boss bought a hot i5 (this was early 2K) to create a 8 stream, Video streaming test bed. It kept crashing streams.
The card manufacturer was puzzled. they ran 4 cards (we had 2) without issue.
Turned out, they were using a (Dell) server. Turns out the data bus is designed different. We got a 1U, 386 server and our problems went away.

Divingduck · 09-16-2018, 06:11 AM

If the bottleneck is hard disk bandwidth for huge disks, RAID 0 with two hd's can help too if the mainboard supports it.

Hawkeye666+ · 09-16-2018, 08:07 AM

Thanks for all the replies - they have given me some new things to think about and some new approaches to consider. And thanks for all of the plug-in suggestions. I am still considering my options and since there is no rush I may take a while before I make any changes.

Please don't hesitate if you think of something new - as usual I need all the help I can get.

Adoby · 09-16-2018, 09:58 AM

I think more RAM would be the first thing to try. A lot more.

With enough RAM it would be possible to copy the whole database to a RAM-disk and have calibre access it from there. Also you could put the temp folder on the RAM-disk. And more memory, with both write and read cache activated, would mean faster hdd access.

You might run calibre from a script that prepare the RAM-disk and copy the database(s) before it starts calibre. And when calibre is exited, copies the database(s) back to the hdd and takes a backup snapshot of the whole calibre libraries, databases and books and all.

Quote:

Plus I am at an age where I look at an investment and decide whether I will live long enough to get my money's worth. Would you put tires with a 100,000 warranty on a car with 100,000 miles already on it?

If it was the last car I expected to ever own, and I really liked driving and could afford it? Oh, yes! In a heartbeat!

It's the same with bibliophiles. They like nice libraries and book cases. An ebookophile(?) is happy to invest in new e-readers and upgrade the computer to manage the books.

Money CAN give at least a modicum of joy and pleasure, but only if it is spent...

Hawkeye666+ · 09-20-2018, 04:04 AM

Thanks everyone for all of the suggestions. They were pretty much what I wanted.

I have decided to leave well enough alone. I was trying to reduce the amount of maintenance time involved but upon reflection the overall work would not be decreased since at some point I would still need the genre involved and I would only be changing one control method for another.

I am going to build a test library of duplicates and see just how big I can make it before I choke down completely. No more guesswork.

Thank you again for the input, and if anyone thinks of something new plug it in.

Adoby · 09-20-2018, 08:34 AM

Perhaps the easiest thing to try is to add a small, cheap and fast SSD. Then you can have the calibre database(s) on the SSD but keep the books themselves on big, cheap and slow spinning iron on the HDD.

And you could also put the TEMP-folder and swap-memory on the SSD.

So you don't need to buy an expensive huge multi-TB SSD to speed up calibre. It is sufficient with a small one with room for just the database files (perhaps times two or three to allow for maintenance work, TEMP and swap).

Also you might consider updating to a free high performance modern operating system. I run the latest Ubuntu Linux. These days Microsoft supports running Ubuntu on a Windows box, but that is not very performant. Better to ditch Windows altogether.

Tarana · 09-20-2018, 12:40 PM

If you are wanting separate libraries, consider instead separating by genre rather than by author. For instance, I put all the non-fiction into a separate library. That includes craft and cookbooks. I also have a separate library for science fiction and fantasy because my late sis read nothing else and didn't want to scroll through all my other books. I set it up this way before Kovid added Virtual Libraries. I have to spend a day figuring out how to combine them now that it is no longer necessary.

My reasons were more about access for other family members rather than the size of the library, but it works the same. Separating libraries by author just increases the time it takes you to maintain the library and find a book.

09-15-2018, 08:44 AM	#1
Hawkeye666+ Junior Member Posts: 7 Karma: 10 Join Date: Sep 2018 Device: Kindle	Calibre Organization Question I have a very large database of Ebooks (>350K) now under Calibre across multiple libraries named according to genre but this has some disadvantages. I am considering reorganizing my collection according to the author and using the tag field to indicate the genre or genres involved. The new libraries would of course be broken down alphabetically by author name rather than by genre. Does anyone have any experience in doing this and is aware of the downsides I have missed? I really don't want to screw this up beyond all redemption - I have been known to do things like that a time or two in the past and I don't think going back is an option once I commit.

09-15-2018, 08:06 PM	#6
BetterRed null operator (he/him) Posts: 20,568 Karma: 26954694 Join Date: Mar 2012 Location: Sydney Australia Device: none	@Adoby - ↑ ↑ ↑ Similarly, I have three 'real' libraries - Texts [150,000+], Media [8,000] (audio and video broad/pod-casts), and Periodicals [6000]. They have different custom columns, and the Media and Periodicals libraries use the Title and Author columns 'perversely'. I also have an Intake, and a Resolution (a parking bay for duplicates etc) library. The numbers in [] are approximate number of books. Memory seems to be what calibre likes most. 64bit Win 7 Pro can address up to 192GB, but I have a gut feeling there's a practical limit of 32GB, assuming you can even physically cram that much onto your mobo. My rig is an i7 with 16GB of RAM, a 256GB NVMe SSD system drive, running Windows 10 Pro 1803. My libraries are on an internal 4TB SATA3 drive. Calibre starts in < 10 seconds irrespective of which library it's opening. I got this system in August 2017, prior to that I had an i5 with 6GB of RAM, 128GB SATA2 SSD system drive, and 2TB SATA2 data drive, with Win7 Pro. At that time my Texts library had about 120,000 books. And calibre start-up times were more or less the same as they are now. I'm with you on the 'will I live that long' question. I had to get a new system because my old one had persistent problems that seemed to emanate from the mobo, three video cards and a memory stick 'blew up' over a couple of years. You could try using the Consolidate All Library Metadata plugin to test out using a single library. BTW I run 2AVs, Defender and MWB; two content indexers Windows (docx, epub, pdf) and X1 (docx and pdf); and Windows File History. IMO they have no discernible affect on calibre performance, on this rig or the previous one. BR Last edited by BetterRed; 09-15-2018 at 09:57 PM.

09-20-2018, 08:34 AM	#12
Adoby Handy Elephant Posts: 1,736 Karma: 26785668 Join Date: Dec 2009 Location: Southern Sweden, far out in the quiet woods Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus	Perhaps the easiest thing to try is to add a small, cheap and fast SSD. Then you can have the calibre database(s) on the SSD but keep the books themselves on big, cheap and slow spinning iron on the HDD. And you could also put the TEMP-folder and swap-memory on the SSD. So you don't need to buy an expensive huge multi-TB SSD to speed up calibre. It is sufficient with a small one with room for just the database files (perhaps times two or three to allow for maintenance work, TEMP and swap). Also you might consider updating to a free high performance modern operating system. I run the latest Ubuntu Linux. These days Microsoft supports running Ubuntu on a Windows box, but that is not very performant. Better to ditch Windows altogether. Last edited by Adoby; 09-20-2018 at 08:44 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Organization in Calibre	jhempel24	Calibre	2	08-20-2010 06:41 AM
calibre organization and sony functions?	tmb2	Calibre	12	03-31-2010 02:39 PM
Folder organization question	timezone	iRex	12	04-17-2009 12:17 PM
Question re: ebook organization	PsyDocJoanne	Sony Reader	14	10-03-2008 10:48 AM

09-15-2018, 01:04 PM	#2
theducks Well trained by Cats Posts: 29,800 Karma: 54830978 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	IMHO One Library if the FIELD structure is the same is best to avoid duplicates. A custom column or Tags works fine. Create Virtual Libraries (derived from standard searches), based upon your desired criteria for each Now you hav the choice of the Full Library, or a restricted view. One reason AGAINST: You want to maintain a solid access restriction (Age...) This also requires that you NOT have a quick switch and The Content server also uses that list for the web based view

09-15-2018, 04:26 PM	#3
Hawkeye666+ Junior Member Posts: 7 Karma: 10 Join Date: Sep 2018 Device: Kindle	Thanks for the input. I can't use just one library because there are too many Ebooks involved. I prefer to hold the number under 40,000 per library to prevent processing impact. Many times deciding what genre to use is pure guesswork, and some authors write in multiple genres. I spend too much time deciding where a book should go once I decide I wish to keep it and rearranging and expanding genre libraries when necessary is not a trivial task. It seems to me that using tags (which is what all virtual libraries do anyway) would allow any genre to be accessed and isolated. The time saved in allocating new books and any resizing of libraries should be worthwhile but I don't want to do this if the result is purely cosmetic and there is no gain in efficiency. I am looking for anything that would negatively impact keeping my collection in multiple libraries by author name rather than by genre as I do now. One problem would be anthologies, but there is not a good solution for those no matter how they are organized. I keep them in a separate library now but they could be located just as easily with an Anthology tag. I use Everything as a cross-library search utility for author and title but I wish there was a way to cross-library search for tags.

09-15-2018, 05:20 PM	#4
Adoby Handy Elephant Posts: 1,736 Karma: 26785668 Join Date: Dec 2009 Location: Southern Sweden, far out in the quiet woods Device: Thinkpad E595, Ubuntu Mate, Huawei Mediapad 5, Bouye Likebook Plus	I suggest that you instead try to make calibre faster. I would hate having my books organized in different libraries based on author. Then you loose the whole point of using calibre: To be able browse your books at random and find something to read that you didn't know you were looking for. Instead try to consolidate into fewer libraries. I have my titles in only three main libraries. Fiction, non-fiction and periodicals. Also I have one for incoming and one for data sheets, documentation and manuals. Get a faster computer. Compact the database. More memory. Use a RAM-disk. Remove all calculated or dynamic fields. Disable antivirus. An SSD.

09-15-2018, 06:26 PM	#5
Hawkeye666+ Junior Member Posts: 7 Karma: 10 Join Date: Sep 2018 Device: Kindle	I agree the proper hardware upgrade would solve a lot of problems but this is a hobby. I don't make a penny from it and I am not willing to replace perfectly satisfy hardware with several thousand dollars worth of new equipment. When W7 dies in 2020 or 2021 that will be soon enough for an upgrade. I do admit I miss my big iron days with VM and REXX that would laugh at what I work with now. I currently have about 350,000 titles under Calibre with another 150,000 or so that are not yet researched and loaded. That kind of volume will choke just about anything but an bleeding edge I7 with a 3T SSD minimum, which is not cheap. Even then I am not sure it would handle everything in one library. Plus I am at an age where I look at an investment and decide whether I will live long enough to get my money's worth. Would you put tires with a 100,000 warranty on a car with 100,000 miles already on it? Thanks for your input - opinions based on experience are exactly what I am looking for.

09-15-2018, 10:35 PM	#7
theducks Well trained by Cats Posts: 29,800 Karma: 54830978 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	I really don't think Calibre needs a Game quality machine. It is DATA intensive. I had this rammed home. The Boss bought a hot i5 (this was early 2K) to create a 8 stream, Video streaming test bed. It kept crashing streams. The card manufacturer was puzzled. they ran 4 cards (we had 2) without issue. Turned out, they were using a (Dell) server. Turns out the data bus is designed different. We got a 1U, 386 server and our problems went away.

09-16-2018, 06:11 AM	#8
Divingduck Wizard Posts: 1,161 Karma: 1404241 Join Date: Nov 2010 Location: Germany Device: Sony PRS-650	If the bottleneck is hard disk bandwidth for huge disks, RAID 0 with two hd's can help too if the mainboard supports it.

09-16-2018, 08:07 AM	#9
Hawkeye666+ Junior Member Posts: 7 Karma: 10 Join Date: Sep 2018 Device: Kindle	Thanks for all the replies - they have given me some new things to think about and some new approaches to consider. And thanks for all of the plug-in suggestions. I am still considering my options and since there is no rush I may take a while before I make any changes. Please don't hesitate if you think of something new - as usual I need all the help I can get.

09-20-2018, 04:04 AM	#11
Hawkeye666+ Junior Member Posts: 7 Karma: 10 Join Date: Sep 2018 Device: Kindle	Thanks everyone for all of the suggestions. They were pretty much what I wanted. I have decided to leave well enough alone. I was trying to reduce the amount of maintenance time involved but upon reflection the overall work would not be decreased since at some point I would still need the genre involved and I would only be changing one control method for another. I am going to build a test library of duplicates and see just how big I can make it before I choke down completely. No more guesswork. Thank you again for the input, and if anyone thinks of something new plug it in.

09-20-2018, 12:40 PM	#13
Tarana Wizard Posts: 3,978 Karma: 38840460 Join Date: Sep 2012 Location: Minneapolis Device: PWSE, Voyage, K3, HDX, KBasic 7 & 8, Nook Glo3, Echos, Nanos	If you are wanting separate libraries, consider instead separating by genre rather than by author. For instance, I put all the non-fiction into a separate library. That includes craft and cookbooks. I also have a separate library for science fiction and fantasy because my late sis read nothing else and didn't want to scroll through all my other books. I set it up this way before Kovid added Virtual Libraries. I have to spend a day figuring out how to combine them now that it is no longer necessary. My reasons were more about access for other family members rather than the size of the library, but it works the same. Separating libraries by author just increases the time it takes you to maintain the library and find a book.