Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 09-15-2018, 09:44 AM   #1
Hawkeye666+
Junior Member
Hawkeye666+ began at the beginning.
 
Posts: 5
Karma: 10
Join Date: Sep 2018
Device: Kindle
Calibre Organization Question

I have a very large database of Ebooks (>350K) now under Calibre across multiple libraries named according to genre but this has some disadvantages. I am considering reorganizing my collection according to the author and using the tag field to indicate the genre or genres involved. The new libraries would of course be broken down alphabetically by author name rather than by genre.

Does anyone have any experience in doing this and is aware of the downsides I have missed? I really don't want to screw this up beyond all redemption - I have been known to do things like that a time or two in the past and I don't think going back is an option once I commit.
Hawkeye666+ is offline   Reply With Quote
Old 09-15-2018, 02:04 PM   #2
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 22,364
Karma: 22265700
Join Date: Aug 2009
Location: (The original) Silicon Valley, USA
Device: K4NT, Galaxy Tab A, Kobo Aura2
IMHO One Library if the FIELD structure is the same is best to avoid duplicates.

A custom column or Tags works fine. Create Virtual Libraries (derived from standard searches), based upon your desired criteria for each
Now you hav the choice of the Full Library, or a restricted view.

One reason AGAINST: You want to maintain a solid access restriction (Age...)
This also requires that you NOT have a quick switch and The Content server also uses that list for the web based view
theducks is offline   Reply With Quote
Advert
Old 09-15-2018, 05:26 PM   #3
Hawkeye666+
Junior Member
Hawkeye666+ began at the beginning.
 
Posts: 5
Karma: 10
Join Date: Sep 2018
Device: Kindle
Thanks for the input. I can't use just one library because there are too many Ebooks involved. I prefer to hold the number under 40,000 per library to prevent processing impact.

Many times deciding what genre to use is pure guesswork, and some authors write in multiple genres. I spend too much time deciding where a book should go once I decide I wish to keep it and rearranging and expanding genre libraries when necessary is not a trivial task.

It seems to me that using tags (which is what all virtual libraries do anyway) would allow any genre to be accessed and isolated. The time saved in allocating new books and any resizing of libraries should be worthwhile but I don't want to do this if the result is purely cosmetic and there is no gain in efficiency.

I am looking for anything that would negatively impact keeping my collection in multiple libraries by author name rather than by genre as I do now.

One problem would be anthologies, but there is not a good solution for those no matter how they are organized. I keep them in a separate library now but they could be located just as easily with an Anthology tag.

I use Everything as a cross-library search utility for author and title but I wish there was a way to cross-library search for tags.
Hawkeye666+ is offline   Reply With Quote
Old 09-15-2018, 06:20 PM   #4
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,422
Karma: 10333332
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Ubuntu LTS, Huawei Mediapad 5, Bouye Likebook Plus
I suggest that you instead try to make calibre faster. I would hate having my books organized in different libraries based on author. Then you loose the whole point of using calibre: To be able browse your books at random and find something to read that you didn't know you were looking for.

Instead try to consolidate into fewer libraries.

I have my titles in only three main libraries. Fiction, non-fiction and periodicals. Also I have one for incoming and one for data sheets, documentation and manuals.

Get a faster computer. Compact the database. More memory. Use a RAM-disk. Remove all calculated or dynamic fields. Disable antivirus. An SSD.
Adoby is online now   Reply With Quote
Old 09-15-2018, 07:26 PM   #5
Hawkeye666+
Junior Member
Hawkeye666+ began at the beginning.
 
Posts: 5
Karma: 10
Join Date: Sep 2018
Device: Kindle
I agree the proper hardware upgrade would solve a lot of problems but this is a hobby. I don't make a penny from it and I am not willing to replace perfectly satisfy hardware with several thousand dollars worth of new equipment. When W7 dies in 2020 or 2021 that will be soon enough for an upgrade.

I do admit I miss my big iron days with VM and REXX that would laugh at what I work with now.

I currently have about 350,000 titles under Calibre with another 150,000 or so that are not yet researched and loaded. That kind of volume will choke just about anything but an bleeding edge I7 with a 3T SSD minimum, which is not cheap. Even then I am not sure it would handle everything in one library.

Plus I am at an age where I look at an investment and decide whether I will live long enough to get my money's worth. Would you put tires with a 100,000 warranty on a car with 100,000 miles already on it?

Thanks for your input - opinions based on experience are exactly what I am looking for.
Hawkeye666+ is offline   Reply With Quote
Advert
Old 09-15-2018, 09:06 PM   #6
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 11,514
Karma: 10584684
Join Date: Mar 2012
Location: Sydney Australia
Device: none
@Adoby - ↑ ↑ ↑

Similarly, I have three 'real' libraries - Texts [150,000+], Media [8,000] (audio and video broad/pod-casts), and Periodicals [6000]. They have different custom columns, and the Media and Periodicals libraries use the Title and Author columns 'perversely'. I also have an Intake, and a Resolution (a parking bay for duplicates etc) library. The numbers in [] are approximate number of books.

Memory seems to be what calibre likes most. 64bit Win 7 Pro can address up to 192GB, but I have a gut feeling there's a practical limit of 32GB, assuming you can even physically cram that much onto your mobo.

My rig is an i7 with 16GB of RAM, a 256GB NVMe SSD system drive, running Windows 10 Pro 1803. My libraries are on an internal 4TB SATA3 drive. Calibre starts in < 10 seconds irrespective of which library it's opening.

I got this system in August 2017, prior to that I had an i5 with 6GB of RAM, 128GB SATA2 SSD system drive, and 2TB SATA2 data drive, with Win7 Pro. At that time my Texts library had about 120,000 books. And calibre start-up times were more or less the same as they are now.

I'm with you on the 'will I live that long' question. I had to get a new system because my old one had persistent problems that seemed to emanate from the mobo, three video cards and a memory stick 'blew up' over a couple of years.

You could try using the Consolidate All Library Metadata plugin to test out using a single library.

BTW I run 2AVs, Defender and MWB; two content indexers Windows (docx, epub, pdf) and X1 (docx and pdf); and Windows File History. IMO they have no discernible affect on calibre performance, on this rig or the previous one.

BR

Last edited by BetterRed; 09-15-2018 at 10:57 PM.
BetterRed is offline   Reply With Quote
Old 09-15-2018, 11:35 PM   #7
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 22,364
Karma: 22265700
Join Date: Aug 2009
Location: (The original) Silicon Valley, USA
Device: K4NT, Galaxy Tab A, Kobo Aura2
I really don't think Calibre needs a Game quality machine. It is DATA intensive.

I had this rammed home. The Boss bought a hot i5 (this was early 2K) to create a 8 stream, Video streaming test bed. It kept crashing streams.
The card manufacturer was puzzled. they ran 4 cards (we had 2) without issue.
Turned out, they were using a (Dell) server. Turns out the data bus is designed different. We got a 1U, 386 server and our problems went away.
theducks is offline   Reply With Quote
Old 09-16-2018, 07:11 AM   #8
Divingduck
Wizard
Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.
 
Posts: 1,119
Karma: 1404167
Join Date: Nov 2010
Location: Germany
Device: Sony PRS-650
If the bottleneck is hard disk bandwidth for huge disks, RAID 0 with two hd's can help too if the mainboard supports it.
Divingduck is offline   Reply With Quote
Old 09-16-2018, 09:07 AM   #9
Hawkeye666+
Junior Member
Hawkeye666+ began at the beginning.
 
Posts: 5
Karma: 10
Join Date: Sep 2018
Device: Kindle
Thanks for all the replies - they have given me some new things to think about and some new approaches to consider. And thanks for all of the plug-in suggestions. I am still considering my options and since there is no rush I may take a while before I make any changes.

Please don't hesitate if you think of something new - as usual I need all the help I can get.
Hawkeye666+ is offline   Reply With Quote
Old 09-16-2018, 10:58 AM   #10
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,422
Karma: 10333332
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Ubuntu LTS, Huawei Mediapad 5, Bouye Likebook Plus
I think more RAM would be the first thing to try. A lot more.

With enough RAM it would be possible to copy the whole database to a RAM-disk and have calibre access it from there. Also you could put the temp folder on the RAM-disk. And more memory, with both write and read cache activated, would mean faster hdd access.

You might run calibre from a script that prepare the RAM-disk and copy the database(s) before it starts calibre. And when calibre is exited, copies the database(s) back to the hdd and takes a backup snapshot of the whole calibre libraries, databases and books and all.

Quote:
Plus I am at an age where I look at an investment and decide whether I will live long enough to get my money's worth. Would you put tires with a 100,000 warranty on a car with 100,000 miles already on it?
If it was the last car I expected to ever own, and I really liked driving and could afford it? Oh, yes! In a heartbeat!

It's the same with bibliophiles. They like nice libraries and book cases. An ebookophile(?) is happy to invest in new e-readers and upgrade the computer to manage the books.

Money CAN give at least a modicum of joy and pleasure, but only if it is spent...

Last edited by Adoby; 09-16-2018 at 11:14 AM.
Adoby is online now   Reply With Quote
Old 09-20-2018, 05:04 AM   #11
Hawkeye666+
Junior Member
Hawkeye666+ began at the beginning.
 
Posts: 5
Karma: 10
Join Date: Sep 2018
Device: Kindle
Thanks everyone for all of the suggestions. They were pretty much what I wanted.

I have decided to leave well enough alone. I was trying to reduce the amount of maintenance time involved but upon reflection the overall work would not be decreased since at some point I would still need the genre involved and I would only be changing one control method for another.

I am going to build a test library of duplicates and see just how big I can make it before I choke down completely. No more guesswork.

Thank you again for the input, and if anyone thinks of something new plug it in.
Hawkeye666+ is offline   Reply With Quote
Old 09-20-2018, 09:34 AM   #12
Adoby
Handy Elephant
Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.Adoby ought to be getting tired of karma fortunes by now.
 
Adoby's Avatar
 
Posts: 1,422
Karma: 10333332
Join Date: Dec 2009
Location: Southern Sweden, far out in the quiet woods
Device: Ubuntu LTS, Huawei Mediapad 5, Bouye Likebook Plus
Perhaps the easiest thing to try is to add a small, cheap and fast SSD. Then you can have the calibre database(s) on the SSD but keep the books themselves on big, cheap and slow spinning iron on the HDD.

And you could also put the TEMP-folder and swap-memory on the SSD.

So you don't need to buy an expensive huge multi-TB SSD to speed up calibre. It is sufficient with a small one with room for just the database files (perhaps times two or three to allow for maintenance work, TEMP and swap).

Also you might consider updating to a free high performance modern operating system. I run the latest Ubuntu Linux. These days Microsoft supports running Ubuntu on a Windows box, but that is not very performant. Better to ditch Windows altogether.

Last edited by Adoby; 09-20-2018 at 09:44 AM.
Adoby is online now   Reply With Quote
Old 09-20-2018, 01:40 PM   #13
Tarana
Wizard
Tarana ought to be getting tired of karma fortunes by now.Tarana ought to be getting tired of karma fortunes by now.Tarana ought to be getting tired of karma fortunes by now.Tarana ought to be getting tired of karma fortunes by now.Tarana ought to be getting tired of karma fortunes by now.Tarana ought to be getting tired of karma fortunes by now.Tarana ought to be getting tired of karma fortunes by now.Tarana ought to be getting tired of karma fortunes by now.Tarana ought to be getting tired of karma fortunes by now.Tarana ought to be getting tired of karma fortunes by now.Tarana ought to be getting tired of karma fortunes by now.
 
Tarana's Avatar
 
Posts: 1,949
Karma: 20079154
Join Date: Sep 2012
Location: Minneapolis
Device: Voyage, K3, PPW 2 3, HDX, KBasic 4, 5, 7 & 8, Nook ST, Glo, HD+, Echos
If you are wanting separate libraries, consider instead separating by genre rather than by author. For instance, I put all the non-fiction into a separate library. That includes craft and cookbooks. I also have a separate library for science fiction and fantasy because my late sis read nothing else and didn't want to scroll through all my other books. I set it up this way before Kovid added Virtual Libraries. I have to spend a day figuring out how to combine them now that it is no longer necessary.

My reasons were more about access for other family members rather than the size of the library, but it works the same. Separating libraries by author just increases the time it takes you to maintain the library and find a book.
Tarana is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Organization in Calibre jhempel24 Calibre 2 08-20-2010 07:41 AM
calibre organization and sony functions? tmb2 Calibre 12 03-31-2010 03:39 PM
Folder organization question timezone iRex 12 04-17-2009 01:17 PM
Question re: ebook organization PsyDocJoanne Sony Reader 14 10-03-2008 11:48 AM


All times are GMT -4. The time now is 07:21 AM.


MobileRead.com is a privately owned, operated and funded community.