Managing multiple libraries in Calibre

[DMcCunney]

I'm starting to contemplate breaking out my Calibre Library into more than one volume.

There are several reasons for this. One is the size of the library.

I have the master library on the HD, and a backup on a USB thumb drive. After I've added books and updated the metadata, I use an open source program called Free File Sync to mirror the changes on the HD copy to the thumb drive. I've already had to migrate to a larger thumb drive, and at some point may need to do so again.

Another, related to the first, is content. My normal eBook viewer is a 7" Android tablet, viewed using the open source FBReader for Android. By preference, I get ePub, but sometimes there isn't an ePub version. Some things I only see in Mobi format (which can be converted to ePub, but since FBReader handles ePub and Mobi, I usually don't bother.)

More often, the content is only available as PDF, and some of those are huge (like stuff from Archive.org, where 100MB or larger file size is not uncommon.) In general, I don't even try to read PDFs on the tablet. FBReader has a PDF viewer plugin that works well in that it will display the PDFs, but the 7" screen makes trying to read them painful, and the big ones take an approximation of forever to open.

So I'm looking at splitting out the Calibre Library by content type, with ePub and Mobi in one library instance, and PDF and other things in a second. The ePub/Mobi library is that one that would contain things to be added to the tablet. There would be a second backup USB thumb drive for the PDFs and the like. (I'm aware of and am experimenting with Virtual Libraries. That's handy for selecting titles to potentially add the the tablet, but doesn't address the physical size of the library problem.)

One issue I don't see a way to address is being able to get a top level overview that shows all volumes in both libraries. I can live without it if I must, but it would be nice to have a list of everything I could search to see what I have.

Another is the best way to use Calibre to do the split.

Is anyone else using Calibre to manage multiple physical libraries? What sort of strategy do you use to do so?
______
Dennis

[theducks]

IMHO the overview part of this is the issue.

Physical Libraries can be on any (full access) local drive


As to Moving Files:
Select Books: Right-click: Copy to Library:<name> (optional delete after copy)

The Selection process gets tricky.

Basic Q Do you want to move ANY book (all formats) that HAS a PDF?

I used a custom column to track Master (source) formats as a tool against future conversions, wher I want to use the Original as the source.

If you have only PDF, piece of cake (chocolate ) ; Use the Tag Browser to select PDF in the custom col or
search: formats:PDF and Not formats:EPUB

The goal is to get only the books that exist solely as a PDF

I believe Betteread has large libraries

[DMcCunney]

Originally Posted by theducks

IMHO the overview part of this is the issue.

Physical Libraries can be on any (full access) local drive

As to Moving Files:
Select Books: Right-click: Copy to Library:<name> (optional delete after copy)

The selection would be on a format search, with a right-click on all hits, and delete after copy.

The Selection process gets tricky.

Basic Q Do you want to move ANY book (all formats) that HAS a PDF?

By preference, I get ePub, and other things if ePub isn't available. The vast majority of the PDFs exist only as PDFs.

If by some chance there are multiple formats including PDF, I think I want to move the PDF but leave the others.

I used a custom column to track Master (source) formats as a tool against future conversions, where I want to use the Original as the source.

I haven't done that yet, but may start doing so down the road. It's a good idea.

If you have only PDF, piece of cake (chocolate ) ; Use the Tag Browser to select PDF in the custom col or
search: formats:PDF and Not formats:EPUB

The goal is to get only the books that exist solely as a PDF

It's a little more complicated than that. Essentially, one library would be ePub/Mobi, and the other would be "Everything else". Most of the everything else will be PDF, but other things like DjVu are creeping into the mix.

So the first cut will likely be moving all ePub/Mobi to a different library, and what's left is the other library. The ePpub/Mobi library is the one that will sync with the tablet.

I believe Betteread has large libraries

I'm open to commentary from all.

I still have the issue of a top level listing of what is in both libraries, just to see whether I have the title at all.

(Hmmm. Calibre holds the data in a SQLite database. There are likely tools for exporting from SQLite to CSV...)
______
Dennis

[BetterRed]

@DMcCunney - How big is your library? My initial reaction to splitting a library into multiples based on format and the capacity of backup thumb drives is - no, no, don't do that!!

I prefer to use good quality hard disks for backup rather than thumb drives - 2TB drives in a USB 3 disk dock. My calibre libraries consume ~700GB (one them has lots of large files - video, audio and image collections), the non calibre data is ~400GB. I use something similar to FFS to do the backups called Goodsynch.

Cloud is another option - I believe Amazon gives unlimited storage for US$60pa. My line speeds are too slow to consider using cloud storage for anything other than casual use.

Have you considered using Calibre Companion on your Android devices. It interoperates with desktop calibre and it can access a calibre library kept in Dropbox and OneDrive accounts - not sure about Amazon.

My use of multiple libraries relates to custom columns and how I use standard columns. I have three 'permanent' libraries - Main, Media and Journals.

My Main library (~85K books) has 6 custom columns, my Media library (~12K books) has 26 custom columns, and my Journals library (~6K books) has the Journal Name in the Author column and the Journal Issue No/Date in Title column and 11 custom columns. All three libraries have a number of Virtual Libraries and book list Views (View Manager plugin).

I tried combining them into into a single library, and contriving schemes to partition the Virtual Libraries and Views. But I found it far too cumbersome with limited benefit, so I reverted to multiple libraries.

I use X1 or Windows Search for content searching. In the context of calibre I use the Drop Search Results and Reading List PIs to store the search results.

I bridge the libraries via links from calibre Books and Authors to Evernote Notes, and via Chrome shortcuts from Evernote Notes to calibre server Author and Book URLs. I would do this irrespective of the number of libraries.

BR

[DMcCunney]

Originally Posted by BetterRed

@DMcCunney - How big is your library?

99+ GB

I am not trying to use Calibre as a general purpose library and indexing utility, though I'm aware I could.

The stuff indexed by Calibre is all essentially text in one format or another.

My initial reaction to splitting a library into multiples based on format and the capacity of backup thumb drives is - no, no, don't do that!!

I prefer to use good quality hard disks for backup rather than thumb drives - 2TB drives in a USB 3 disk dock. My calibre libraries consume ~700GB (one them has lots of large files - video, audio and image collections), the non calibre data is ~400GB. I use something similar to FFS to do the backups called Goodsynch.

The point to the USB thumbdrive is less backup than portability. Calibre is installed on the machines that travel, too, and when I travel, the Calibre library can travel with me.

Yes, there are portable HDs. But those are mostly spinning platter devices, and things with moving parts break. I've had one old low capacity USB thumbdrive fail on me, and they are cheap and getting cheaper. (The 128GB USB3 thumbdrive that currently holds the backup Calibre library instance cost about $16. Storage is down to under ten cents a GB and dropping. I'm probably getting another 128GB thumbdrive and duplicating the current Calibre drive as a "belt and suspenders" move.)

Cloud is another option - I believe Amazon gives unlimited storage for US$60pa. My line speeds are too slow to consider using cloud storage for anything other than casual use.

I have a 100mbit/sec cable connection with 20mbit/sec upload, but I'm not at the point of using the cloud for backup.

Have you considered using Calibre Companion on your Android devices. It interoperates with desktop calibre and it can access a calibre library kept in Dropbox and OneDrive accounts - not sure about Amazon.

I have it installed on the tablet. The problem I have is that it can't do what I originally wanted. What I was looking for was an Android utility that could tell me what all I had on my tablet's microSD card using the metadata Calibre deposits there without invoking my eBook viewer.

It appears CC can do that sort of thing, but only for volumes transferred via Wifi. I have thousands of volumes on the tablet transferred via USB cable, and I really don't want to redo everything just to use CC.

I'm not especially interested in keeping this sort of thing on Dropbox or OneDrive. I make increasing use of Google Drive for stuff I want access to from anywhere, but that tends to be stuff I want to provide access to to others as well because we are collaborating on the projects the stuff is for.

My use of multiple libraries relates to custom columns and how I use standard columns. I have three 'permanent' libraries - Main, Media and Journals.

My Main library (~85K books) has 6 custom columns, my Media library (~12K books) has 26 custom columns, and my Journals library (~6K books) has the Journal Name in the Author column and the Journal Issue No/Date in Title column and 11 custom columns. All three libraries have a number of Virtual Libraries and book list Views (View Manager plugin).

I tried combining them into into a single library, and contriving schemes to partition the Virtual Libraries and Views. But I found it far too cumbersome with limited benefit, so I reverted to multiple libraries.

In your use case, I'd do the same. I have about 20K volumes all told, and no present need to rice, slice, and dice them as you do.

I use X1 or Windows Search for content searching. In the context of calibre I use the Drop Search Results and Reading List PIs to store the search results.

I'm still thinking about content searching. I dual boot Windows and Linux, so I'd want equivalent functionality available from either OS.

Back when, I ran Google Desktop under Windows as a content searching tool, because it could open and index content in archives with the right plugins. Finding files isn't a problem. Finding specific information in files is another matter. (In the old days, it was all text and I'd just use grep. No longer...)

I bridge the libraries via links from calibre Books and Authors to Evernote Notes, and via Chrome shortcuts from Evernote Notes to calibre server Author and Book URLs. I would do this irrespective of the number of libraries.

I don't use Evernote or Chrome. (I may get to Evernote at some point. I vastly prefer Firefox to Chrome.)

If you want to get a single top level list of what's in all three of your libraries, how do you do it?
______
Dennis

[theducks]

A 1T laptop (2.5") sized (external case) drive is nicely portable. Not as small as flash, but very nice performance even if only USB2.
As a Bonus, you can install Calibre Portable and still use the Library with the Desktop version, thus giving the ability to use the Library at any Windows computer

[DMcCunney]

Originally Posted by theducks

A 1T laptop (2.5") sized (external case) drive is nicely portable. Not as small as flash, but very nice performance even if only USB2.

Size and simplicity trump. I'm aware drives are getting smaller, higher capacity, and cheaper. For the current use case, they are far more than required.

As a Bonus, you can install Calibre Portable and still use the Library with the Desktop version, thus giving the ability to use the Library at any Windows computer

I have Calibre Portable on another thumbdrive, for the use case where the machine I'm using doesn't have Calibre installed. (Both current Calibre Portable, and the last version that still runs under XP.)

But that would be for use on someone else's machine. Mine all have Calibre on them, under Windows and Linux.

I just basically need the stuff I am likely to sync to my tablet on the thumb drive that travels, which means ePub/Mobi, and it will be quite some time before a 1TB drive will be required to handle that. (An additional constraint is tablet capacity. The max MicroSD card size the current one takes is 32GB.)
______
Dennis

[kovidgoyal]

You are never going to have builtin tools that present a unified view of multiple libraries in calibre. After all separate libraries are *separate*

But you can use the create catalog function in calibre to create a csv of all books in any library and them use whatever tool you like to process that csv.

[DMcCunney]

Originally Posted by kovidgoyal

You are never going to have builtin tools that present a unified view of multiple libraries in calibre. After all separate libraries are *separate*

That was my assumption, and I figured I'd need an external tool.

But you can use the create catalog function in calibre to create a csv of all books in any library and them use whatever tool you like to process that csv.

And that was the piece I needed to know.

If I can get it into a CSV, I can get that into something that presents the view I want.

I essentially have two needs:

A top level unified view of possible multiple libraries. (I don't have them yet, but am likely to.)

A way to see what all is on my tablet based on the metadata Calibre places there when it transfers books, without invoking my eBook viewer.

Create catalog and output to CSV handles first. Suggestions on the second are gratefully accepted.
______
Dennis

[BetterRed]

theducks, on shirt pocket drives.

@David - theducks post reminded of something

I know of a calibre user who has zillions of fan fiction stories and a boatload of custom columns who keeps a 'backup' copy of their calibre library on a portable drive. When they travel they attach the portable drive to their Samsung Tab, and then they access the library via CC's Cloud feature.
As I understand it, CC's Cloud interface uses the calibre library/database directly, its read only of course.

And this maybe the clincher for you, I'm fairly certain they maintain the backup copy with FFS over USB cable!

You may need this plugin ==>> Save Composite Custom Columns
------------------------

I use Chrome for calibre-server because it's not Firefox, which is also my main browser, I'm thinking of switching to Edge or firewalled Midori for calibre-server access.
------------------------

I've never needed (or even wanted) a single top level list. I create CSV's via the Catalogue feature, or via the calibredb --list command. If I selected just common columns I guess I could concatenate the output of multiple calibredb commands into a single file.

I use nirsoft's CSVFileView to view the CSV files (IMO spreadsheets are overkill), it has an export to HTML feature which is useful for sharing with others.
------------------------

When I travelled a lot I preferred shirt pocket drives to thumb drives, they were bigger capacity wise and for me at least less easily lost. As a consequence of always losing them I tend to buy cheap thumb drives which fade like newsprint left in the sun

I have a few 20+ year old SCSI drives that are still readable.
------------------------

There's a search tool called Recoll that runs on Windows and Linux, and there's a Recoll plugin for calibre - not sure what its status is, a peek suggests 'uncertain'. I think Docfetcher also runs on Linux and Windows.

For some document types (but not ebook formats - except PDF) X1 shows results with search terms highlighted, which is the main reason I have it. I keep a searchable copy of everything, most of my 'books' originate as PDF, DOC(X), or ODT documents, but if I 'only' have something else (ePUB or Mobi etc) I convert to TXT or DOCX. Maybe one day someone will write an IFilter for ePUB at least.

BR

[DMcCunney]

Originally Posted by BetterRed

theducks, on shirt pocket drives.

@David - theducks post reminded of something

I know of a calibre user who has zillions of fan fiction stories and a boatload of custom columns who keeps a 'backup' copy of their calibre library on a portable drive. When they travel they attach the portable drive to their Samsung Tab, and then they access the library via CC's Cloud feature.
As I understand it, CC's Cloud interface uses the calibre library/database directly, its read only of course.

And this maybe the clincher for you, I'm fairly certain they maintain the backup copy with FFS over USB cable!

Not going there on the portable drive. For my purposes, a USB stick is fine. If it gets lost or damaged, it's dirt cheap to replace , and performance is more than adequate. I have no current needs that would require higher capacity than USB sticks currently offer.

(Mind you, I can use a portable drive directly with my tablet. I have a USB portable backup drive. I can plug it into a USB hub, and attach the hub via an OTG Adapter to the tablet microUSB port. The tablet is rooted, and I have a driver installed that can mount the portable drive and access the NTFS file system on the drive. I need to use a powered hub because the tablet can't power the drive via USB. It lacks the power to do so. If I plug the drive in without a powered hub, the tablet will see it, but it will go away again.)

You may need this plugin ==>> Save Composite Custom Columns

I just went through an Install Plugins exercise, and I think that's one I installed.

I use Chrome for calibre-server because it's not Firefox, which is also my main browser, I'm thinking of switching to Edge or firewalled Midori for calibre-server access.

Edge? <gack>

I have Midori here, but you might want to look at Qupzilla. It's an open source browser based on Webkit and Qt, and available for a lot of things, including Windows, Linux, OS/X and OS/2.

I've never needed (or even wanted) a single top level list. I create CSV's via the Catalogue feature, or via the calibredb --list command. If I selected just common columns I guess I could concatenate the output of multiple calibredb commands into a single file.

I'm assuming I won't have the same volumes in multiple libraries. So a top level list is a way to see whether I have something at all without having to open each library to look. I can do that by exporting and combining CSV files, and importing the result into a spreadsheet.

I use nirsoft's CSVFileView to view the CSV files (IMO spreadsheets are overkill), it has an export to HTML feature which is useful for sharing with others.

I have an assortment of Nirsoft utilities. I may have that one. Whether a spreadsheet is overkill depends on what you need to do.

When I travelled a lot I preferred shirt pocket drives to thumb drives, they were bigger capacity wise and for me at least less easily lost. As a consequence of always losing them I tend to buy cheap thumb drives which fade like newsprint left in the sun

As mentioned, capacity isn't a prime selector in what I use. I tend not to lose stuff like this. But if I do, it's dirt cheap to replace, especially since it won't be the only copy of the data stored on it.

I have a few 20+ year old SCSI drives that are still readable.

I think I may have one. I no longer have a machine that takes SCSI drives, however.

There's a search tool called Recoll that runs on Windows and Linux, and there's a Recoll plugin for calibre - not sure what its status is, a peek suggests 'uncertain'. I think Docfetcher also runs on Linux and Windows.

I think I've heard of Recoil, but haven't poked at it. And I believe Docfetcher does indeed run under Windows and Linux.

For some document types (but not ebook formats - except PDF) X1 shows results with search terms highlighted, which is the main reason I have it. I keep a searchable copy of everything, most of my 'books' originate as PDF, DOC(X), or ODT documents, but if I 'only' have something else (ePUB or Mobi etc) I convert to TXT or DOCX. Maybe one day someone will write an IFilter for ePUB at least.

I've contemplated doing things like opening the Mobi and ePub files and extracting the HTML to someplace else so that it can be conveniently searched. But it looks like a lot more trouble than it would be worth for my purposes.
______
Dennis

[BetterRed]

If you have/had your calibre library FFS backup on an NTFS formatted memory stick, and you plugged it into the OTG adapter, then I would think you could use Calibre Companion's Cloud (CCC) feature to access it on your tablet? See ==>> What cloud providers does CC support?. Note the last one - local storage.

Edge is part of the furniture, so might as well use it. All I need for calibre server is a simple tabbed browser - Mosaic with tabs would do. Just not sure if MS's use of Edge for their 'apps' might interfere with my use of it for calibre server - hence maybe Midori or similar. But I'm waiting to see what Calibre 3.0 includes before I go any further with replacing Chrome

BR

[kovidgoyal]

Originally Posted by BetterRed

But I'm waiting to see what Calibre 3.0 includes before I go any further with replacing Chrome

Just FYI, the server in calibre 3.0 requires a modern browser. I doubt very much that a Qt WebKit based browser would work (the browser engine in Qt WebKit is several years old).

[DMcCunney]

Originally Posted by kovidgoyal

Just FYI, the server in calibre 3.0 requires a modern browser. I doubt very much that a Qt WebKit based browser would work (the browser engine in Qt WebKit is several years old).

Qupzilla has handled everything I've thrown at it. (In an earlier version, I changed the user agent string it sent to say it was Chrome, and all worked as expected.

I was inaccurate in my earlier post. Qupzilla is now based on QtWebEngine
______
Dennis

[DMcCunney]

Originally Posted by BetterRed

If you have/had your calibre library FFS backup on an NTFS formatted memory stick, and you plugged it into the OTG adapter, then I would think you could use Calibre Companion's Cloud (CCC) feature to access it on your tablet? See ==>> What cloud providers does CC support?. Note the last one - local storage.

I actually have an NTFS formatted memory stick, though not the one the Calibre library is currently backed up to. I'll keep that suggestion on file for later testing, because it is an intriguing notion.

Edge is part of the furniture, so might as well use it. All I need for calibre server is a simple tabbed browser - Mosaic with tabs would do. Just not sure if MS's use of Edge for their 'apps' might interfere with my use of it for calibre server - hence maybe Midori or similar. But I'm waiting to see what Calibre 3.0 includes before I go any further with replacing Chrome

Yeah, I understand the logic. But I've been a user of Mozilla code since it was still an internal Netscape project, and IE has always given me hives. Nothing I've seen of Edge has made me like it better, but I admit I haven't poked at it hard. (And a current annoyance is finding Win10 helpfully resetting my default app choices, like suddenly discovering Edge getting called to open PDFs. Er, no. There's a reason I use SumatraPDF, ThankYouVeryMuch... )
______
Dennis