How to Rebuild a 20,000+ Calibre Companion Library on an Android Device

[jvorzimmer]

On a recent trip to Greece and Turkey I accidently deleted my 20,000+ Calibre Companion ebook library. I won’t go into the details about A) Why I have that many ebooks on a tablet (I create ebooks for a living; is the short answer) or B) How I accidently deleted my library. That is, other than to say, that COVID-19 was partially to blame. I had to free up space to install various apps to prove I was vaccinated AND recently tested. And had to load large PDFs of my test results on to my tablet. One such app was veriFLY and another the Viking app (The trip was a cruise around the Aegean Sea).

I want to make it clear that I didn’t actually lose the ebooks themselves since, luckily, they were stored on an external SD card. But Calibre Companion was pretty much unusable since my backup was two years old (I will be much more diligent about backing it up going forward).

When I got home I thought I would take the opportunity to rebuild the entire Calibre Companion library on my android tablet (A Samsung Tab A 8” running Android Pie (9)). When I first started using CC in December 2012 I never anticipated my library growing from under a 1,000 books to over 20,000 (95Gb), so I was using the default template in CC, which puts all the ebooks in a single directory. Somewhere around the time I reached 18,000 books I noticed a tremendous slowdown on transferring books to the tablet. I thought I would at least rebuild it more efficiently, with subdirectories for each author, which would speed things up. What would even be faster would be to have folders for each letter of the alphabet and author directories within those of authors whose last names start with that letter.

I pondered different ways of rebuilding my CC library, such as moving my entire Calibre library to a folder on the external SD card and using CC’s option to use another device or SD card as a cloud library. The downside of this method is the obvious one of needing at least enough free space roughly twice the size of your library and the fact that it would take days to move, since CC would still have to move every ebook as well as its accompanying metadata. The upside of it is there would be no connection issues as as there would be with using the Calibre content server or a cloud service.

In the end what worked was surprisingly simple (only in retrospect, I must add). I exported my entire Calibre library using Calibre’s “Save to Disk” feature using a template that would match CC’s own template for saving the directory structure. I got some help from help from capink and chaley on this. I must also add that I’m running Calibre 4.23 on a Windows 7 media server. I was unaware that the template language had changed with version 5 of Calibre.

The “Save to Disk” template I used was:
{authors:'re(sublist($, 0, 1, '&'), '\.$', '')'}/{title:re([—®™],_)} - {authors:re(\.$,)}
To match CC’s template:
{first_author}/{title} – {authors}

There are some replacements in the Calibre template where certain special characters are handled differently by the two applications. I also made sure use the above template in CC’s settings: File names > Use CC template everywhere. That was key to making this work.

It took about two and a half hours to export the books out of Calibre. I then moved this to the directory of books on my tablet’s 256Mb SDXC card inserted into my computer’s USB 3 port using a USB 3 card reader. This took about an hour and a half. I then inserted the SDXC card back into my android tablet and made sure CC was looing for books in the directory I had created.

I then connected to my Calibre library’s content server to push over the metadata for the books. I then made sure to select “Automatic management” in “Metadata management” under Calibre’s “Sending Books to devices” preferences. It doesn’t matter what Calibre’s template is here in these preferences since CC would be using its own template.

It then took about four hours to get the metadata pushed over to Calibre Companion. As long as Calibre finds the book on the device, it won’t send the file, just the metadata (At least I believe it was Calibre and not the CC app that makes that call. Correct me, anybody, if I’m wrong on this.)

After what seemed a painfully long time, all 20,000+ books showed up in my Calibre companion library. I checked to make sure that CC could match all the metadata to files by running the “Books with missing files” menu item in CC. Turns out that only 11 books couldn’t be matched to files. It turns out that those 11 books had single typographical quotes instead of a single straight quote somewhere in the titles. Since I was still connected to the Calibre server, I replaced the typographical quotes with straight quotes so that the file names would be handled correctly by CC. I then updated the cached metada and bada bing, bada boom, CC found the files. All 20,000+ books are back in Calibre Companion.

If you’ve read this far and you don’t have a library the size of mine, I would warn you that’s a pain in the ass to maintain. For one, you have to have a tablet to which you can add fairly large external SD cards (that eliminates iPads) and you have to have a hardware/android combination that allows you to use the external SD card as internal storage for apps. Before Android 5, I had to root my Android device and might have to again in the future as Google keeps adding restrictions to the use of external SD cards by apps.

Anyway, I hope you found this informative. Of course, I can’t guarantee that anyone else who tries this method will have the same success I had. There are just too many variations of hardware and Android versions for a single solution for all.

[chaley]

Moderator Notice
Moved to CC forum