Can Calibre Index books stored locally and remotely?

[leecreighton]

Hi all,

New poster here, who assures you that he's searched the forums, read the FAQ, and discovered that the link to the intro tutorial in the Sticky Post is dead.

My situation:
I have about 5000 PDF books (with the odd epub or mobi book) stored on DropBox, as well as a hundred or so on pCloud, a DropBox competitor that can be accessed over WebDAV. I also have about 2000 Kindle ebooks that are not generally stored anywhere I have access to. They live with Amazon and are downloaded as needed.

I'd like to use Calibre as the master indexer, the librarian, if you will. What I can't figure out is whether calibre will be happy leaving the books where they are. None of my computers, especially my daily-use MacBook Pro, has enough space to store all my books. And as you can guess, I have a hell of a time remembering if I own a book, and where it is.

Calibre seems to be the best solution as long as it doesn't require local copies of the books. I assume initially I just add the respective folders to calibre and it will augment its index as needed. I'd appreciate any advice on how to keep calibre updated as I add files to my various places.

Addtional questions, if they're answerable, otherwise I'l just have to try:
How large do you think a book index of that size will be?
I also own over a thousand physical books. Should I let calibre manage them as well?

[ilovejedd]

Calibre doesn't delete your original files but it creates a copy of all imported books to its local library folder so I don't think it'll work for your scenario as-is.

You can, however, use something like the Import List plugin to batch add books without formats/files if you just want to use Calibre for cataloging.

[kovidgoyal]

calibre the main program requires local copies of books -- setup the calibre content server and you can access those books anywhere on any device, without needing local copies.

[leecreighton]

Although it may not seem like it, my Mac thinks that Dropbox and pCloud are local. I don't use the apps that, say, Dropbox furnishes, precisely because it requires local copies of everything. Why would I buy 1 TB (which is the lowest-tier paid plan) of space if everything I put there has to also be on my machine?

Instead, I use something called CloudMounter, which mounts Dropbox, Google Drive, pCloud, Box drive, or any remote drive while making the Mac think it's just any-old network drive. No copies needed. Dropbox and the others therefore function as extra, non-redundant storage. So I wonder if calibre would be able to work with that.

I had no idea Calibre had a server version, but that sounds exactly like the complication I was trying to avoid. I'll look, but what I'm hearing is that I can run the server on my local machine, and it will index files on multiple drives. I then...wait...do I then run Calibre as well, to access the server that's on the same machine?

[kovidgoyal]

the server is part of calibre, all you need is an always on, internet connected machine. run calibre and its server on that machine and access your library on any other machine, anywhere using a simple browser.

[leecreighton]

That's one of the problems I was trying to get around. I've got one computer, the MacBook Pro I'm using now.

[kacir]

Originally Posted by leecreighton

Although it may not seem like it, my Mac thinks that Dropbox and pCloud are local...

Instead, I use something called CloudMounter, which mounts Dropbox, Google Drive, pCloud, Box drive, or any remote drive while making the Mac think it's just any-old network drive. No copies needed. Dropbox and the others therefore function as extra, non-redundant storage. So I wonder if calibre would be able to work with that.

Calibre works like that:
It creates a "Calibre Library" folder on a local disk on a computer and when you import the book to Calibre it makes a local copy in an elaborate file structure, for example ~/Calibre Library/Author Name/Book name [index number]/ and keep all book related files for that single book in that directory. There will be book files (various formats, such as mobi, epub), a backup file with book metadata, cover image.
You can not change this structure and the author of Calibre has posted repeatedly that he will change this philosophy.
The only thing you could do is to create an "empty" book record, that will contain just a short text file with the address where your book is located instead of the actual book file.

As long as the operating system things that your remote cloud file repository is a local disk you can use Calibre to create a "Calibre Library" directory anywhere. It will still want to make a copy of the book for its library directory.

Please note, the strength of Calibre is that it manages your folder structure, so you do not have to. You query the database, work with your books and then use Calibre to copy selected books to your reading device or an export folder. Another strength of Calibre is that when you want to import a book you somply drag and drop it (or even hunreds of books at the time) and it will read the metadata from book files or filename according to your template and get the metadata for you. When you are not satisified with the metadata or a cover, you simply hit Ctrl+D and Calibre will search the net for a better cover or a blurb.

Also note, that many people have had problems in the past when they tried using dropbox or other cloud repository for the library. It works for a while and then you notice that books are missing, or your library is corrupted. Because Calibre thinks it has the "Calibre Library" directory on a local disk and treats it as such.

Originally Posted by leecreighton

I had no idea Calibre had a server version, but that sounds exactly like the complication I was trying to avoid. I'll look, but what I'm hearing is that I can run the server on my local machine, and it will index files on multiple drives. I then...wait...do I then run Calibre as well, to access the server that's on the same machine?

Calibre doesn't have a server version. It is the same Calibre, running on your PC or other designated computer with its library directory (on a local disk or a network disk (not recommended) or a cloud (*strongly* not recommended)). You can go then to the GUI and go to the Preferences, Sharing, Sharing over the net, and start the server. It will make your library accessible via http to the other computers on a local network. So you can then access your library from other devices through a browser.

There is also a portable version of Calibre that you can put on an external disk and maintain the library on that disk. That might be the path of least resistance for you. Please note that a typical book is under 1 MB, so a library with 2000 [Amazon] books might be 10GB, with a very generous margin of error. As long as the books are not scanned pdfs, or heavily illustrated or perhaps comics. You can buy 1TB external hard-disk for under 50 bucks on Amazon. You can buy 32GB USB flash disk for under 10 dollars.

[vikarti]

Originally Posted by ilovejedd

Dropbox doesn't delete your original files but it creates a copy of all imported books to its local library folder so I don't think it'll work for your scenario as-is.

This is not strictly true.
Dropbox Professional supports SmartSync (aka 'only keep files online but emulate their presence locally') https://www.dropbox.com/en/help/desktop-web/smart-sync but it's not very cheap and it's open question how calibre will react to delays in access.

[BetterRed]

Originally Posted by kacir

There is also a portable version of Calibre that you can put on an external disk and maintain the library on that disk. That might be the path of least resistance for you. Please note that a typical book is under 1 MB, so a library with 2000 [Amazon] books might be 10GB, with a very generous margin of error. As long as the books are not scanned pdfs, or heavily illustrated or perhaps comics. You can buy 1TB external hard-disk for under 50 bucks on Amazon. You can buy 32GB USB flash disk for under 10 dollars.

The OP is on a MacOS system, the portable version of calibre is a repackaged Windows 32 bit install.

That said if the OPs Macbook Pro has a USB port (preferably 3.x), he could keep a consolidated library on an external drive. and if 1Tb isn't enough a 4TB WD drive can be had for ~$100. Periodic incremental backups could be done to the cloud using a file sync tool, I do it every day, so it only takes a few minutes.

BR

[ilovejedd]

Originally Posted by vikarti

This is not strictly true.
Dropbox Professional supports SmartSync (aka 'only keep files online but emulate their presence locally') https://www.dropbox.com/en/help/desktop-web/smart-sync but it's not very cheap and it's open question how calibre will react to delays in access.

Whoops, I meant Calibre, not Dropbox.