Headless Calibre Server

[greene48]

I've been trying for the past couple days to set up Calibre-Server on my headless unbuntu 12.04 server.
I've mostly followed this guide: http://madness.codydeschenes.com/201...alibre-server/

The server is up and running alright; I can access it on my laptop by pointing my browser to the headless server's ip, and the correct port for calibre-server.
The problem is that calibre-server can't locate my library. I get the following error: "No books found".
I have set the correct path to my library with the "--with-library=" command, so i am not sure what the issue is.

The problem could be that I've never used Calibre before (previously just dragged and dropped books to my tablet), so I don't have a metadata.bd file.
Does metadata.db need to exist prior to running Calibre-server? Or can calibre-server create this file?
I tried running "calibredb add [library location]" and this did create a metadata.db file, but I still get the same error.
I have no way of running calibre for the desktop, so I need to find an alternate way to create this database file.

Another issue could be folder structure. I have it set up with folders for Authors, then individual folders for books. Inside each book folder is the book file, a cover picture and a metadata.opf file

Any ideas/help would be appreciated.

[aleyx]

Yup, you need a properly populated .db. The server queries the database to locate the actual files.

I use a headless Ubuntu box for the content server as well, but the library is shared over my LAN, and I update the .db using the Calibre GUI on my laptop.

[greene48]

Thanks,
Yea that's what I was thinking.

I have my library shared over my LAN as well, but I'd like to avoid having to manually update the .db from another computer.
At the moment I'm runnning a script which automatically downloads books, organizes them, and generates cover pictures and metadata. If possible I'd like to just be able to update the .db file in the same way.

Also, my laptop is a chromebook pixel, so I can't easily run calibre on it. I could install linux on it (with crouton) but I'd rather not have to do this just to update calibre.

[aleyx]

Originally Posted by greene48

Thanks,
Yea that's what I was thinking.

I have my library shared over my LAN as well, but I'd like to avoid having to manually update the .db from another computer.

Depending on your definition of "manually", I'd really avoid doing it from _any_ computer ^_^

A Calibre .db should only be accessed through the Calibre GUI or through calibredb...

Originally Posted by greene48

At the moment I'm runnning a script which automatically downloads books, organizes them, and generates cover pictures and metadata. If possible I'd like to just be able to update the .db file in the same way.

Can you show us the script? What exactly does it do with the metadata? There's probably a way to adapt it to use calibredb.

Originally Posted by greene48

Also, my laptop is a chromebook pixel, so I can't easily run calibre on it. I could install linux on it (with crouton) but I'd rather not have to do this just to update calibre.

Can't help you with the Chromebook part of it, sorry. As far as consumer OSes go, I've only ever used Windows and Linux. I'm only now experimenting with an old Android tablet I've been given, so ChromeOS is waaay out of my comfort zone...

[greene48]

Well, by "manually" I mean opening up the Calibre GUI to update the .db.
I can't do this since ChromeOS is basically just the chrome browser. So I can't install calibre on it. But I could install linux on my Chrombook to install Calibre if I need to run the Calibre GUI.
But this wouldn't be my ideal solution.
Do you need to open up the Calibre GUI everytime you want to add a new book to your .db?

I oversimplified the way I'm processing my books. The "script" I'm using is actually a python program I have installed on my server called LazyLibrarian: https://github.com/herman-rogers/LazyLibrarian-1
This can be used to downloads books from nzb sites, (by passing them to Sabnzbd+), store them in the correct location, and then process them with cover pictures and metadata.
At the moment, I am only using the "postprocessing.py" script that comes with the program. I used it to process my current book collection, and have it setup to process any new book I move into my processing folder.
That script is located here: https://github.com/herman-rogers/Laz...postprocess.py

If this could be adapted to update the metadata.db file as well it would be ideal. Can you update the metadata.db file from the CLI?

[aleyx]

Originally Posted by greene48

Well, by "manually" I mean opening up the Calibre GUI to update the .db.

That's a relief ^^

Originally Posted by greene48

I can't do this since ChromeOS is basically just the chrome browser. So I can't install calibre on it. But I could install linux on my Chrombook to install Calibre if I need to run the Calibre GUI.
But this wouldn't be my ideal solution.
Do you need to open up the Calibre GUI everytime you want to add a new book to your .db?

No, you can use calibredb, which is a CLI tool.

Originally Posted by greene48

I oversimplified the way I'm processing my books. The "script" I'm using is actually a python program I have installed on my server called LazyLibrarian: https://github.com/herman-rogers/LazyLibrarian-1
This can be used to downloads books from nzb sites, (by passing them to Sabnzbd+), store them in the correct location, and then process them with cover pictures and metadata.
At the moment, I am only using the "postprocessing.py" script that comes with the program. I used it to process my current book collection, and have it setup to process any new book I move into my processing folder.
That script is located here: https://github.com/herman-rogers/Laz...postprocess.py

If this could be adapted to update the metadata.db file as well it would be ideal. Can you update the metadata.db file from the CLI?

So after your postprocessing, what you have is a hierarchy of directories, anf the leaf directories have an ebook, a cover.jpg and an .opf. Is that right? If so, I don't think there's ll be any difficulty in using calibredb to import your books into the Calibre library, and thus the content server.

Be careful not to use the same hierarchy of folders for your Calibre library and your postprocessed repo, tho. In fact, the whole Calibre library should never be directly accessed by anything but Calibre (except for backups, of course. Backups are Good(tm)).

[greene48]

Originally Posted by aleyx

That's a relief ^^
So after your postprocessing, what you have is a hierarchy of directories, anf the leaf directories have an ebook, a cover.jpg and an .opf. Is that right?

Yes that's correct ^^

Originally Posted by aleyx

Be careful not to use the same hierarchy of folders for your Calibre library and your postprocessed repo, tho. In fact, the whole Calibre library should never be directly accessed by anything but Calibre (except for backups, of course. Backups are Good(tm)).

Not sure what you mean here.
Do you mean I should have one folder hierarchy with all my postprocessed books, cover files, and .opf files, and then a separate folder hierarchy for my actual Calibre library?
So this means I need to have all my books in 2 locations? And somehow keep the 2 folders in sync?

Thanks again for the help.

[aleyx]

Originally Posted by greene48

Not sure what you mean here.
Do you mean I should have one folder hierarchy with all my postprocessed books, cover files, and .opf files, and then a separate folder hierarchy for my actual Calibre library?
So this means I need to have all my books in 2 locations? And somehow keep the 2 folders in sync?

Not really. Calibre's whole point is book management; it's therefore meant to be a main repo (database + filesystem), accessed through its various interfaces (GUI, CLI, content server).

Once you include Calibre into your process, your previous repo's books should be discarded once imported in Calibre (that's a manual operation, Calibre won't do it for you).

You can, of course, keep your current directory structure. Once your scripts populate them again, you can rerun the import and Calibre will update itself with the new files, adding or updating the books when needed. Then you can clear up your repo again.

Note that if your books are dynamically generated periodicals (from news sites for exemple), Calibre already has its own mechanism to fetch them and keep them current. But I don't use it, so I don't know how feasible it is to implement it on a headless server. Those In The Know may be able to answer that.

Originally Posted by greene48

Thanks again for the help.

You're welcome.

[greene48]

Originally Posted by aleyx

Not really. Calibre's whole point is book management; it's therefore meant to be a main repo (database + filesystem), accessed through its various interfaces (GUI, CLI, content server).

Once you include Calibre into your process, your previous repo's books should be discarded once imported in Calibre (that's a manual operation, Calibre won't do it for you).

You can, of course, keep your current directory structure. Once your scripts populate them again, you can rerun the import and Calibre will update itself with the new files, adding or updating the books when needed. Then you can clear up your repo again.

Ok, understood.
I've been reading about the Calibre library structure, as well as the calibredb CLI.

Sounds like I may have to install Calibre Gui to get my inital library created. But once I have it created I should be able to add a section to my "postprocess.py" script that will run a "calibredb --recurse /my/processed/folder/path --library-path=/my/library/path".
That should add new books to my library.

Also, I may be able to generate an initial library through the CLI by running "calibredb restore_database --really-do-it".
This should "restore" (create in my case) a database from the metadata stored in my OPF files.

I'll give this a shot.

[aleyx]

At worst you could copy a base/settings from an existing setup on another computer.

Heck, I could do that myself for you if you don't have a Linux setup nearby: create a new user, run Calibre, create an empty library, close Calibre, and put the .calibre dir and the library structure in a .tar.gz here. Won't take but a few minutes.

[greene48]

Thanks, that seems to have worked to build my library.
I extracted your blank library to a different location on my server, then ran "calibredb add --recurse /path/to/my/books --library-path=/path/to/library".

Seems like it worked. I received the messages "Added book ID's ...." and "Notifying calibre of the change". And when I check my new library folder, my books have been added.

So that problem is solved. But now when I try to access Calibre Server through the web interface I get a "500 Internal server error" message page.
So I must have broke something else somewhere along the way. I'll have to go back through my setup to see if I can figure it out.

[greene48]

Got it working now. I think I might have had 2 instances of Calibre Server trying to run on the same port. A reboot of my server solved it.
Calibre-Server is accessible through my browser, and it's displaying all my books.

Only thing left to do is to try to build the "calibredb add" command into my postprocess.py file.

Thanks again!

[Adoby]

One nice way to set it up could be to have three copies of your calibre library.

Copy A is the main copy that you add books to and fiddle with metadata. You use it from the GUI as normal.

Copy B is a backup copy of A. You can use a sync utility to update it from A, as needed. Or automatically every time you quit calibre, by editing the script that is used to launch calibre. Preferably B is on a network share on a NAS or some other computer that is on 24/7.

Using a sync utility makes it extremely fast to sync any changes.

Copy C is a copy of B. It is also used by the calibre content server. It can be updated regularly by a sync task that is run at predetermined times by the OS, or manually by restarting the calibre server. The sync from B to C could be run every time the calibre server is started, in the script.

By having the extra B copy you can start and stop the main GUI and the calibre server independently, without having to be careful about not syncing and changing the calibre library when another instance of calibre is running. Also you get a nice backup system. Could extend it to keep more versions.

[aleyx]

A "killall calibre-server" is all that's needed to stop the content server. It's read-only on the DB, so no harm done.

I use a simple calibre-server.sh at boot:

#!/bin/bash
calibre-server --port 8081 --restriction FanFiction --daemonize
calibre-server --port 8080 --restriction Novels --daemonize

Which give me two content server instances on two ports, each giving me one of my virtual libraries. Still haven't gotten around making it an init.d service script... Ah well. I'm lazy that way.

Then I have calibre-update.sh to, well, update Calibre when a new version is out:

#!/bin/bash
killall calibre-server
sudo python -c "import sys; py3 = sys.version_info[0] > 2; u = __import__('urllib.request' if py3 else 'urllib', fromlist=1); exec(u.urlopen('http://status.calibre-ebook.com/linux_installer').read()); main(install_dir='/opt')"
$HOME/bin/calibre-server.sh

I haven't yet found (or looked at, in all honesty) a way to automatically check if a new version is available and run calibre-update.sh accordingly in a cron. The installer at http://status.calibre-ebook.com/linux_installer doesn't check the local version of Calibre, so it downloads and install even if it's not necessary...

One nice way to set it up could be to have three copies of your calibre library.

Hm... I don't really see it. I mean, the backup copy is a given, of course, and should be a part of your global backup policy anyways. But I don't see the use for (C), as the content server automatically picks up changes to the Calibre library.

My current library (~1400 books, many of them having several formats) is currently 971MB. I'm not hurting for disk space, but still...

Edit: I could see it if the Calibre doing the updates lived on one computer, the content server on another, and you didn't want any of them to work on a share. But that's not the case here; there's only one server and it's headless.

[Adoby]

I think that it would be difficult to only have calibre as a headless server. Easier or even necessary to also have calibre with the normal GUI so you can add books, convert and update metadata and so on. So to me, having a headless server more or less means that you also have to have a normal calibre install with the GUI on another computer. And somehow copy, share (not recommended) or sync between the two. If A and C are on local filsystems, then B must be on a network share.