Headless server with non-default DB location

[paperrhino]

I'm experimenting with setting up a headless calibre-server on Ubuntu Server 17.4. I've come a long way but am facing a bit of a problem.

I've got it running as a service but my books are on a file share so I've set the CALIBRE_OVERRIDE_DATABASE_PATH=/home/calibre/metadata.db environment variable so it saves the DB not on the CIFS mounted file system per the warnings in the FAQ.

I've even managed to install and configure one plugin from the command line. so far so good.

My problem is with adding books. When I try to run the following command (with the override DB environment variable correctly set) calibre creates a new metadata.db in the library folder rather than editing the metadata.db that calibre-server is actually using.

xvfb-run calibredb add /mnt/calibre/toadd/ -r --with-library /mnt/calibre/library

When I list the contents of the newly created /mnt/calibre/library/metadata.db I see only the new books. When I list the contents of the "real" metadata.db I get all the old books and none of the new ones.

Is alternative DB locations not supported by calibredb or am I doing something wrong?

Many thanks.

[gbm]

Originally Posted by paperrhino

I'm experimenting with setting up a headless calibre-server on Ubuntu Server 17.4. I've come a long way but am facing a bit of a problem.

I've got it running as a service but my books are on a file share so I've set the CALIBRE_OVERRIDE_DATABASE_PATH=/home/calibre/metadata.db environment variable so it saves the DB not on the CIFS mounted file system per the warnings in the FAQ.

I've even managed to install and configure one plugin from the command line. so far so good.

My problem is with adding books. When I try to run the following command (with the override DB environment variable correctly set) calibre creates a new metadata.db in the library folder rather than editing the metadata.db that calibre-server is actually using.

xvfb-run calibredb add /mnt/calibre/toadd/ -r --with-library /mnt/calibre/library

When I list the contents of the newly created /mnt/calibre/library/metadata.db I see only the new books. When I list the contents of the "real" metadata.db I get all the old books and none of the new ones.

Is alternative DB locations not supported by calibredb or am I doing something wrong?

Many thanks.

I think you need the share UUID ex:

/mnt/1589cd3b-4281-4652-af95-1aed1a229bcc

Look in /opt/calibre/resources/ at the contents of calibre-portable.sh.

Edit

Verily the path and that they are mounted of your share using a terminal and the cd and l commands.

The only device I have that does not mount with a UUID is my NOOK.
bernie

[paperrhino]

Thank you for your reply. I'm not certain I fully understand what you are asking me to do.

I've looked at calibre-portable.sh and I see where it creates a conf file and in there I see where it sets the CALIBRE_OVERRIDE_DATABASE_PATH which looks a lot like what I'm already doing.

My share is a cifs mounted share so I don't think it has a uuid beyond the //machine/dir path.

And I don't understand how that would help. I wanted to have my books stored in one directory and my metadata.db stored in another. The problem wasn't that calibredb couldn't find the books library folder, it was that it was ignoring the CALIBRE_OVERRIDE_DATABASE_PATH environment variable.

In any case, I determined the big problem with having the library on a network share is trying to run multiple instances of Calibre on the same library. Since I'm only ever going to have this one server putting the library on a network share has minimal risk of corruption for me. So I'm just using the default location for metadata.db for now and it is working just fine.

Now, as I purchase eBooks I can drop them into a toadd folder and a cron job will automatically add them to the Calibre library. Then I can browse and download books through the web or apps like Calibre Companion.

Thanks again for the help.

[gbm]

Looks to me like you could use calibre-portable.sh to do what you need in setting the variables and changing the line"

$CALIBRE --with-library "${CALIBRE_LIBRARY_DIRECTORY}"

to

$CALIBRE-SERVER --with-library "${CALIBRE_LIBRARY_DIRECTORY}"

You have it working that all that matters.
bernie

[paperrhino]

I did review the script and I am doing the exact same thing as the script. I have the CALIBRE_OVERRIDE_DATABASE_PATH environment variable set just as it does in that script and I pass the path to the books using --with-library.

My metadata.db file is not located in the default location, hence my setting of the override environment variable.

calibre-server runs just fine. It uses the metatadata.db file I indicate in the override environment variable and it puts the book files in the directory I indicate in the --with-library flag.

The problem is calibredb seems to ignore the override environment variable. Therefore I cannot add books to my library from the command line when metadata.db is not in the default location.

Thanks for trying to help. I just wanted to make is clear what the problem was in case someone else has the same problem in the future.

[paperrhino]

Originally Posted by gbm

Looks to me like you could use calibre-portable.sh to do what you need in setting the variables and changing the line"

$CALIBRE --with-library "${CALIBRE_LIBRARY_DIRECTORY}"

to

$CALIBRE-SERVER --with-library "${CALIBRE_LIBRARY_DIRECTORY}"

You have it working that all that matters.
bernie

But I already am seeing the environment variables like in the Shell script. And I'm calling calibre-server just like in the Shell script. All of that works.

The problem is when I call the separate program calibredb, this program ignores the environment variables. And since the portable script doesn't mention calibredb at all it is no help.

While I do have it working, it would be nice to know if calibredb is capable of supporting a non-default location for the db file. If calibredb, and calibre-server support this it seems to reason that calibredb should as well.

[Krazykiwi]

This is probably stating the obvious, but it's ridiculously easy to set a bunch of env vars, and then put yourself outside the context you've set them in. Like logging in as another user who doesn't have it set in the first place, su'ing without the -, or forgetting to export the vars in a script.

Since it's headless, you're remote logging in, and I assume not directly as the user the service is running as.

Does wrapping calibredb in say, a calibredb-wrapper script that does nothing but set and export the vars you need, and runs calibredb work?

[paperrhino]

This is probably stating the obvious, but it's ridiculously easy to set a bunch of env vars, and then put yourself outside the context you've set them in. Like logging in as another user who doesn't have it set in the first place, su'ing without the -, or forgetting to export the vars in a script.

Indeed but as part of my debugging I've echoed the environment variable immediately before running calibredb to verify that the variable is set.

Since it's headless, you're remote logging in, and I assume not directly as the user the service is running as.

That is correct, however, I have also tried it running as the calibre user using:

sudo -u calibre
su - calibre
sudo -u calibre /bin/bash

In each case I've verified that the environment variable is set before running calibredb.

Does wrapping calibredb in say, a calibredb-wrapper script that does nothing but set and export the vars you need, and runs calibredb work?

I've done this as well.

In every case, rather than using the metadada.db file that is in /home/calibre and defined in the environment variable it creates a new one in /mnt/books/library.

Has anyone else actually done this? It seems unlikely that I'm the first person to try. Either calibredb doesn't pay attention to the environment variable or I'm doing something stupid.

[Krazykiwi]

Originally Posted by paperrhino

Indeed but as part of my debugging I've echoed the environment variable immediately before running calibredb to verify that the variable is set.

I figured, but it's always worth asking even the obvious questions when troubleshooting. And at least that's ruled out entirely.

This isn't something I've done before, but I'm watching with interest because it's something I had been planning to do at some point and just hadn't got around to.

Based on your description, and the debugging you've already done, I'm inclined to agree with you: calibredb is not taking any notice of the environment variables. I'd file a bug at this point.

[kovidgoyal]

I fond that highly unlikely. The code that uses CALIBRE_OVERRIDE_DATABASE is in the database backend. All calibre utilities use that same code.

I'm guessing you only set, not exported the env var. Or xvfb-run is stripping the the env var. Incidentally why are you using xvfb-run? calibre does not need it, it uses a headless QPA backend, which was introduced several years ago, IIRC.

[paperrhino]

Originally Posted by kovidgoyal

I fond that highly unlikely. The code that uses CALIBRE_OVERRIDE_DATABASE is in the database backend. All calibre utilities use that same code.

I'm guessing you only set, not exported the env var. Or xvfb-run is stripping the the env var. Incidentally why are you using xvfb-run? calibre does not need it, it uses a headless QPA backend, which was introduced several years ago, IIRC.

The only tutorial I found that showed how to set up a headless server that worked for me used it. When I tried to run calibredb without using it I would get errors.

[kovidgoyal]

Really, what errors?