What's your headless setup & workflow?

elPedr0 · 11-01-2022, 09:47 AM

I'm getting to grips with my family's ebooks - previously stuck on our individual readers.

I've got calibre-server running on a linux server and calibre-web on the same box providing a friendly, well-featured interface for each of the users to browse books and send them to their devices. Everything's central and each user can log on from any terminal or mobile device without installing or configuring anything - so far so good.

However, there are some things in my workflow (as the general family librarian) that I still need to optimise and would love some tips. Two questions have already arisen:

1) When populating the library by importing books from an ereader, I often want to take advantage of plugins that would be available within the calibre GUI, but that don't necessarily support a headless setup. Those of you with a headless setup - do you use a separate isolated instance of calibre on a desktop to prepare the ebooks to your satisfaction and then 'calibredb add' the modified ebooks into the headless calibre database?

2) Our collection has a mix of formats mostly AZW3 and MOBI but also AZW, EPUB, and PRC. Do you tend to convert everything to a single format when you intially import, or do you just leave each file in its original format and convert each time a user sends it to a reader (if necessary)?

Answers to these questions and any other headless tips are welcome

elPedr0 · 11-04-2022, 10:23 AM

Here's an update on my experience so far, perhaps it will be useful for people who have similar plans.

AIMS:
* I want my Calibre library to be always available and not at risk of being turned off by a user.
* I want a number of different users to be able to be able to access the calibre library using an interface designed for calibre, but without them having to install any specific software.
* I want my users to be able to access the library from any device and retain any of their personal settings.
* I want a single admin user (me) to be able to manage the library to keep metadata clean, remove duplicates, etc.

CONSEQUENTIAL ARCHITECTURE:
* Install calibre on a headless server at home, which is in any case always on (so won't represent additional material energy consumption) and is not in danger of being turned off by a user (so calibre is more likely to be always available). This will provide calibre-web with the tools and APIs it needs to serve the library to the users.
* Install calibre-web on the same headless server to provide the users with a great front-end via a browser.
* Optional: Use Nginix to provide an easy url (without port numbers) for my users to access calibre-web.

LIMITATION:
With the setup described above, my ability to manage the library is very limited. Having played with the calibre GUI on a separate machine and an isolated copy of my library, I now realise the power of tools provided by calibre and third-party plugins to clean and maintain my library. Calibre provides some control through its calibredb command (and others), but plugins do not necessarily provide CLI control and in anycase the presentation of the data in the GUI can be really useful when cleaning and maintaining your library. However, my calibre installation is on a headless server, so I don't have a desktop environment and can't access the GUI.

EXPERIMENTAL and therefore not supported usage:
Because I am now hooked on the GUI, I am planning to install Calibre on a desktop, and point that installation to the same library that is being managed by the server. This is not recommended practice because it can lead to data loss if one library (i.e. metadata.db file) is being managed by two instances of calibre. My workaround will be to ensure that I shut the headless calibre server down before I open the calibre application on the desktop (and vice versa). That way only one calibre server will be interacting with the library at any one time.
Downsides:
* The only protection I'll have from my own idiocy (e.g. forgetting to shutdown the server) will be regular backups.
* Users won't be able to access calibre-web while I'm doing library maintenance stuff with the desktop calibre.
Upsides:
* I will use the desktop version of calibre to add books to the library and ensure consistent, clean metadata.

ALTERNATIVE OPTIONS instead of running an additional calibre on a desktop:
1) use a 'calibredb add' cron job to monitor and then automatically add to the library any books dropped into a designated directory.
This was working well, before I realised that my metadata was starting to get a bit messy - for example creating two authors J.R.R. Tolkein and J. R. R. Tolkein (with spaces and without). If that doesn't really matter to you, then creating a book black hole like this is actually a really convenient option.

2) Install some sort of desktop environment on my server and then remote into that whenever I want access to the calibre GUI.
I don't really see too much of a problem with this, but don't have much experience of putting desktop environments on my servers and so didn't want to go to the hassle of getting it right.

I'll try to remember to report back if everything gets hosed, to warn you not to follow in my footsteps!.

kovidgoyal · 11-04-2022, 10:38 AM

Just run calibre on the main library on your desktop. Rsync the library to your headless server periodically via cron or similar. If none of the user accounts you create for the server have write permissions, then doing so is mostly safe. I am speaking of the calibre content server here, not calibre-web, I have no idea what semantics calibre-web has.

The only thing you will lose this way is the ability for your users to have persistent annotations made via the in browser viewer. Since those are stored in the db and rsyncing will clobber them.

kovidgoyal · 11-04-2022, 10:42 AM

Though if I were you, I would just run the calibre content server on your headless server, keep an intake library on the desktop where books are initially added and metadata cleaned up/converted and then add the book to the headless library by whatever method you prefer, auto-add folder, calibredb script, etc.

And if in the future you need to make small changes you can do so via the server itself which allows for editing metadata, converting, etc.

For larger changes you can shut down the server, copy the library to your desktop run the GUI on it, and then copy it back.

11-01-2022, 09:47 AM	#1
elPedr0 Member Posts: 13 Karma: 10 Join Date: Oct 2022 Device: none	What's your headless setup & workflow? I'm getting to grips with my family's ebooks - previously stuck on our individual readers. I've got calibre-server running on a linux server and calibre-web on the same box providing a friendly, well-featured interface for each of the users to browse books and send them to their devices. Everything's central and each user can log on from any terminal or mobile device without installing or configuring anything - so far so good. However, there are some things in my workflow (as the general family librarian) that I still need to optimise and would love some tips. Two questions have already arisen: 1) When populating the library by importing books from an ereader, I often want to take advantage of plugins that would be available within the calibre GUI, but that don't necessarily support a headless setup. Those of you with a headless setup - do you use a separate isolated instance of calibre on a desktop to prepare the ebooks to your satisfaction and then 'calibredb add' the modified ebooks into the headless calibre database? 2) Our collection has a mix of formats mostly AZW3 and MOBI but also AZW, EPUB, and PRC. Do you tend to convert everything to a single format when you intially import, or do you just leave each file in its original format and convert each time a user sends it to a reader (if necessary)? Answers to these questions and any other headless tips are welcome

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Touch setup playing up after factory reset, can't progress beyond Connected & Charging	kobo_sucks	Kobo Reader	7	10-26-2013 01:02 PM
Edge & Calibre workflow	fekhner	enTourage eDGe	4	06-21-2012 01:02 PM
Headless Calibre Server Setup	godzilla8nj	Related Tools	4	03-10-2010 01:39 PM
Opinion on workflow (and enhancing it) - research-type workflow	TheDarkTrumpet	Which one should I buy?	8	03-02-2009 10:41 AM
Problem & fix to Windows dev env setup	ShellShock	Calibre	6	12-07-2008 10:13 AM

11-04-2022, 10:23 AM	#2
elPedr0 Member Posts: 13 Karma: 10 Join Date: Oct 2022 Device: none	Here's an update on my experience so far, perhaps it will be useful for people who have similar plans. AIMS: * I want my Calibre library to be always available and not at risk of being turned off by a user. * I want a number of different users to be able to be able to access the calibre library using an interface designed for calibre, but without them having to install any specific software. * I want my users to be able to access the library from any device and retain any of their personal settings. * I want a single admin user (me) to be able to manage the library to keep metadata clean, remove duplicates, etc. CONSEQUENTIAL ARCHITECTURE: * Install calibre on a headless server at home, which is in any case always on (so won't represent additional material energy consumption) and is not in danger of being turned off by a user (so calibre is more likely to be always available). This will provide calibre-web with the tools and APIs it needs to serve the library to the users. * Install calibre-web on the same headless server to provide the users with a great front-end via a browser. * Optional: Use Nginix to provide an easy url (without port numbers) for my users to access calibre-web. LIMITATION: With the setup described above, my ability to manage the library is very limited. Having played with the calibre GUI on a separate machine and an isolated copy of my library, I now realise the power of tools provided by calibre and third-party plugins to clean and maintain my library. Calibre provides some control through its calibredb command (and others), but plugins do not necessarily provide CLI control and in anycase the presentation of the data in the GUI can be really useful when cleaning and maintaining your library. However, my calibre installation is on a headless server, so I don't have a desktop environment and can't access the GUI. EXPERIMENTAL and therefore not supported usage: Because I am now hooked on the GUI, I am planning to install Calibre on a desktop, and point that installation to the same library that is being managed by the server. This is not recommended practice because it can lead to data loss if one library (i.e. metadata.db file) is being managed by two instances of calibre. My workaround will be to ensure that I shut the headless calibre server down before I open the calibre application on the desktop (and vice versa). That way only one calibre server will be interacting with the library at any one time. Downsides: * The only protection I'll have from my own idiocy (e.g. forgetting to shutdown the server) will be regular backups. * Users won't be able to access calibre-web while I'm doing library maintenance stuff with the desktop calibre. Upsides: * I will use the desktop version of calibre to add books to the library and ensure consistent, clean metadata. ALTERNATIVE OPTIONS instead of running an additional calibre on a desktop: 1) use a 'calibredb add' cron job to monitor and then automatically add to the library any books dropped into a designated directory. This was working well, before I realised that my metadata was starting to get a bit messy - for example creating two authors J.R.R. Tolkein and J. R. R. Tolkein (with spaces and without). If that doesn't really matter to you, then creating a book black hole like this is actually a really convenient option. 2) Install some sort of desktop environment on my server and then remote into that whenever I want access to the calibre GUI. I don't really see too much of a problem with this, but don't have much experience of putting desktop environments on my servers and so didn't want to go to the hassle of getting it right. I'll try to remember to report back if everything gets hosed, to warn you not to follow in my footsteps!.

11-04-2022, 10:38 AM	#3
kovidgoyal creator of calibre Posts: 45,347 Karma: 27182818 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Just run calibre on the main library on your desktop. Rsync the library to your headless server periodically via cron or similar. If none of the user accounts you create for the server have write permissions, then doing so is mostly safe. I am speaking of the calibre content server here, not calibre-web, I have no idea what semantics calibre-web has. The only thing you will lose this way is the ability for your users to have persistent annotations made via the in browser viewer. Since those are stored in the db and rsyncing will clobber them.

11-04-2022, 10:42 AM	#4
kovidgoyal creator of calibre Posts: 45,347 Karma: 27182818 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Though if I were you, I would just run the calibre content server on your headless server, keep an intake library on the desktop where books are initially added and metadata cleaned up/converted and then add the book to the headless library by whatever method you prefer, auto-add folder, calibredb script, etc. And if in the future you need to make small changes you can do so via the server itself which allows for editing metadata, converting, etc. For larger changes you can shut down the server, copy the library to your desktop run the GUI on it, and then copy it back.

Advert