Calibre content-server doesn't seem to see new books

[offby1]

I keep my ebook library in Dropbox, and on one of my machines I have a launchd daemon that runs the calibre content server:

calibre-server --with-library=$HOME/Dropbox/ebooks

I can confirm that the server serves my library, but it does not appear to detect changes to the library that happen on another system; I mainly run Calibre on another machine and manipulate the library there, relying on Dropbox to sync it.

I can tell (using calibredb list) that my newly added books are just not appearing. I've killed and restarted the calibre server, reloaded the launchd daemon, really everything I can think of, and no good has come of it.

How can I make the content server pick up these changes?

[chaley]

Are you saying that the books do not appear if you restart the content server, but are there if you run calibredb with the same --with-library argument?

[offby1]

No, I am not seeing them either way. The only places I can see them are A) on the filesystem of the machine that is hosting the content server, and B) in the Calibre instance running on my other computer.

[DoctorOhh]

Originally Posted by offby1

I can confirm that the server serves my library, but it does not appear to detect changes to the library that happen on another system; I mainly run Calibre on another machine and manipulate the library there, relying on Dropbox to sync it.

I haven't attempted this, but if calibre-server has the library opened in dropbox then the metadata.db will be locked and it will not be updated via any external device while locked.

If you search for conflicted copy in your dropbox folder I'm betting you find a few. These happen when another device updates something that is in use on the second device.

If you use dropbox to sync, only one device should be accessing the dropbox library at a time. You may want to consider a different method for syncing the folders. Preferably at a time when neither library is being used.

[chaley]

Originally Posted by DoctorOhh

I haven't attempted this, but if calibre-server has the library opened in dropbox then the metadata.db will be locked and it will not be updated via any external device while locked.

I use dropbox as my transport between the machine where I run calibre and the linux server that runs my content server. My experience indicates that dropbox uses a delete and create scheme on linux, which usually avoids the locked file problem. Unfortunately that also means that programs having the deleted file open (e.g., the content server) continue to see the deleted file, not the new file. I get around this by using monit to monitor the changed timestamp of metadata.db, restarting the content server. To be sure that I don't suffer from conflicts I copy the metadata.db file to /tmp and tell calibre to use the copy.

I agree with doctorohh - there must have been a conflict on metadata.db at some point. You need to ensure that all computers are seeing the same files.

[offby1]

There's no metadata.db conflict; I assumed that must have been the case, but it turns out no. For what it's worth, the content server _only_ reads the DB, it never (at least, not under my control) writes to the DB.

I have resolved the issue, however; it turns out I'd misunderstood how and when calibre writes to the DB; it writes on exit, not on edit, which meant that my changes had not persisted until I quit Calibre on my editing host.

*facepalm*

Thanks for the feedback.

[DoctorOhh]

Originally Posted by offby1

There's no metadata.db conflict; I assumed that must have been the case, but it turns out no. For what it's worth, the content server _only_ reads the DB, it never (at least, not under my control) writes to the DB.

I have resolved the issue, however; it turns out I'd misunderstood how and when calibre writes to the DB; it writes on exit, not on edit, which meant that my changes had not persisted until I quit Calibre on my editing host.

*facepalm*

Thanks for the feedback.

Excellent, thanks for the update.

[chaley]

Originally Posted by offby1

There's no metadata.db conflict; I assumed that must have been the case, but it turns out no. For what it's worth, the content server _only_ reads the DB, it never (at least, not under my control) writes to the DB.

I have resolved the issue, however; it turns out I'd misunderstood how and when calibre writes to the DB; it writes on exit, not on edit, which meant that my changes had not persisted until I quit Calibre on my editing host.

*facepalm*

Thanks for the feedback.

just for completeness because it might matter someday -- calibre does write to the db on each update. In most cases it opens a database transaction, writes everything it needs to write, then commits the transaction. However, even though the db is up-to-date, calibre is still holding the lock on the db. Depending on the OS and some other things, that lock will prevent other programs from opening and copying the db.

Also, even though the content server does not write to the db, it does open it for update. Python, calibre's implementation language, does not support read-only database access. Because the db is open for write there is a lock, which can block other programs from accessing the db.