Pushed it over the edge...now what?

[johnelle]

Let me say up front that yes I am using Calibre on the LAN which I know is unsupported but it seems actually fairly common.

I have been merrily loading books and it was never very fast but I could at least make progress working in the background. However in the last week I seem to have fallen over some cliff--a metadata update on one book takes 15 minutes or more. Its really unusable at this point.

Calibre V2.45
Lan Write Speed (90Kbits/sec)
Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
16GB Memory
Windows 10

It almost seems the db has hit some limit. Is there any way to reorganize the db? Any kind of optimization that will fix this serious problem? This isn't just a "gee I would like it to startup faster" issue--its really not usable any more.

[HarryT]

Have you tried doing a "Calibre Library / Library Maintenance / Check Library" operation?

[chaley]

You have probably exceeded some network cache size limit, either for the db or for the directory structure, causing the system to thrash. If this is the case there isn't anything that calibre can do.

I know you don't what to hear this, but using calibre on network file systems is simply asking for disaster. Some day things *will* go very badly, potentially destroying your library.

[johnelle]

I did run the library check and it found no errors.

Performance remains the same. I moved to a machine with a faster connection (about 300K bits/sec write) and it performs the same. I can look for network settings but I suspect it actually is the DB since the performance is bad even on this larger machine with a few less hops to the router.

I understand the risks but in reality all you lose is the metadata. The important data (i.e. the book contents) will still be there. The tool is free so I guess I can't complain but in this day and age to not be able to support network storage seems a bit arcane.

[chaley]

Originally Posted by johnelle

Performance remains the same. I moved to a machine with a faster connection (about 300K bits/sec write) and it performs the same. I can look for network settings but I suspect it actually is the DB since the performance is bad even on this larger machine with a few less hops to the router.

If a local cache starts thrashing then performance will fall by orders of magnitude. That cache might be a db cache, a directory structure cache, a disk block cache, a connection cache, or any number of other things.

The tool is free so I guess I can't complain but in this day and age to not be able to support network storage seems a bit arcane.

You might have a point, but in the end it is irrelevant. Calibre is how it is and NASes are how they are. You can accept the risk and the performance problems, or you can move your library to a local disk and if you want sync it to your NAS. Your choice.

[BetterRed]

@johnelle - calibre is a database application, to function on a network means it needs to have a robust client-server architecture, shared storage devices (NAS's, servers etc) don't provide that out of the box. Furthermore would have to work on a number of different operating systems (OS/X, Windows, Linux) and file systems (HFS, NTFS, Ext, NFS etc)

FWIW - Kovid is working on a new calibre server component that may address some/all of your issues, at guess I would say we'll see it next year. Kovid won't release it until he is confident it works reliably, and I anticipate he will have a beta testing phase.

Meanwhile, have you considered using the CALIBRE_OVERRIDE_DATABASE_PATH feature, see ==>> Environment variables. Or using symbolic links at the server that target the metadata.db file(s) on the client that does the updating. The efficacy of either will depend on what you're trying to achieve by having the library on a server.

BR

[johnelle]

Thanks for the environmental setting...that made it more straightforward to implement the local DB solution. Big difference. Haven't quite got the sharing working on other machines yet but eventually I will beat them into submission like anything windooz.

[BetterRed]

Originally Posted by johnelle

Thanks for the environmental setting...that made it more straightforward to implement the local DB solution. Big difference. Haven't quite got the sharing working on other machines yet but eventually I will beat them into submission like anything windooz.

@johnelle - If you want those other machines to update the library concurrently -- forget it; at some point you'll end up with a corrupt library database.

The database software used by calibre (SQLite) is designed for a single user environment. It's the same software Mozilla and Google use in their browsers, there's a rumour that MS uses it in Windows 10

If you were to run calibre-server on the other machines -- it's currently read only -- then you could keep a copy of the database on those machines and use the environment variable to refer to it.

The 'update' machine would put a copy of the database file on the server when its instance of the calibre library manager closes, and the other machines would take a copy of that file when their instances of calibre-server starts.

NB - I have never tried this. You might need to have the 'update' machine write a 'semaphore' to the server before it copies the database there and remove it after it's done copying. The other machines would look for the 'semaphore' before taking a copy and starting calibre-server and wait if the 'semaphore' is present, or tell the user to try again in a few minutes.

BR