Question on Calibre's requirement for files to be local

[haertig]

I know that Calibre does not tolerate it's database being on a remote server, accessed via a network.

Does this apply to ALL of the files that Calibre touches, or just "it's own" files? e.g. metadata.db and metadata_db_prefs_backup.json found at the top level of the book library. There are also all the metadata.opf files in each individual books directory.

It would be nice if we could put all the actual book files - the .epub's, .azw3's, cover.jpg, etc. on a remote server that uses NFS, Samba, or whatever to mount that to our "primary Calibre computer", and leave the metadata* files as the only local ones.

I have been working for years with all of Calibre's library on my primary desktop. Works fine. But when your primary desktop goes wonky (I have some kind of motherboard issue going on at the moment I think) it's a pain to switch your Calibre instance over to a secondary computer. Doable, and I also have solid backups I could restore from, ... but still a pain.

It would sure be handy if we only really have to worry about metadata.db and the json file being local, with the actual book files being a remote mount. Can this be done, safely, or does Calibre really require everything to be local? My particular situation is only one Calibre user at a time (me!), and only using it from one computer at a time. I imagine this is the case for most Calibre installations.

[chaley]

Originally Posted by haertig

I know that Calibre does not tolerate it's database being on a remote server, accessed via a network.[...]

First, all the files in the calibre library folder structure are its "own files". Just because you can see the folder structure and files doesn't change that.

The entire calibre library should be local. If it isn't then if an operation fails, which it often does, books are lost. The usual reason is that folder renames on network file systems fail. YMMV.

Why not use calibre portable on a quality external drive/stick? With portable, moving between computers is trivial.

[BetterRed]

Have a look at the responses in this recent thread ==>> Problems Getting Desired FileName Format

Originally Posted by Quoth

. . .

…pretend the Calibre files don't exist. They only exist at all because it's never worked well for any database to import that sort of thing [covers and format files] inside a database.

That's why there is an Export to Disk File command and also why Calibre creates a copy of any file imported (as if it was really inside a database).

BR

[haertig]

Originally Posted by chaley

Why not use calibre portable on a quality external drive/stick? With portable, moving between computers is trivial.

Great suggestion! I am looking at getting one of these for this purpose:

https://www.amazon.com/dp/B07D7PDLXC...v_ov_lig_dp_it

I may need to get the 256Gb size though. I store audiobooks (each is a ZIP file containing several MP3 files) in Calibre as well as eBooks. Audiobooks are relatively huge. My Calibre installation is currently 106Gb, so it would fit on a 128Gb flash drive, but I should probably get the 256Gb size to allow for future growth in audiobooks.

[theducks]

That will work. I prefer a Hard drive for faster writes than Flash. But they cost 3X what that one does.

[Sirtel]

I used a flash drive with my Calibre library a while ago, to switch between computers. It was painfully slow, often froze and threw up errors. And it wasn't a particularly cheap drive. I gave up. An external hard drive is a much better idea.

[haertig]

Originally Posted by theducks

That will work. I prefer a Hard drive for faster writes than Flash. But they cost 3X what that one does.

An external hard drive (mechanical, not SSD) is typically in the 120-ish MB/s range for reads and writes (writes being the slower of the two). This flash drive is faster than than. They quote up to 400MB/s reads, which probably equates to 150 MB/s real world numbers, but that still beats a mechanical hard drive.

The reason I am looking at moving Calibre to a flash drive is because my main desktop computer is sick at the moment. I am currently typing this post while running my entire OS off of flash memory plugged into a usb port on a spare computer. Booted from the usb flash drive. And it is quite fast. Amazingly so, actually. antiX Linux (v21) on this specific flash drive:

https://www.amazon.com/dp/B07XHYVN62...roduct_details

This spare computer has Windows loaded on it's primary SSD, and that's not to my liking, so I went with booting Linux-from-usb-flash instead. Now I'm adding Calibre-on-flash until I can get my main computer back up and running reliably - it's looking more and more like motherboard problems Since I can't pull my Calibre library off of the dead computer, I will pull it off of my server that does our household backups instead - backups are done every 24 hours, and my Calibre library is up to date on the backup media. So, Calibre will come from backup media and copied to flash, and then plugged into the spare computer.

[theducks]

Check all fans,
Check all (heat sink) Fins and grills
Overheats can cause erratic operation.

Inspect your MoBo for bulging Caps (the tops get a dome)

[haertig]

Originally Posted by theducks

Check all fans,
Check all (heat sink) Fins and grills
Overheats can cause erratic operation.

Inspect your MoBo for bulging Caps (the tops get a dome)

Thanks. I've done all that. Symptoms are a random and intermittent "lights out" shutdown.

Dust build-up inside the system and on the heatsinks was minimal, but I blew what little there was out with Dust-Off. Inspected both mobo and power supply for bulging caps - none found (but caps can go bad without bulging). Replaced power supply(it was a high end unit before, it's an even more high end supply now). Replaced memory (all modules). The system has always run super cool (a high-end case with four 120mm fans plus the cpu cooler facilitates that). But I added a fifth 120m fan just for grins, even though temperature sensors showed no high temps and all heat sinks were cool to the touch. Tried different OS'es (booting from flash drives) even though it does not feel like a software issue.

Nothing has worked, dang it!

The only remaining thing on my troubleshooting list is to remove the cpu cooler, clean, and reapply fresh thermal paste. But immediately after a shutdown - I had the case open at the time - I grabbed all the heatsinks within five seconds of the shutdown. All were cool, I couldn't detect any heat at all. I checked this because thermal sensors can sometimes lie.

BUT, the only way I can force this shutdown repeatably is to start transcoding video. When the cpu temp hits about 50 Celcius, which is far under the max specs for this cpu, ... lights out. Repeatably. I checked BIOS to see if somehow a setting there got corrupted related to thermal shutdowns, but it is set to alert at 65 Celcius - and it never alerts. So this repeatability certainly points to a thermal shutdown, albeit way too early. The random shutdowns, when I am not trying to force them with transcoding, are occurring at the computers normal operating temp of 30 Celcius. So it both "feels thermal", and then it "doesn't feel thermal" (especially since the heat sinks are cool to the touch at the random shutdowns).

So fresh thermal paste is on the agenda for tomorrows troubleshooting. Can't hurt. The current paste has been on there for about four years with no heat issues, but it's time to redo it anyway. The cpu cooler itself is running fine. The system will alarm if the cpu fan ever drops below 100 rpm, which it never has, except when I have stuck my finger in there and manually stopped it to test the alerting feature.

Thanks for trying to help! If you have any other ideas for me to try, I'm all ears!!! I don't know what else to troubleshoot. I'm thinking bad cap on the mobo, even though nothing is bulging. I won't be replacing caps on the mobo, I'll just buy a new one (and new cpu, and new memory to go with it). I would prefer not to spend the hundreds of dollars that this will cost at the moment, but I may have to. The system is old, but still perfectly adequate for my needs (well, if it runs, that is).

[theducks]

Transcoding uses the GPU in many cases. Check the power connections on the graphics board (if you have one). I have lost quite a few graphics (non game quality) boards to failed GPU fans. (I just needed to handle HD monitors)

[haertig]

Originally Posted by theducks

Transcoding uses the GPU in many cases. Check the power connections on the graphics board (if you have one). I have lost quite a few graphics (non game quality) boards to failed GPU fans. (I just needed to handle HD monitors)

Only integrated graphics on this computer. Only one card installed - in a PCIe slot to add usb 3.0 (yeah, it's that old - the mobo only has usb 2.0 natively).

p.s. - In case you're wondering, even though I boot off of a flash drive on occasion with this computer, it can't boot usb directly. I start the boot from a CD/DVD, and then transfer the boot to usb (using the 3.0 port) halfway through the boot.

[theducks]

Did you enable USB boot in the BIOS? (I think 1 of my system has 2 places that must be set. The second is moving it up on Boot drives order.)

[ownedbycats]

I had sorta-similarish issues in one of my computers about a decade back due to a GPU with capacitor plague, though I think plague usually comes with bulging/leaking.

[compurandom]

A decade ago "capacitor plague" was common and caused by a batch of fraudulent capacitors that got into the supply chain.

Another possibility for your sick system is that possibly the power supply is dying and can't handle a full load.

But to get this back on the original thread topic, I know of three reasons why calibre doesn't work well on a network filesystem:

• Calibre doesn't expect to share its files. If you run two copies of calibre on the shared disk, they will step on each others toes and corrupt things.
• Many network filesystems (Onedrive, Dropbox, Google drive) do not preserve filenames, and like to mangle them for you. This corrupts things badly.
• Some network filesystems are not stable, and if they disconnect while calibre is running, this corrupts things. Samba can disconnect in the middle of a file operation. sshfs is worse, as it requires a manual restart. NFS is designed to handle this, and will hang the client until the server comes back.

I can't say for sure, but I suspect calibre would be fine on NFS as long as you didn't run it twice on the same library. But if your desktop is unstable, it could still crash with calibre running, and this still could mess things up.

[Terisa de morgan]

Originally Posted by haertig

Great suggestion! I am looking at getting one of these for this purpose:

https://www.amazon.com/dp/B07D7PDLXC...v_ov_lig_dp_it

I may need to get the 256Gb size though. I store audiobooks (each is a ZIP file containing several MP3 files) in Calibre as well as eBooks. Audiobooks are relatively huge. My Calibre installation is currently 106Gb, so it would fit on a 128Gb flash drive, but I should probably get the 256Gb size to allow for future growth in audiobooks.

I've used something like that for some time, and I've always found Sandisk the best brand, they long last but they always die in the end depending on the number of writes (and I'm talking about a year maximum).