Question on Calibre's requirement for files to be local

[haertig]

Originally Posted by compurandom

NFS is designed to handle this, and will hang the client until the server comes back.

Yeah, it locks 'em up good and tight sometimes, doesn't it? I always thought this was due to a suboptimal design in NFS. It's interesting to hear it described as a design feature. I never thought of it that way.

The motherboard (and original power supply) in this computer are from the "capacitor plague" generation. I just replaced the power supply with a high end EVGA modular Platinum one, so that's good now. Still having the random shutdowns with the new one though, so evidently it was not the original power supply causing the problem. I'll keep it as an emergency spare, because it too was a high end model back in it's day. I'm thinking the mobo may have unceremoniously reached end of life. No bulging capacitors or magic smoke, but it is old. When's the last time you ran into a mobo that still uses DDR2 RAM??? Again, another "high end in it's day" component, but like a Ford Model T, it's day has long since passed.

It's good that you pointed out that things like flakey computers put the Calibre library/database at risk. Remote access over a network is not the only thing that can kill Calibre. Everyone should be reminded of this from time to time. I am aware of this, but not terribly concerned in my case - I have multiple redundant backups, and they are spread across three separate servers.

[haertig]

Originally Posted by Terisa de morgan

I've used something like that for some time, and I've always found Sandisk the best brand, they long last but they always die in the end depending on the number of writes (and I'm talking about a year maximum).

I'm surprised to hear that your flash memory dies that quickly. I would expect a lot of Calibre to be a "write once, read slightly more frequently" operation. My normal usage is to add a new book to Calibre, set all it's metadata, then leave it alone for the rest of it's life. The files may be read when I pull books out of Calibre onto my Kindle, but they shouldn't be re-written often, if at all. Unless Calibre is doing something funky in the background that I'm not aware of. Given this usage pattern, I would expect flash memory to outlast me. Especially if you buy a flash drive a decent bit larger than the size of your Calibre library, so the wear-leveling algorithms can spread out the writes even further.

[Capricorn]

You might also consider if the HDD in your computer is the culprit: A number of years ago, I had a flaky system for 9 months. I checked everything several times, just like you did, including the HDD for bad blocks, and never found anything. I checked drivers, I looked at the log files of the system and never found anything. I removed any software not really needed. Nothing changed. After nine months, the HDD simply died on me one day. After replacing it, no more flakiness. Simply a stable system again. So, I reckon my HDD was on the cusp of dying, but not really dying for nine months.

And by the way: that was a brand new HDD, that caused the problem. So, it must have had some problems from the start.

[mbovenka]

Originally Posted by Capricorn

And by the way: that was a brand new HDD, that caused the problem. So, it must have had some problems from the start.

That's the bathtub curve for you...

[theducks]

The worse thing is when flash dies: It just dies without warning.
With a HD, you usually get start getting errors, although some fails can be spectacular. (I had an external drive lose control of its speed and it started vibrating all around my desk before I got it shut down).

[theducks]

Originally Posted by Capricorn

You might also consider if the HDD in your computer is the culprit: A number of years ago, I had a flaky system for 9 months. I checked everything several times, just like you did, including the HDD for bad blocks, and never found anything. I checked drivers, I looked at the log files of the system and never found anything. I removed any software not really needed. Nothing changed. After nine months, the HDD simply died on me one day. After replacing it, no more flakiness. Simply a stable system again. So, I reckon my HDD was on the cusp of dying, but not really dying for nine months.

And by the way: that was a brand new HDD, that caused the problem. So, it must have had some problems from the start.

Use a tool that can display the S.M.A.R.T. status on your drive. (A Linux Live CD usually has one)

[haertig]

I looked at the SMART data for the two hard drives and the SSD in the computer a month or so ago. No problems there. However, the drives are old. 8-10 years for one, even older for the other. The SSD, is maybe four or five years old? The only thing I have on the hard drives is my Calibre library on one drive, and some misc (unused) Linux distros on the other. I'll disconnect the hard drives and see if that helps. That's not something I would have thought to do. I do have a spare SSD also, that I could install if required.

[ownedbycats]

Originally Posted by theducks

The worse thing is when flash dies: It just dies without warning.
With a HD, you usually get start getting errors, although some fails can be spectacular. (I had an external drive lose control of its speed and it started vibrating all around my desk before I got it shut down).

I once had a head crash in an external. It's a noise I'm absolutely terrified to hear again.

[ableal]

Another way to have the same library (or set of libraries) available across several computers is to use one of the cloud services that allow folder/directory synchronization on different machines. The files are still local, they just get copied to the cloud service and to other machines where the service runs.

I know that Mega.nz works, and also use Google Drive for similar things. OneDrive, Dropbox, etc also offer the functionality. Only problem I've had is the sync agent silently crashing, and things like folders that were renamed (e.g. the exact spelling of an author changed) having both old and new versions. No big deal.

[haertig]

Originally Posted by ableal

The files are still local, they just get copied to the cloud service and to other machines where the service runs.

You can do this yourself, without needing the cloud, using a peer-to-peer sync client. "Syncthing" is a respected example of this type of software. It's free - or at least there is a free version available, maybe they have paid versions as well. There is no central server with Syncthing. Or you can set up your own self-hosted cloud if you're a security, privacy, and control person like I am. I use Nextcloud for this.

I would be hesitant to use syncing with any of these products while Calibre is running however. "Hesitant" is not the best way to say it, ... "I wouldn't do it, period" is a better way of stating things. If you implement a method to shut down Calibre on all servers, and then do the syncing, restarting Calibre afterwards - that would work just fine.

[ownedbycats]

I've once considered whether it would be possible to use a virtual hard disk (VHD), so there would only be a single file containing the library which could be mounted and unmounted as needed - the main downside I can think of is that it'd be a rather big file, so syncing would take a while.

[theducks]

Originally Posted by ownedbycats

I once had a head crash in an external. It's a noise I'm absolutely terrified to hear again.

I have opened up a couple of those. Itty Bitty metal pieces wit other stuff thrown in.

[compurandom]

Originally Posted by ableal

Another way to have the same library (or set of libraries) available across several computers is to use one of the cloud services that allow folder/directory synchronization on different machines.

These are exactly the kinds of things that kill calibre libraries. Most of the cloud sharing services don't honor filenames and like to mangle them. Dont Do It!

[ownedbycats]

Originally Posted by theducks

I have opened up a couple of those. Itty Bitty metal pieces wit other stuff thrown in.

I had saved in my bookmarks this rather impressive head crash that literally scraped the platter down to the glass. Of course, it's an IBM Desk Deathstar.

[compurandom]

Originally Posted by ownedbycats

rather impressive head crash that literally scraped the platter down to the glass.

I have a couple of those hanging over my desk, multiple brands, both aluminum and glass. The glass one I ran shred on prior to surplussing, and aborted it nearly half way through when I saw all the sequential bad blocks.

Remember, on a clear disk, you can seek forever.