Discussion - Backups, Synchronisation, Networks etc

[stumped]

Moderator Notice

split from this thread ==>> Moving a few books from Calibre on desktop to Calibre on laptop

BR

the only downside of sync utilities, assuming you send files over a local network,
it that you may need to grant some remote write permissions to the calibre folders so that the sync utilitiy can do its thing.
Not a huge security risk but something you will have to enable IF you run the utility from desktop had ask it to push the changes to laptop. Less so if you run it on the laptop and have it pull the changes from the desktop source. I have done stuff in that latter manner with free file sync, which will grumble about not being able to file lock the remote files that it is reading, while it reads them, but will will work
Whatever you do, be clear your mind and in your settings which is the source and which is the mirror, for a one way sync, especially if the mirroring operation is set to delete any files no longer found on source.

if both PCs run windows, then setting up a homegroup for documents is one possible secure way to do the needed file permissions, but I don't use homegroup - don't even know if windows still promote it. I do know that my LAN permissions tend to get messed up by each big windows update and have to be enabled all over again for me to be able to move/copy my files & folders

As an aside, I did find an android app that will do a folder sync, reading from a shared LAN folder on a PC , or a cloud folder, and writing to the android device. so you could mantain a calibre mirror library on a tablet or phone. It is called foldersync and comes in free and pro versions from google store. The pro version allows filters so e.g. I could mirror only the epub section of my calibre library.

[DoctorOhh]

Originally Posted by ilovejedd

Instead of doing a complete copy every single time, I would suggest using something like FreeFileSync in one-way mirror mode (with the desktop as master). More efficient particularly if one has a few thousand books. I do this for my Calibre Portable folder.

I use FreeFileSync as well it works great.

[Sirtel]

I've tried all the local network solutions mentioned in this thread. One downside of them is that they tend to be slow. The fastest way for me is Dropbox. Now Dropbox might be overkill if you move your books between devices seldom and the amount of books is not that big, network solutions are better in that case (I need to sync my laptop and tablet libraries every week, sometimes every day). Also the Dropbox free storage is pretty small, you might need the paid version.

Content server doesn't work for me because I don't have my devices up and running all the time. If you do, that might be the best solution.

[stumped]

Originally Posted by Sirtel

I've tried all the local network solutions mentioned in this thread. One downside of them is that they tend to be slow....

.

surely that depends on your LAN.

With Gigabyte ports on PCs & a Gigabyte Switch , I can move big files ( films, TV shows) at 50 - 100Mbs, pretty much same speed as going between hard drives within the same PC. A lot of small files will be slower but a typical calibre sync takes under a minute including starting up the utility.

I can push ~10GB of video to a plex server , which is on a USB3 drive connected to an Nvidia Shield TV in just a few minutes, with windows showing transfer speed of about 50Mbs

without having Gb connects throughout the chain that 50-100 drops to 10 at best. I have not benchmarked wifi but if a target laptop has a wifi 5Gz ac card or dongle and is nearby [ strong signal] , that should easily and quickly sync a hundred books in the time it take you to make a cup of coffee.

I agree that dropbox is also quite nifty - much faster than google drive, and I got mine to the 15GB max of free storage via referrals which is all I need. One think to be wary of with Dropbox that it is allegedly very hard to cancel a paid+ recurring payment dropbox plan and they never give refunds, ever, so they are not a company I would give my credit card details to

[Sirtel]

Originally Posted by stumped

surely that depends on your LAN.

With Gigabyte ports on PCs & a Gigabyte Switch , I can move big files ( films, TV shows) at 50 - 100Mbs, pretty much same speed as going between hard drives within the same PC. A lot of small files will be slower but a typical calibre sync takes under a minute including starting up the utility.

I can push ~10GB of video to a plex server , which is on a USB3 drive connected to an Nvidia Shield TV in just a few minutes, with windows showing transfer speed of about 50Mbs

without having Gb connects throughout the chain that 50-100 drops to 10 at best. I have not benchmarked wifi but if a target laptop has a wifi 5Gz ac card or dongle and is nearby [ strong signal] , that should easily and quickly sync a hundred books in the time it take you to make a cup of coffee.

I agree that dropbox is also quite nifty - much faster than google drive, and I got mine to the 15GB max of free storage via referrals which is all I need. One think to be wary of with Dropbox that it is allegedly very hard to cancel a paid+ recurring payment dropbox plan and they never give refunds, ever, so they are not a company I would give my credit card details to

I only have wifi (laptop and tablet, no desktop), so the speed is not that fast. Slower than Dropbox, anyway. Of course it's not so slow as to be unusable, but as I use Dropbox for back-up anyway it's just more convenient for syncing too. (And no, of course Dropbox is not my only way of making back-ups. It's just an additional way).

I use PayPal with Dropbox. It's easy to cancel a recurring payment with PayPal.

[HarryT]

Originally Posted by stumped

surely that depends on your LAN.

With Gigabyte ports on PCs & a Gigabyte Switch , I can move big files ( films, TV shows) at 50 - 100Mbs, pretty much same speed as going between hard drives within the same PC. A lot of small files will be slower but a typical calibre sync takes under a minute including starting up the utility.

I can push ~10GB of video to a plex server , which is on a USB3 drive connected to an Nvidia Shield TV in just a few minutes, with windows showing transfer speed of about 50Mbs

without having Gb connects throughout the chain that 50-100 drops to 10 at best. I have not benchmarked wifi but if a target laptop has a wifi 5Gz ac card or dongle and is nearby [ strong signal] , that should easily and quickly sync a hundred books in the time it take you to make a cup of coffee.

Note that "bps" is "bits per second", and "Bps" is "bytes per second". I think all your speeds are actually bytes per second, aren't they?

[stumped]

Originally Posted by Sirtel

).

I use PayPal with Dropbox. It's easy to cancel a recurring payment with PayPal.

good point - I had not thoguht of that.

your bottleneck is probably the file copy protocol used by the tablet. I have noticed how slow that can be, using file explorer type apps to go from a shared windows folder to a tablet. but I rarely need to do that. I have plex servers for all my video content and the plex android client is pretty good, so I don't need any local tablet storage for video or music ( spotify handles the music). that only leaves books to find space for.

a netflix download ( to watch later) is pretty fast, to tablet, but I have fast wifi. An hour of netflix at best quality downloads in about 5 minutes. lack of space on the tablet is my bottleneck, so usually I stream everything.

i do have all my calibre library backed up and synced into dropboix so I can get at it when the library PC is OFF, that works very well.

[ilovejedd]

Originally Posted by Sirtel

I've tried all the local network solutions mentioned in this thread. One downside of them is that they tend to be slow. The fastest way for me is Dropbox. Now Dropbox might be overkill if you move your books between devices seldom and the amount of books is not that big, network solutions are better in that case (I need to sync my laptop and tablet libraries every week, sometimes every day). Also the Dropbox free storage is pretty small, you might need the paid version.

I don't believe it's that local network copies are slower. The bottleneck is always on the slowest link in the chain so if your LAN is limited to, say, 100Mbps, then that's also the max internet (ergo Dropbox) speed you'll get.

Rather, it's likely because Dropbox runs and syncs continuously in the background so one doesn't really notice its actual transfer speed. Dropbox also doesn't require both computers to be running at the same time and that wait time before you can shut down computers when doing LAN copies can be a major factor in making the LAN seem slower.

I've actually done FreeFileSync or robocopy transfers of my full Dropbox folder between computers and/or NAS. A complete copy over LAN usually takes minutes whereas an initial Dropbox sync takes hours.

[stumped]

Originally Posted by HarryT

Note that "bps" is "bits per second", and "Bps" is "bytes per second". I think all your speeds are actually bytes per second, aren't they?

Hard question for this time of night

When I say 50 to 100 those are Microsoft units as shown in windows explorer. For a LAN copy. Much faster than writing to a usb2 stick which goes at about 8 on my best ones

I do understand the bits v bytes but I have to stop and ponder which one Microsoft uses....

Yes they would be bytes. I have gigabit ports and cables so theoretically I can move at 1000 mbs or 125 mBs if there were no overheads?. ( Give or take a zero or two! ), i know that one is 8 X the other, . I have seen actual speeds of 90 on screen during file copies for single very large files between 2 wired pcs. Also I know that my internet speed of 200mbs is too fast for a 10/100 fast internet port, if it is running at full capacity. Virgin media quote bits not bytes

Another real life measurement know I can copy over 100gb of mixed content in an hour if I am copying entire drives to a new pc.

So my 3gb of books in calibre would take very little time.

With the via Dropbox route there is also isp assymetry to consider. My virgin media cable internet is specced at 20x faster down then up. By design. So I can download at 200mbs but I can only upload at 10, theoretically, even less in practice. I cannot browse and max out the upload channel at the same time.
Thus it is great for installing games from steam but rubbish for being an outgoing video streamer. Which i am not.
To upload to Dropbox then to download to a target device is affected by my internet package upload limitations. Fine for my size of book library though

[Sirtel]

Originally Posted by ilovejedd

I don't believe it's that local network copies are slower. The bottleneck is always on the slowest link in the chain so if your LAN is limited to, say, 100Mbps, then that's also the max internet (ergo Dropbox) speed you'll get.

Rather, it's likely because Dropbox runs and syncs continuously in the background so one doesn't really notice its actual transfer speed.

No, it doesn't. I only start Dropbox when I want to sync and exit afterwards. To do otherwise is to ask for library corruption.

Actually I haven't tried syncing my library over local network for a long time. Possibly things have changed (for example, I have a new and much faster laptop). I might try again someday and compare the results. I'd still have to copy the books manually over, though, vs Dropbox doing it for me. Running FreeFileSync over local network is too slow, certainly much slower than Dropbox (well, my library is huge, 30 GB, so that's probably the reason).

[Sirtel]

Originally Posted by Sirtel

Running FreeFileSync over local network is too slow, certainly much slower than Dropbox (well, my library is huge, 30 GB, so that's probably the reason).

I just remembered that the speed of FreeFileSync might depend on the settings. Mine are set to compare the file content, not only the size and modification time. That probably slows the whole operation down considerably.

[ilovejedd]

Originally Posted by Sirtel

I just remembered that the speed of FreeFileSync might depend on the settings. Mine are set to compare the file content, not only the size and modification time. That probably slows the whole operation down considerably.

Yep, it does quite significantly. As far as I know, FFS does a bit-for-bit comparison meaning it essentially reads practically the entire 30GB on both source and destination when you sync (assuming you don't frequently change the actual ebook files for the bulk of your library via Polish or conversion or something).

If using compare by content method, there's pretty much no way FFS will be faster than Dropbox's differential sync. That's probably true even if you're just doing local HDD to HDD backups.

[HarryT]

That sounds somewhat dangerous to me. The only way I can think of that the contents of a file could change without the "last modified" date altering is data corruption - eg a failing disk. If that were to happen, would you really want those corrupted files to be copied to your backup device?

[Sirtel]

Originally Posted by HarryT

That sounds somewhat dangerous to me. The only way I can think of that the contents of a file could change without the "last modified" date altering is data corruption - eg a failing disk. If that were to happen, would you really want those corrupted files to be copied to your backup device?

Hmm haven't really thought of that, but you may well be right. I wonder why such an option is there at all then?

[ilovejedd]

Originally Posted by Sirtel

Hmm haven't really thought of that, but you may well be right. I wonder why such an option is there at all then?

That's actually useful for integrity checking. Note, FreeFileSync does allow 2-way syncing and you can do a comparison before you overwrite any files so it can be used to check for corrupt files.