Discussion - Backups, Synchronisation, Networks etc

[BetterRed]

Originally Posted by HarryT

That sounds somewhat dangerous to me. The only way I can think of that the contents of a file could change without the "last modified" date altering is data corruption - eg a failing disk. If that were to happen, would you really want those corrupted files to be copied to your backup device?

Some well known image editors have an option to change the file system Modified timestamp to one of Exif timestamps (create, digitized, edited). Some also allow files to be altered and saved/overwritten with their original Modified timestamps, such as Irfanview's bulk conversion advanced options dialogue:



BR

[stumped]

Originally Posted by Sirtel

I just remembered that the speed of FreeFileSync might depend on the settings. Mine are set to compare the file content, not only the size and modification time. That probably slows the whole operation down considerably.

yes. run it on default settings except for one way mirror.

if it does my 3gb library in a minute it should do yours in under 10 ?

remember it only copies the changes. i think it keeps a timestamps database somewhere so it spots unchanged files very quickly after 1st run

you could also compare its speeds with that of another candidate like synctoy

[Sirtel]

I tried to copy some photos over LAN. While my internet speed is about 150 Mb/s download and 20 Mb/s upload, the speed over LAN is far slower, a couple of Mb/s max, often less than 1 Mb/s. Both my laptop and tablet have SSD drives. Why such a difference in speed, no idea. When I googled the problem, it seemed to be common with Windows 10 and there's no good solution. There were many, many complaints of the same all over the net.

So it seems I have to stick with Dropbox after all.

[ilovejedd]

Originally Posted by Sirtel

I tried to copy some photos over LAN. While my internet speed is about 150 Mb/s download and 20 Mb/s upload, the speed over LAN is far slower, a couple of Mb/s max, often less than 1 Mb/s. Both my laptop and tablet have SSD drives. Why such a difference in speed, no idea. When I googled the problem, it seemed to be common with Windows 10 and there's no good solution. There were many, many complaints of the same all over the net.

Do note, Windows reports file copies in megabytes per second (MB/s) while ISPs advertise in megabits per second (Mb/s).

150 Mb/s = 18.75 MB/s
20 Mb/s = 2.5 MB/s

[HarryT]

Originally Posted by BetterRed

Some well known image editors have an option to change the file system Modified timestamp to one of Exif timestamps (create, digitized, edited). Some also allow files to be altered and saved/overwritten with their original Modified timestamps, such as Irfanview's bulk conversion advanced options dialogue:

Attachment 166863

BR

I'm afraid it still strikes me as being rather a recipe for disaster. Consider the situation: you run an incremental backup at the end of each day. Your disk begins to fail, and slowly your documents begin to be corrupted, perhaps over a period of months. Each day, those newly-damaged files will overwrite the "good" versions on your backup device. You finally realise there's a problem and discover only then that your backup device contains corrupted files rather than the good versions it used to.

Not an option I'd use personally.

[ilovejedd]

Originally Posted by HarryT

I'm afraid it still strikes me as being rather a recipe for disaster. Consider the situation: you run an incremental backup at the end of each day. Your disk begins to fail, and slowly your documents begin to be corrupted, perhaps over a period of months. Each day, those newly-damaged files will overwrite the "good" versions on your backup device. You finally realise there's a problem and discover only then that your backup device contains corrupted files rather than the good versions it used to.

Not an option I'd use personally.

True but it's also possible that bit rot/corruption is occurring in the backup drive rather than the source.

It's not an option I'd use on automatic/batch mode but it's useful for integrity check when running FFS manually.

[stumped]

that is not normal

most likely one or both PCs has network drivers that do not play nice with windows 10, but were fine with windows 7 if that's what you had when purchased.
this dell vostro PC I am typing on has a broadcom onboard network adaptor that is not reliable in win 10 . and there is no upgrade, so I had to put in a cheap Gbit network card & now it is fine.

probably not worth the hassle of troubleshooting, that would be easier with 3 machines so that you could try to isolate which might be the slow one.
how is the LAN set up, are you using wifi on both, no cabling to check? or a switch
you can look at network adaptor properties, maybe set them to default values and/or ask windows to check for updates or try rollbacks to previous versions
Alternatively a desktop network Gbit card is very cheap, and probably you can get a fast USB laptop network dongle there are no card slots. and returnable if you use amazon and they don't improve things. that way you bypass doing the frustrating troubleshooting. i am not sure what the options would be for a tablet, except to try alternative apps. is it windows or android ? MS have depreciated support for SMB v1 which a lot of android apps still use

[Sirtel]

Originally Posted by ilovejedd

Do note, Windows reports file copies in megabytes per second (MB/s) while ISPs advertise in megabits per second (Mb/s).

150 Mb/s = 18.75 MB/s
20 Mb/s = 2.5 MB/s

Ah, that's good to know. Probably very few people are aware of this, so there's universal wailing and gnashing of teeth over slow LAN speeds.

[Sirtel]

Originally Posted by stumped

that is not normal

most likely one or both PCs has network drivers that do not play nice with windows 10, but were fine with windows 7 if that's what you had when purchased.
this dell vostro PC I am typing on has a broadcom onboard network adaptor that is not reliable in win 10 . and there is no upgrade, so I had to put in a cheap Gbit network card & now it is fine.

probably not worth the hassle of troubleshooting, that would be easier with 3 machines so that you could try to isolate which might be the slow one.
how is the LAN set up, are you using wifi on both, no cabling to check? or a switch
you can look at network adaptor properties, maybe set them to default values and/or ask windows to check for updates or try rollbacks to previous versions
Alternatively a desktop network Gbit card is very cheap, and probably you can get a fast USB laptop network dongle there are no card slots. and returnable if you use amazon and they don't improve things. that way you bypass doing the frustrating troubleshooting. i am not sure what the options would be for a tablet, except to try alternative apps. is it windows or android ? MS have depreciated support for SMB v1 which a lot of android apps still use

Both the laptop and tablet have Windows 10 and had the same when I bought them. I've tinkered with various network adapter properties and checked the drivers, no change.

Wifi on both, yes.

[stumped]

Originally Posted by ilovejedd

Yep, it does quite significantly. As far as I know, FFS does a bit-for-bit comparison meaning it essentially reads practically the entire 30GB on both source and destination when you sync...

no. if i run a sync, shortly after a previous one it takes no time at all, it is definitely relying on saved data to determine what if anything has changed. maybe folder timestamps if windows has those ?

if i run one after say 24 hours the check takes longer & I can see a scrolling list of file names as it checks them

but it is still 10x faster than a previously owned shareware called synchromagic, that did use a slow checking method

in my version , which may not be the latest, pressing F6 brings up the comparison options & you can choose either file time and size,[ Identify equal files by comparing modification time and size.] [ or file content.
i think a new install defaults to the time and size method

ps I just ran one, mirroring a source calibre folder to 2 target folders ( one in dropbox - so that is to the local dropbox copy on the same C drive (SSD) and one to 2nd hard drive. spinning drive)
I see an items found count race up to about 20,000 and a time in secs get up to about 20 secs for the scanning phases, then ti did a handful of copies of actula changes made today...
when FFS is done, I see the dropbox tray icon show blue for syncing for a few seconds, as the book changes in C:/dropbox get uploaded.
PS I assume that dropbox itself uses file name and timestamp to determine what is newer than cloud ( and maybe some secret proprietary checks also) > I used to fret that if ever i restored a C drive from a windows 7 tpe system image, then Drobox could get confused , but the days of relying on that method are gone now. its faster to clean install W10 and then put all your backed up files and reinstall your programs back, than to want to use a drive image which may throw a sulk and not work anyway.

[ilovejedd]

Originally Posted by stumped

no. if i run a sync, shortly after a previous one it takes no time at all, it is definitely relying on saved data to determine what if anything has changed. maybe folder timestamps if windows has those ?

if i run one after say 24 hours the check takes longer & I can see a scrolling list of file names as it checks them

but it is still 10x faster than a previously owned shareware called synchromagic, that did use a slow checking method

in my version , which may not be the latest, pressing F6 brings up the comparison options & you can choose either file time and size,[ Identify equal files by comparing modification time and size.] [ or file content.
i think a new install defaults to the time and size method

I was referring to the compare by content option which Sirtel was using and given its nature, would obviously take a long time.

The immediate subsequent sync taking a shorter time is likely due to Windows RAM caching of recently read file system tables rather than any database that FreeFileSync keeps (if it had been a FreeFileSync database, then it should still be just as quick after 24 hours).

Yes, FreeFileSync defaults to file size/modification time and that default setting is very quick for updates as that just transfers changed files same as Dropbox does. The compare by content, however, is not.

[stumped]

OK that all makes sense.
PS i mirror all of the calibre app date folder to dropbox also , treating it as a black box that I don't need to understand in detail
going back to the OP issue of slow sync over LAN, yto tablet: I have a guitarpro tab library of ~1000 files & to get that mirrored onto ( android ) tablet from PC or dropbox I use a syncfolder app to mirror. That is noticeably slower, doesn't matter if the source is dropbox or shared file on PC. I think tablets are just slow at that sort of operation. I can watch the mirror happening and it is taking a second or so to decide if each file has changed or not, and these are small files, typically less than 100k each. Dropbox on android will not export / save to device a folder, it will only export single files, hence the need for an app rather than just copy everything.
more proof as to tablets being slow,. I recall doing some testing and wanting to put a couple of 1GB video files onto tablet in various formats to try out various players. That was much slower than copying PC to PC. Even with the wifi link reporting a 100Mb connect speed, it could be 10 mins or moreto copy 1GB from a shared PC folder via an app

[ilovejedd]

Originally Posted by stumped

I think tablets are just slow at that sort of operation.

To be expected on most particularly those with lower capacities. eMMC isn't really as fast as your normal SSDs. Heck, ones used in most tablets are probably slower than HDDs.

Originally Posted by stumped

Dropbox on android will not export / save to device a folder, it will only export single files, hence the need for an app rather than just copy everything.

I believe Dropbox does allow full folder syncing for paid accounts.

Originally Posted by stumped

Even with the wifi link reporting a 100Mb connect speed, it could be 10 mins or moreto copy 1GB from a shared PC folder via an app

Throughput will always be worse than the link speed (often significantly worse). In this case, though, you're probably also hitting the speed limit of the NAND flash used.

[stumped]

Dropbox. The export single files only is a a limitation of the dropbox android client. , Not a free vs paid plan thing. 99% sure of that. There is only one version of the dropbox android client app. Google store has other 3rd party apps that workaround it though, probably they just automate going through a folder and exporting each file

The OP said he has a tablet with an SSD?

NAND flash speeds, yes. Makes sense to use tablets as streaming clients with most files held elsewhere, be they books, music or videos. Rather than add relatively expensive fast SD cards. Plex media server and client does pretty much everything except books and probably even has a plug in for those! But calibre companion is my tool of choice. I never got into using Kodi over LAN.

[ilovejedd]

Originally Posted by stumped

Dropbox. The export single files only is a a limitation of the dropbox android client. , Not a free vs paid plan thing. 99% sure of that. There is only one version of the dropbox android client app. Google store has other 3rd party apps that workaround it though, probably they just automate going through a folder and exporting each file

https://www.dropbox.com/help/mobile/offline-folders

Save folders for offline viewing on a mobile device
The feature discussed in this article is available to Dropbox Plus, Professional, Business, and Enterprise customers.

Select your operating system for instructions:
Android
iOS

Originally Posted by stumped

The OP said he has a tablet with an SSD?

NAND flash speeds, yes. Makes sense to use tablets as streaming clients with most files held elsewhere, be they books, music or videos. Rather than add relatively expensive fast SD cards. Plex media server and client does pretty much everything except books and probably even has a plug in for those! But calibre companion is my tool of choice. I never got into using Kodi over LAN.

Technically, modern smartphones and tablets also use NAND flash/solid state technology. It's the implementation that varies. The controller/design used in mobile devices with constrained space and thermal allowances obviously won't be as fast as the implementation for SSDs designed for desktop/laptop use.

SD cards are super cheap compared to the premium Apple expects for storage upgrades. Granted, Apple's storage design is pretty fast (perhaps NVMe level?).

Mind, it seems to me Sirtel might be hitting wifi throughput limits and interference as they mentioned "a couple of Mb/s". As previously noted, Windows reports copy speeds in MB/s so that 1 MB/s is actually equivalent to 8 Mb/s. Given wifi throughput is highly variable and dependent on many factors (e.g. distance from router, obstructions, presence of other wifi networks, etc.), the file transfer speeds might actually be quite normal.