FFS backup - exclude *.opf puzzle

[cybmole]

I decided to do what some other folks here have suggested: exclude the opf fills when making multiple library backups. I thought that all I'd need to do was set a filter fro that file type in free file sync and then all the metadata.opf files in the backup folder would be zapped by the one way sync function..... but it did not so that.- it did not remove any of them!

after checking my syntax many times & re-reading all the given examples,
it seems I have to remove them from the target directory manually as a one-off and then trust that the filter will block any new ones from being backed up - is that correct ?

PS - for the bytes saved i'ts hardly worth the hassle but I've had a BAD GOOGLE day, feel free to ignore the following rant...

I had google throwing a hissy fit about open dns certs & not letting my into gmail/google plus for some arcane reason that's too complex to fix ( changed my dns instead as a cop out), & I had google drive refusing to sync (that is maybe connected to the certs issue maybe not) & than after uninstall, re-install of google drive telling me I'd used nearly all my 15Gb when i reckoned i'd only used half... so zapping OPFs was just a little something to help get the total space down. it seems to be admitting now that i've only actually used 8Gb which is progress,...

I get that google chrome now sees opendns's attempt to route pages via their blocking filters as some sort of man-in-the-middle-attack & so chrome gives a security warning & refuses access to goggle services. I don't know what genius at google chose this week to implement that & I'm hoping opendns will send me a for-dummies workaround... i don't know if that impacts on how google drive works either but the backup library , minus it's opf files seems to be in sync still..
its only my backup of last resort anyway, I would hope never to have to use it as I have a local one also.

[eschwartz]

Filterin out the metadata.opf files partially negates the purpose of creating a backup, in return for miniscule space saving.
Don't do it.


Aside -- wow Google, nice job. How many institutes of public education have you pissed off now? My school used them for a while, but the IT volunteer walked off so they switched to Barracuda. Yippee -- now there is an actual honest-to-goodness MITM. And Chrome/Google is not blocking that, naturally.

[cybmole]

Ok , I can reinstate that file type within sync. I would still like confirmation that ffs was working as designed ,when it did not behave as I expected.

As for google chrome, I will see what opendns support come back with

Off topic, but google also made changes to their plug in support recently ,which now blocks the virginmedia tv anywhere web service from running in chrome. That has upset many uk customers. I forget the exact module name but it is to do with flash insecurity.

[BetterRed]

@cybmole - why don't you ask the folks at FFS, I've found them to be responsive and very helpful.

But as far as I can tell, a file is only removed from the mirror if its removed from the source, not if a filter is applied. Filters can be turned off and reinstated, so filters not removing what's on the mirror makes sense to me. Also the three incremental file synch products with which I'm familiar (FFS, Good Synch and Cheetah) all work the same.

IMO its best to keep backups as 'vanilla' as possible. It's almost inevitable that you'll be 'stressed' when you need them, so the fewer the exceptions the better. I'd rather find something in the backup that I don't need, rather than fail to find something I do need

Regarding Google. Products don't get customer service, customers get that, another pair of eyeballs is just another product they can sell to their customers - advertising agencies and multi-nat marketeers. TINSTAAFL thrives in Mountain View CA

BR

[DaltonST]

@cybmole:

I decided to do what some other folks here have suggested: exclude the opf fills when making multiple library backups. I thought that all I'd need to do was set a filter fro that file type in free file sync and then all the metadata.opf files in the backup folder would be zapped by the one way sync function..... but it did not so that.- it did not remove any of them!

after checking my syntax many times & re-reading all the given examples,
it seems I have to remove them from the target directory manually as a one-off and then trust that the filter will block any new ones from being backed up - is that correct ?

PS - for the bytes saved i'ts hardly worth the hassle

1. It has nothing to do with bytes saved. Not at all. It is all about saving hours and hours a week of backup time for huge libraries with hundreds of thousands of small files that change all of the time due to mass-maintenance of metadata, such that you avoid backing up because of the time it takes. If you make external and verified copies of metadata.db, you do not need to back up the .opf files if it becomes so burdensome that don't back anything up.

2. FFS does what you tell it to do, and does not do what you tell it to not do. You told it to filter the .opf files, so it did. Filter means "ignore".

3. To delete all of a particular file extension easily, use a simple .bat file that recursively traverses the directory structure by using the appropriate command-line switches as shown below, such as del /s /q z:\*.opf .

C:\Windows\System32>del /?
Deletes one or more files.

DEL [/P] [/F] [/S] [/Q] [/A[[:]attributes]] names
ERASE [/P] [/F] [/S] [/Q] [/A[[:]attributes]] names

  names         Specifies a list of one or more files or directories.
                Wildcards may be used to delete multiple files. If a
                directory is specified, all files within the directory
                will be deleted.

  /P            Prompts for confirmation before deleting each file.
  /F            Force deleting of read-only files.
  /S            Delete specified files from all subdirectories.
  /Q            Quiet mode, do not ask if ok to delete on global wildcard
  /A            Selects files to delete based on attributes
  attributes    R  Read-only files            S  System files
                H  Hidden files               A  Files ready for archiving
                I  Not content indexed Files  L  Reparse Points
                -  Prefix meaning not

4. Be sure to have multiple backups of all of your metadata.db files stored externally to your PC. I keep all of mine on a NAS, on an external SSD drive, and on an external HD.



DaltonST

[eschwartz]

Originally Posted by DaltonST

1. It has nothing to do with bytes saved. Not at all. It is all about saving hours and hours a week of backup time for huge libraries with hundreds of thousands of small files that change all of the time due to mass-maintenance of metadata, such that you avoid backing up because of the time it takes. If you make external and verified copies of metadata.db, you do not need to back up the .opf files if it becomes so burdensome that don't back anything up.

Interesting claim. Personally, I do not find it takes much time to back up the changes to hundreds of thousands of files, or at least, a few thousand (not everyone's library is multiple tens of thousands of books, and for hundreds of thousands of files I expect hundreds of thousands of books, minus the three metadata.db/backup-restored.db/prefs files, times three or perhaps four files per book record).

• Perhaps that is because as a backup service I allow it to happen unobtrusively, as a service.
• Perhaps it is because I do incremental changes. It is not entirely unlikely that there are other calibre users out there, that do NOT constantly perform mass operations that completely redefine metadata for all books. Most of my books remain pretty much the same after I process them, and my custom columns don't change much. (When they do, yes -- that impacts all metadata.opf files.)
• Perhaps both...

[JSWolf]

Originally Posted by DaltonST

1. It has nothing to do with bytes saved. Not at all. It is all about saving hours and hours a week of backup time for huge libraries with hundreds of thousands of small files that change all of the time due to mass-maintenance of metadata, such that you avoid backing up because of the time it takes. If you make external and verified copies of metadata.db, you do not need to back up the .opf files if it becomes so burdensome that don't back anything up.

And then you have an incomplete library should it need to be restored from a backup.

[eschwartz]

Originally Posted by JSWolf

And then you have an incomplete library should it need to be restored from a backup.

Incomplete isn't the problem -- the metadata.opf files are themselves merely a backup of the DB. However, the metadata backup exists to make your library useful in case the DB gets FUBARed, which means one would be well advised to keep them.

Assuming the backup contains an entirely working DB, which is not something I personally want to be dependent on, calibre will simply regenerate every one of those metadata.db files -- so if his backup didn't take time copying them, calibre will take even more time making them.
Why does he even care -- they are both background non-intrusive processes???


Either way, backing up the DB and not the OPFs is at least potentially dangerous, for, practically speaking, no net gain.

---

My backups are meant to protect against two points of failure -- database corruption, and library corruption.
I do not see how it hurts to protect against both happening at the same time. It is not significantly more unlikely than the possibility that one alone would happen.

[JSWolf]

Originally Posted by eschwartz

Assuming the backup contains an entirely working DB, which is not something I personally want to be dependent on, calibre will simply regenerate every one of those metadata.db files -- so if his backup didn't take time copying them, calibre will take even more time making them.

That is a very good point.

[cybmole]

it was suggested in some other thread that excluding opfs would speed up sync with cloud storage backups, as it's an extra file per book to check and sync. so folks with slow connections were all for it. but in terms of data, it "saved" somethign like 160MB of 3+GB

but as was just said in another response, it's easy to do something in calibre that causes all opfs to need re synching , even for books that have not otherwise changed( though I am not sure what the "something" is). so if google drive is running at it's usual snail pace, you notice that it's still chugging away when you want to shut down.

in my case, the cloud backups are like a last resort archive , I'd probably only go there for a single book that I'd rashly deleted from main live & daily backup but which I now wanted back.

[cybmole]

Originally Posted by BetterRed

@cybmole - why don't you ask the folks at FFS, I've found them to be responsive and very helpful.

But as far as I can tell, a file is only removed from the mirror if its removed from the source, not if a filter is applied.
BR

yep, that makes sense.

I used to use synchromagic shareware & maybe that worked differently. that was Ok, but much slower than FFS & no longer updated; I sill have a paid for licence somewhere.

as for google / opendns, I spent more time googling conflicts, I now think I'd recently told opendns to block "chat" as I don't use it, but that my be an integral part of google plus so blocking it has side effects. we'll see.

I am not that fussed about opendns, I thought I could maybe get some ads off of my tablet via that, by blocking the ad server URLs ( outbrain, taboola etc. ) but all that does is makes the links fail - you still get the rubbish I won an ipad for $ not a lot thumbmails, and all the other click bait images

[BetterRed]

Originally Posted by cybmole

it was suggested in some other thread that excluding opfs would speed up sync with cloud storage backups, as it's an extra file per book to check and sync. so folks with slow connections were all for it. but in terms of data, it "saved" somethign like 160MB of 3+GB

but as was just said in another response, it's easy to do something in calibre that causes all opfs to need re synching , even for books that have not otherwise changed( though I am not sure what the "something" is). so if google drive is running at it's usual snail pace, you notice that it's still chugging away when you want to shut down.

in my case, the cloud backups are like a last resort archive , I'd probably only go there for a single book that I'd rashly deleted from main live & daily backup but which I now wanted back.

If you correct a spelling or casing error in something like a tag or publisher that will trigger an update of all the relevant opf's, if you add a column and populate it via a bulk edit S&R that will also do it.

I did the latter to two libraries last last year, the change triggered opf updates to about 80K books. IIRC the daily backup to a WD Black Caviar in a USB 3.0 dock took an extra 10 minutes or so. Not sure how much longer the weekly backup took, I was busy sleeping - it's on a commercial FTP server over a 2Mbps WAN link

The most likely point of failure on my system is me. That's why I prefer the KISS approach.

BR