metadata.opf question

[robin58]

I never really paid attention to the metadata.opf files until I started using a weekly automated program to create a mirrored back-up copy of my library.

I have noticed every week now 1000's of books I know I haven't opened, read, edited or changed metadata on were being reported by the software I use as being in need of updating as they had changed.

I used UltraCompare today to take a peek inside several of the files to see exactly what was being changed and it appears it is only

<package xmlns="http://www.idpf.org/2007/opf" unique-identifier="uuid_id" version="2.0">

and

<dc:contributor opf:file-as="calibre" opf:role="bkp">calibre (1.24.0) [http://calibre-ebook.com]<

As this seems to be somewhat trivial info, I assume it's safe to start excluding opf files other than ones for books I know I have updated? I know the files are minuscule, but in the aggregate it adds up when doing an online backup.

I am also wondering why calibre appears to only update some books in this fashion.

[eschwartz]

How do you intend on excluding them in an automated manner??

If your backup program is any good (and most are ), it should know how to minimize the effort needed to update synced files, for instance, by only syncing the changed portion.

[cybmole]

Originally Posted by robin58

I am also wondering why calibre appears to only update some books in this fashion.

this is the more interesting question - if the books are not being read or amended, or metadata edited, why are the files changing ?

[eschwartz]

Originally Posted by cybmole

this is the more interesting question - if the books are not being read or amended, or metadata edited, why are the files changing ?

A tag was edited most likely, depending on where this (and other actions) was done, this will cause all books sharing that metadata to be backed up. Nothing *changed* except for the timestamp usually, and apparently a new piece of info calibre decided to store by default in the OPF, which was not previously present and thus had to be added.

[theducks]

Originally Posted by eschwartz

A tag was edited most likely, depending on where this (and other actions) was done, this will cause all books sharing that metadata to be backed up. Nothing *changed* except for the timestamp usually, and apparently a new piece of info calibre decided to store by default in the OPF, which was not previously present and thus had to be added.

Changing/deleting any tag used in a book:
The (backup) OPF does get rewritten because Tags are stored in clear text.

OPF files are small

[BetterRed]

@robin58 - Strange, I would have thought the two lines quoted would have been amongst those least likely to change.

But why are the files changing? There are things that change the book metadata that are not immediately obvious, such as - adding, changing (the definition) and removing custom columns; changing the case or spelling of an author, publisher, series, tag value etc,

The opf files are there so that the calibre database (metadata.db) can be rebuilt if and when necessary. If you don't back them up, then in the event of a disaster you may not be able to rebuild your library... easily.

You may have to add all the books again... that should fairly easy. You may have to put the metadata back together again... that might be hard if it was created manually and it includes custom columns that are not stored in the books formal metadata. I add a belt to my backup braces - all the metadata is in the ePUB's book jacket.

How frequently do you do the backup ? What software do you use? And when you say 'online backup' does that mean over a local network, or over a wide area network?

BR

[robin58]

How do you intend on excluding them in an automated manner??

I was simply going to exclude all opf files. Both Goodsync which I had been using and FreeFileSync which I have also been trying recently allow this fairly simply.

A tag was edited most likely, depending on where this (and other actions) was done, this will cause all books sharing that metadata to be backed up. Nothing *changed* except for the timestamp usually, and apparently a new piece of info calibre decided to store by default in the OPF, which was not previously present and thus had to be added.

I have no tags. They had become a mess a while back and I simply removed all tags from all books.

Changing/deleting any tag used in a book:
The (backup) OPF does get rewritten because Tags are stored in clear text.

OPF files are small

See my comment above. I already had deleted all tags from every book I had.

Strange, I would have thought the two lines quoted would have been amongst those least likely to change.

But why are the files changing? There are things that change the book metadata that are not immediately obvious, such as - adding, changing (the definition) and removing custom columns; changing the case or spelling of an author, publisher, series, tag value etc,

The opf files are there so that the calibre database (metadata.db) can be rebuilt if and when necessary. If you don't back them up, then in the event of a disaster you may not be able to rebuild your library... easily. You may have to add all the books again, which should fairly easy, and put the metadata back together again, might be hard if it was created manually and it includes custom columns that are not stored in the books metadata.

How frequently do you do the backup ? What software do you use? And when you say 'online backup' does that mean over a local network, or over a wide area network?

The change is what puzzled me too. Today I synced everything before the update. I had 3,000+ opf files that were supposedly different and 2 books that I had added during the week. I updated my backup. Then proceeded to update to the latest version of Calibre (1.2.4) and added another book. I did nothing else. Shut Calibre down. And then for the heck of it ran the filesync programs again. I had 549 books that reported changes (incidentally both Goodsyn and FreeFileSync reported the exact same differences). That's when I took the peek inside and only those two references I quoted in my first post above were in any way different.

I understand they're an important factor in restoring should something become corrupt, but unless I'm missing something if the version number of Calibre is going to prevent an accurate restoration, then having a backup is useless anyways, right?

As for frequency, I back it up once a week to a networked drive which then is also mirrored on an online storage as it is a watched folder.

[eschwartz]

The problem isn't a lack of the appropriate version number as a contributor... If you exclude the OPF from being backed up, then it ... isn't ... backed up!

[BetterRed]

@robin58 - I use Goodsynch, matter of fact I just d/l latest version (9.7.9), and I'm using calibre 1.24

The calibre version number in the dc:contributor entries in my opf's appears to reflect the version number when metadata was last changed. Many of them have 0.9.nn in there, some might even have 0.8.nn.

'Tis a pity that there is no option to display the Library metadata backup status counter in the status line, then one could see them being written. You might be able to hear them being written - on my rig when there's a bulk change of opf's happening they are written at about 1 per 2 sec, so the drive sounds like a slow ticking... clock.

Maybe the file system modified timestamp on the opf files can give you a clue as to when they're being written.

Its my understanding that Kovid has not changed the database schema for a number of years, I assume he hasn't changed the opf schema for a similar time.

BR

[robin58]

@eschwartz,

I think you're missing my point. If I have backups and the only difference is the version number inside as detailed above, then surely that wouldn't prevent a restoration? Would it?

@BetterRed,

After my above post I went and ran FreeFileSync again. Now bear in mind I haven't even opened Calibre since earlier.

The differences speak for themselves. First screenshot is the capture of all the files reporting different opf file sizes. Second screenshot is the comparison using UltraCompare set to only show differences between the 2 files (again this is just one file but in checking a dozen they're all the same differences).

Other than the same 2 lines as above, there is a reference to the cover.jpg file in which for some reason has

title="Cover" type="cover"

now instead of

type="cover" title="Cover"

Is it this simple reversal of entries somewhere that's causing this?

[eschwartz]

Originally Posted by robin58

@eschwartz,

I think you're missing my point. If I have backups and the only difference is the version number inside as detailed above, then surely that wouldn't prevent a restoration? Would it?

You should be fine, just so long as you can predict with 100% accuracy that you will never ever EVER update the metadata in your library, nor will you ever add more books, because if so, you will find out the metadata change wasn't backed up, and the new books don't even have the out-of-date OPF your current books have.

And your restore will fail to take this into account, which is somewhat contrary to the purpose of having a backup, if you ask me. You will of course have the actual ebook files, but I doubt you'd enjoy redoing all your metadata.

This assumes that something catastrophic happens to your db, but that's no less likely than something catastrophic happening to your hard drive.

Since the OPF is quite small, and your sync program can probably handle uploading only the changes, why not just let it sync too?

Other than the same 2 lines as above, there is a reference to the cover.jpg file in which for some reason has

now instead of

It just happened to write it in a different order the next time it updated the metadata. I don't know why, as I am not familiar with the rules behind calibre's metadata backup process, but it's no bigger deal than changing the version number on the calibre-as-a-contributor entry.

[BetterRed]

@robin58 - you can look inside the opf files with a text editor - its rebadged xml.

I have this at the end of all my opf's - even if there is no cover

<guide>
        <reference href="cover.jpg" title="Cover" type="cover"/>
</guide>

I just did a test with goodsynch 9.7.9 and calibre 1.24. I forced a bulk change on my Test library, updated metadata on a couple of 'books' I'm editing and added 3 journals - no surprises, only see the synchs I expected.

BR

[BetterRed]

Originally Posted by robin58

And then for the heck of it ran the filesync programs again. I had 549 books that reported changes (incidentally both Goodsynch and FreeFileSync reported the exact same differences).

Are you saying you're running two synch programs against the same files?

If one or both of them change the archive attribute then maybe the other one sees that on the next go round. Also are you doing two way synchs, or one way copies.

Originally Posted by robin58

After my above post I went and ran FreeFileSync again. Now bear in mind I haven't even opened Calibre since earlier.

That would imply that something other than calibre is writing to the opf files - or there is a calibre component running as a daemon. Have a look in Task Manager (better yet Process Explorer) for any other calibre programs. I have no idea which ones - Process Explorer will let you sort tasks by .exe location.

Originally Posted by robin58

The differences speak for themselves. First screenshot is the capture of all the files reporting different opf file sizes. Second screenshot is the comparison using UltraCompare set to only show differences between the 2 files (again this is just one file but in checking a dozen they're all the same differences).

@robin58 - those screen shots don't mean much to me, I'd like to see the file modified dates on the opf files, something like my attachment, besides they're too hard to read. And the mod date on the metadata.db file.

BTW: my screen shot comes from the Browse Flat feature of xplorer2, couldn't live without it - there's a free Lite version that does Browse Flat

BR

[cybmole]

I am a regular user of freefilesync - in one -way ( mirror) mode. it backs up my calibre library & also my calibre app folder to a different hard drive

I have not noticed the issue under discussion. a typical backup finds only a few changed files which seems to equate to recent adds, deletes & edits. I dont think it is re-copying hundreds of opf files

PS I am using the last v5 of FFS, the UI for v6 looked very strange so I rolled back & disabled update notifications
update; tying harder to replicate this.

I ran my FSS mirror ( which writes to 2 separate backup locations ) there were ~100 changes

then I closed calibre & re-ran FFS compare , just a handful of changes detected: to metadata.db & related files
re-opened calibre, ran FFS compare again - same result

there are 2 compare settings in ffs, I use compare filze size & data. it could be that using compare content gives a different result?

I. Compare by File time and size
This variant considers two files with the same name as equal when both modification time and file size match. The following categories are distinguished:

file exists on one side only
left only
right only
file exists on both sides
different date
left newer
right newer
same date
equal
conflict (same date, different size)


II. Compare by File content
Two files with the same name are marked as equal if and only if they have the same content. This option is more useful for consistency checks rather than backup operations since it is naturally slower. The file modification time is not taken into account at all.

[BetterRed]

@cybmole - does FFS use, set or or reset any attributes. Goodsynch can take their values into account in its comparisons? I ask because the OP appears to be using both.

@robin28

Are you using the latest versions of Goodsynch (GS) and FFS, and are you using the 'free' or 'paid' version of GS. I use 'paid' because it has features I want (not for calibre libraries) that aren't available from 'free'.

I wonder if those results from UltraCompare can be 'trusted', to me they don't make any sense.

Yesterday I noticed that GS has the ability to override the text content compare engine, the default is the UNIX 'diff' engine. If you have you 'plugged' UltraCompare into GS (and/or FFS) as the text compare engine, then I suggest you take it out - 'diff' may not be the fastest compare utility on the planet, but not many would doubt its efficacy.

Assuming file is present on both sides, I have GS configured to compare file mod date, size and MD5 checksum to decide if file has changed, I don't compare content.

BR