EPUBs from calibre and calibre-server differ

[beedaddy]

I noticed that the exported EPUB from calibre and the downloaded EPUB from calibre-server differ. It's always content.opf that differ, maybe because calibre (desktop) always updates that part during export and calibre-server doesn't?

I would like to know whether it is possible to get identical results (which would be good for koreader sync of reading positions).

[kovidgoyal]

calibre-server also updates metadata when exporting. What exactly is the difference use the calibre diff tool to check.

[beedaddy]

I depends. Sometimes it's just the different position (line) in the content.opf. In that case, I did a "update metadata in the book files" beforehand. And sometimes one differs in rec_index or still contains (deleted) custom column data. I've attached two diff-screenshots.

[kovidgoyal]

That's just some lines changing position not an actual difference. For the second screenshot, custom column metadata is deleted presumably because the EPUB is now in a library without that cusotm column.

[beedaddy]

During testing, I accidentially deactivated "update metadata in saved copies". So here's an update of the second screenshot (book).

[JSWolf]

Originally Posted by beedaddy

During testing, I accidentially deactivated "update metadata in saved copies". So here's an update of the second screenshot (book).

I don't have Save to disk update the metadata. I prefer to have clean copies.

[beedaddy]

Originally Posted by beedaddy

During testing, I accidentially deactivated "update metadata in saved copies". So here's an update of the second screenshot (book).

If both would behave identically, i.e. both files would have the same md5 hash, it would simplify things for the KOReader. Synchronization of the reading position, for example, if the book on device A comes from calibre-server and on device B from calibre (desktop).

[kovidgoyal]

Both use the exact same code to update metadata, so I don't see how they could possibly behave differently. Most likely updating metadata will shuffle some lines around since it works by first deleting metadata entries to reset and then creating them. Trying to make the updates in place or avoid the deletes completely is way too much work for me, patches welcome. KOReader really should be hashing only the spine and the actual content files, not the entire epub. Indeed simply unpacking and repacking an epub can yield a file with a different hash, but identical actual contents. This is because EPUB is a zip container and zip stores unused per file metadata such as timestamps. File has is absolutely the wrong way to be checking for file equality with EPUB.

[beedaddy]

I understand your point about the hashes and absolutely agree. But if I export the book 10x with Calibre (desktop), I get the same file (same hash A) each time. And if I download it 10 times via Calibre (server), I also get the slightly different file 10 times identically (same hash B). Hence my assumption that something is different after all.

(Regarding KOReader, I could use not the binary based hash but filename based hash. But then the hassle starts to get them equal, and I can't configure the calibre-server "offered" filename at all.)

[kovidgoyal]

Like I said they both use the exact same code, indeed updating metadata in EPUB files is implemented in only one place in all of calibre. You are likely seeing changes becuase of metadata update itself not being byte identical even when nothing is actually changed.

[DNSB]

Perhaps Polish or ModifyEpub could be used to update the metadata in the library copy so the copy being sent does not need to be updated?

[kovidgoyal]

Originally Posted by DNSB

Perhaps Polish or ModifyEpub could be used to update the metadata in the library copy so the copy being sent does not need to be updated?

The problem is the update happens regardless. And updating will change the file in the "byte identical" sense even if the actual metadata doesnt change.