Problem with epub zip files sent via Wireless Device?

[JimmXinu]

I'm seeing problems where some of the epubs sent to CC in my phone from Calibre have (slightly) damaged zip files when they arrive.

(For anyone who didn't know, an epub is a zip file with a bit of standardized structure.)

I say '(slightly)' because some book readers, including Calibre, can still read them.

I noticed the problem from Android app @Voice Aloud Reader reporting "This ebook file appears to be damaged" for some, but not all, of my recent uploads. When I reported it to them with the epub, they pointed out the damaged zip.

I'm using Calibre 4.18 on Windows 10 and Calibre Companion 5.4.4.10 (401) on Android 10(?) on a Samsung Galaxy S10e, connecting as a Wireless Device.

Attached is one of the damaged epubs as copied from the CC dir on the phone and exhibits the following errors. The same book taken from the Calibre dir directly, saved from Calibre, and sent to a Nook from Calibre has no issues.

Errors:

Spoiler:

When added to Calibre, debug reports this, but Calibre otherwise silently fixes it:

Added cc2 Live Forever to db in: 0.1
Added 1 books in 0.3 seconds
Worker Launch took: 0.0840001106262
WARNING: Header ('content.opf') and directory ('META-INF/container.xml') filenames do not match inside ZipFile
Using directory filename 'META-INF/container.xml'
Traceback (most recent call last):
  File "site-packages\calibre\customize\ui.py", line 428, in get_file_type_metadata
  File "site-packages\calibre\customize\builtins.py", line 195, in get_metadata
  File "site-packages\calibre\ebooks\metadata\epub.py", line 217, in get_metadata
  File "site-packages\calibre\ebooks\metadata\epub.py", line 165, in get_zip_reader
  File "site-packages\calibre\ebooks\metadata\epub.py", line 148, in __init__
  File "site-packages\calibre\ebooks\metadata\epub.py", line 99, in __init__
  File "site-packages\calibre\ebooks\metadata\epub.py", line 153, in open
  File "site-packages\calibre\utils\zipfile.py", line 999, in read
  File "site-packages\calibre\utils\zipfile.py", line 623, in read
  File "site-packages\calibre\utils\zipfile.py", line 685, in read1
  File "site-packages\calibre\utils\zipfile.py", line 640, in _update_crc
BadZipfile: Bad CRC-32 for file u'META-INF/container.xml'

unzip -t:

unzip -t "..\..\cc2 Live Forever - Petrificus Somewhatus.epub"
Archive:  ..\..\cc2 Live Forever - Petrificus Somewhatus.epub
    testing: mimetype                 OK
META-INF/container.xml:  mismatching "local" filename (content.opf),
         continuing with "central" filename version
    testing: META-INF/container.xml   OK
file #3:  bad zipfile offset (local header sig):  263
file #4:  bad zipfile offset (local header sig):  3224
file #5:  bad zipfile offset (local header sig):  3586
file #6:  bad zipfile offset (local header sig):  3931
file #7:  bad zipfile offset (local header sig):  4574
file #8:  bad zipfile offset (local header sig):  16728
At least one error was detected in ..\..\cc2 Live Forever - Petrificus Somewhatus.epub.

Any help getting this fixed would be appreciated.

[ottischwenk]

Originally Posted by JimmXinu

I'm seeing problems where some of the epubs sent to CC in my phone from Calibre have (slightly) damaged zip files when they arrive.

Why should this error be in CC?
Standard routines from the operating system are used here.
If so, then the problem lies there, or else in a hardware defect.

[JimmXinu]

Originally Posted by ottischwenk

Why should this error be in CC?
Standard routines from the operating system are used here.
If so, then the problem lies there, or else in a hardware defect.

I don't know that it is. Which is why I described this as a problem I am having, not a bug. I'm hoping that this is something seen before in the CC community with a ready fix. I will be perfectly happy if it's something I'm doing wrong, but I would like to get it fixed.

I have seen the same epub sent without issue through the other parts of the path (Calibre) and if it was a hardware defect on the phone I would expect to see low level read errors, not just zip file errors.

The fact Calibre can still read these and silently fixes them suggests to me that it may be a differing OS/library thing of some kind. I will be looking at the Calibre code for inspiration, but obviously I can't do that with CC.

[stumped]

But cc does not open the zip( the eoub). It simply files it and hands a copy to reader app as needed. So it can't be the cause of the corruption.

But anyway. There are 2 connection options. Content server and wireless device. See if both corrupt or only one of the two. That may help narrow things down.
And also try getting the books from calibre using the device browser to connect. Type in the calibre content server URL.. Not using cc at all.
Compare what happens with all 3 routes

Ps what app is actually reading the book. Try repairing, reinstalling that app.

[chaley]

Originally Posted by stumped

But cc does not open the zip( the epub). It simply files it and hands a copy to reader app as needed. So it can't be the cause of the corruption.

Unless something has changed, this is true. CC doesn't ever "process" the epub. It simply copies the network stream to the file. If that reliably didn't work then CC would have far greater problems than you are seeing.

I suspect that something on the calibre side is borking the epub when it is created to be copied. When sending via a device, calibre creates a new copy, applies plugboards and cover changes, embeds new metadata, then hands it to the device driver (the wireless device). Do you use any plugins that could get in the middle of that, such as an epub metadata writer?

[stumped]

Originally Posted by chaley

Unless something has changed, this is true. CC doesn't ever "process" the epub. It simply copies the network stream to the file. If that reliably didn't work then CC would have far greater problems than you are seeing.

I suspect that something on the calibre side is borking the epub when it is created to be copied. When sending via a device, calibre creates a new copy, applies plugboards and cover changes, embeds new metadata, then hands it to the device driver (the wireless device). Do you use any plugins that could get in the middle of that, such as an epub metadata writer?

Good hypothesis, if it is calibre side, then leaving out cc and using a device browser to fetch from calibre should show the same corruption
I can't type the URL syntax for that test from memory,. But it is easily found.

[JimmXinu]

I was not aware that CC doesn't open the zip file. As you say, that likely points back to code in Calibre instead. I have a couple metadata plugboards that are applied that I've used for years, but no metadata modifying plugins.

I'm not able to find anything about manually testing the wireless device interface. Can I get a pointer or a phrase to google for that? Pointing my browser at ipaddr:9090 gets some basic library metadata in JSON after a lengthy (~5minute) wait.

I am assuming (possibly wrongly) that the 'smart device' AKA wireless device interface is different than the Content Server.

I also noticed that the corrupted zip files were ~2x the size of the correct one.

Of course, now I can't duplicate the problem even sending the same file from Calibre. So that helps a lot. And now I can't find any other cases of it, just to insure I look crazy.


Until I can duplicate the problem again, it's basically impossible to troubleshoot. So I'll go away again for now.

Thanks for your help, folks.

[chaley]

Originally Posted by JimmXinu

Of course, now I can't duplicate the problem even sending the same file from Calibre. So that helps a lot. And now I can't find any other cases of it, just to insure I look crazy.

Main screen turn on. "How are you, Gentlemen? All your base are belong to us. You have no chance to survive. Make your time. Ha ha ha."

I'm not able to find anything about manually testing the wireless device interface. Can I get a pointer or a phrase to google for that? Pointing my browser at ipaddr:9090 gets some basic library metadata in JSON after a lengthy (~5minute) wait.

You won't have joy. The wireless protocol is reasonably complicated. It is a "push" interface, requiring the client to respond in certain ways at certain times.

Koreader has an independently-written client. If it is reasonable for you to install it then the next time something breaks you could use koreader to see if it breaks in the same way. You could also look at the code of this client, given that it is open source. You can also look at the code for the wireless device driver.

I am assuming (possibly wrongly) that the 'smart device' AKA wireless device interface is different than the Content Server.

Yes. The two have little to do with each other. The point of commonality is that calibre will prepare the file to be transferred in a similar way.

I also noticed that the corrupted zip files were ~2x the size of the correct one.

Now that is interesting. A protocol error would result in byte-for-byte corruption. Doubling the number of bytes points strongly to a problem at the source.

Thanks for your help, folks.

You are welcome. It is little enough payback for all you do for the calibre community.

[JimmXinu]

Originally Posted by chaley

You are welcome. It is little enough payback for all you do for the calibre community.

Kind words from one of the giants of the community.

[JimmXinu]

Okay, I believe I've tracked down this problem and I believe that it is with the Calibre Companion android app.

It looks like CC, when writing an epub that already exists, overwrites the existing file but doesn't truncate the file if the new epub is shorter.

I'll spare you the details of how I got there and just tell you about the proof. I added debug output Calibre to report the bytes sent, and save the temp epub created to be sent.

After picking one epub, I:

1. Made a file of all zeros, longer than the epub;
2. Put it in place in CC's directory for that epub;
3. Sent that epub to CC via wireless device;
4. Confirmed that the epub in CC's dir had not shrunk;
5. Confirmed that the file in CC's dir contained the same bytes as the sent epub, followed by zeros out to the length of the all-zeros file I had written.

I ran the same test on koreader and did not see this issue.

Before you ask, I have checked and in all the cases I can still find the files for, the epub was legitimately shorter due to text or cover image changes.

The only user-level work around I see right now is to explicitly delete each book before sending it.

If there is a better place I should be reporting this to make sure the new CC team see it, please let me know.

[chaley]

Originally Posted by JimmXinu

It looks like CC, when writing an epub that already exists, overwrites the existing file but doesn't truncate the file if the new epub is shorter.

Looking at the old code, I think this can happen if your file is on an SD card/managed by the Android SAF. The documentation for opening a file for write doesn't say whether or not the file is trucated. Some posts on Stack Overflow and your experience says that it isn't.

Good hunting!

[JimmXinu]

I confess to complete ignorance of the Android SAF.

But I do keep my CC files on 'Phone' instead of 'SD Card', FWIW.

[chaley]

Originally Posted by JimmXinu

But I do keep my CC files on 'Phone' instead of 'SD Card', FWIW.

If you are running Android 10 and if you changed the CC folder name then you are using the SAF. And it is possible that the new devs have removed that one exception; using the default folder CC creates at install time. And thinking about it, since they are making releases they must have changed it, because they must "target" a version of Android that forces use of the SAF.

[JimmXinu]

I had to change from SD Card to Phone to avoid some other permissions problem a while back, so I think you are doubly correct.

Right now I'm trying to figure out why FBReader (on Android) and Calibre's reader work fine with the corrupt zip files.

In Calibre's case, it looks like it falls back to calibre.utils.localunzip which ignores the fact the zip central dir structure isn't in the right place and reads the 'local' filenames.

[JimmXinu]

FYI, I saw there was a CC android app update recently; so I went back and tested for this problem again.

It's still happening for me.