PDF Corruption during SEND TO DEVICE

mikefulton · 08-30-2014, 03:31 PM

I'm having a weird problem with certain PDF files.

Example is "GOOGLE MAPS HACKS" from O'Reilly. It's DRM-free, purchased and downloaded from their website, then imported into CALIBRE.

If I open the file (from the Calibre library folder) in ADOBE READER, it works fine. So there's clearly nothing wrong with the PDF as it sits in the library. Windows Explorer says the file size is 13,110,462 bytes.

However, if I copy the file to my Kindle Fire HDX using Calibre's SEND TO DEVICE feature, the file won't open on the device. When I browse to the Kindle/Internal Storage/Documents folder with Windows Explorer, it says the file size is 13,115,779 bytes, an increase of 5317 bytes over the original.

If I copy the file from the Calibre library folder to the Kindle's DOCUMENTS folder using WINDOWS EXPLORER, then it opens fine on the device, and the file size is 13,110,462 bytes as expected.

This doesn't happen with all PDF files. The majority work just fine. However, this problem is occurring with perhaps 20% of them, so it's not insignificant.

The inescapable conclusion is that CALIBRE is somehow modifying and breaking the file when it sends it to the Kindle. What is it doing, and how do I make it stop?

kovidgoyal · 08-30-2014, 03:39 PM

When you send any file to a device, metadata in it is updated. If you dont want it to be updated, then disable the metadata writer plugin for that particular format in preferences->plugins.

As for the PDF being corrupted, I highly doubt it. Copy the "corrupted" PDF of your device and try opening it in adobe reader, it will open. If it does not open, then open a bug report and attach the before and after PDF.

mikefulton · 08-30-2014, 06:13 PM

Quote:

Originally Posted by kovidgoyal

When you send any file to a device, metadata in it is updated. If you dont want it to be updated, then disable the metadata writer plugin for that particular format in preferences->plugins.

As for the PDF being corrupted, I highly doubt it. Copy the "corrupted" PDF of your device and try opening it in adobe reader, it will open. If it does not open, then open a bug report and attach the before and after PDF.

Trying to open them in Adobe Reader is what I was doing. After I noticed the difference in file size, I copied the file back off the device and tried opening it. Reader would give me a message saying there was something wrong with the file.

Unfortunately, it's not doing it with every file, and I didn't keep any samples when I was trying to figure things out. But when I can nail down an example, I'll post a bug report with the before-and-after files.

It seems like it's related to caching in some way. I deleted a PDF from the library, but when I relaunched Calibre and double-clicked the title, the file reappeared but with an updated time-stamp, like it had just been recreated somehow.

It's worth noting that I had no problems at all until after installing v2.0. I was using 1.48 before that and hadn't noticed any similar issues.

BetterRed · 08-30-2014, 06:38 PM

@mikefulton - did you try doing as Kovid suggested

Quote:

Originally Posted by kovidgoyal

When you send any file to a device, metadata in it is updated. If you dont want it to be updated, then disable the metadata writer plugin for that particular format in preferences->plugins.

See attachment

My experience is that some PDF's already have dubious embedded metadata, and when calibre injects more it exacerbates the 'dubiousity'. I've never had a corrupt file as a result, but the embedded metadata can end up in a bit of mess. When that happens I clear the existing metadata from the PDF with exiftool and repeat the Save to Disk, Send to Device, Embed Metadata operation.

BR

mikefulton · 08-30-2014, 07:52 PM

I believe I've nailed down the problem, and it's only peripherally related to the metadata.

The Kindle Fire HDX doesn't like PDF files that have an "&" ampersand character in the filename.

If there is an ampersand in the filename, like when there are two authors, the file is shown in the list, but it will not open.

If the file has the NEW icon, it goes away, so it's clearly looking at the file and then returning to the list without showing it.

If you edit the metadata in Calibre such that the ampersand goes away in the filename when you send to the device again, then the file will open properly.

This doesn't appear to be an issue for MOBI or AZW3 files, only PDF.

So, not really an issue with Calibre so much as with the Kindle's software, although it might be a good idea to have Calibre do a character substitution in Calibre the way it already does for colons and certain other characters.

This doesn't address the question of why I ended up with PDF files that wouldn't open in Acrobat, but I saw that early in the whole process of researching this, and so far I've not been able to reproduce it. Not gonna worry about it until I do.

08-30-2014, 03:31 PM	#1
mikefulton Member Posts: 17 Karma: 10 Join Date: Aug 2014 Device: Kindle Fire HDX	PDF Corruption during SEND TO DEVICE I'm having a weird problem with certain PDF files. Example is "GOOGLE MAPS HACKS" from O'Reilly. It's DRM-free, purchased and downloaded from their website, then imported into CALIBRE. If I open the file (from the Calibre library folder) in ADOBE READER, it works fine. So there's clearly nothing wrong with the PDF as it sits in the library. Windows Explorer says the file size is 13,110,462 bytes. However, if I copy the file to my Kindle Fire HDX using Calibre's SEND TO DEVICE feature, the file won't open on the device. When I browse to the Kindle/Internal Storage/Documents folder with Windows Explorer, it says the file size is 13,115,779 bytes, an increase of 5317 bytes over the original. If I copy the file from the Calibre library folder to the Kindle's DOCUMENTS folder using WINDOWS EXPLORER, then it opens fine on the device, and the file size is 13,110,462 bytes as expected. This doesn't happen with all PDF files. The majority work just fine. However, this problem is occurring with perhaps 20% of them, so it's not insignificant. *The inescapable conclusion is that CALIBRE is somehow modifying and breaking* the file when it sends it to the Kindle. What is it doing, and how do I make it stop?**

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Calibre Companion PDF Corruption	asacamano	Calibre Companion	2	02-01-2013 09:30 AM
Moon + reader pdf corruption problem (Solved)	daverush	Android Devices	0	09-30-2012 01:05 AM
Send to device rules - Android device	brainvision	Devices	6	08-16-2012 12:10 AM
Cannot send to SD but can send to device	spintellectual	Devices	3	07-13-2012 11:40 AM
PDF to PDF corruption	PoP	Conversion	5	09-10-2011 11:12 AM

08-30-2014, 03:39 PM	#2
kovidgoyal creator of calibre Posts: 43,850 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	When you send any file to a device, metadata in it is updated. If you dont want it to be updated, then disable the metadata writer plugin for that particular format in preferences->plugins. As for the PDF being corrupted, I highly doubt it. Copy the "corrupted" PDF of your device and try opening it in adobe reader, it will open. If it does not open, then open a bug report and attach the before and after PDF.

08-30-2014, 07:52 PM	#5
mikefulton Member Posts: 17 Karma: 10 Join Date: Aug 2014 Device: Kindle Fire HDX	I believe I've nailed down the problem, and it's only peripherally related to the metadata. The Kindle Fire HDX doesn't like PDF files that have an "&" ampersand character in the filename. If there is an ampersand in the filename, like when there are two authors, the file is shown in the list, but it will not open. If the file has the NEW icon, it goes away, so it's clearly looking at the file and then returning to the list without showing it. If you edit the metadata in Calibre such that the ampersand goes away in the filename when you send to the device again, then the file will open properly. This doesn't appear to be an issue for MOBI or AZW3 files, only PDF. So, not really an issue with Calibre so much as with the Kindle's software, although it might be a good idea to have Calibre do a character substitution in Calibre the way it already does for colons and certain other characters. This doesn't address the question of why I ended up with PDF files that wouldn't open in Acrobat, but I saw that early in the whole process of researching this, and so far I've not been able to reproduce it. Not gonna worry about it until I do.