KindleUnpack (MobiUnpack): Extracts text, images and metadata from Kindle/Mobi files - Page 22

wallcraft · 03-13-2012, 01:00 AM

Quote:

Originally Posted by KevinH

Attached is an update to Mobi_Unpack_v0.45, a python 2.X program which works with both older mobi and newer KF8 mobi formats. It includes a Graphical User Interface frontend.

This is failing on the first KF8 I tried it on. So the problem may be at my end. I get the same failure using ebook-convert in Calibre 0.8.42, presumably from its version of the unpack code.

The ebook is asin B005IEGK5C (Turing's Cathedral: The Origins of the Digital Universe by George Dyson) and I downloaded it to K4PC version 1.9.1. The sample exhibits the problem. What I get is:

Code:

Conversion Log 

Command = "python mobiunpack.py"
Mobi Path = "C:\Users\Alan\Documents\My Kindle Content\B005IEGK5C_EBSP.azw"
Output Path = "C:\Users\Alan\Documents\My eBooks\SOFTWARE\Mobi_Unpack_v045\Mobi_Unpack\Dyson_TC"
Options = "-d"


Please Wait ...

MobiUnpack 0.45
  Copyright (c) 2009 Charles M. Hannum <root@ihack.net>
  With Additions by P. Durrant, K. Hendricks, S. Siebert, fandrieu, DiapDealer, nickredding.
Unpacking Book...
Palm DB type:  BOOKMOBI
Mobi Version:  8
Codec:  utf-8
Title:  Turing&#x2019;s Cathedral: The Origins of the Digital Universe
Palmdoc compression
firstaddl 2c
ncxidx ffffffff
exth flags 1058
skelidx 27
dividx 24
othidx 29
fdst 86


Processing K8 format Ebook ...
Traceback (most recent call last):
  File "lib\mobi_unpack.py", line 1184, in <module>
    sys.exit(main())
  File "lib\mobi_unpack.py", line 1172, in main
    unpackBook(infile, outdir)
  File "lib\mobi_unpack.py", line 736, in unpackBook
    uncompressed_data = zlib.decompress(data, -15)
zlib.error: Error -3 while decompressing data: invalid block type
Warning: Unknown metadata with id 450 found
Warning: Unknown metadata with id 451 found
Warning: Unknown metadata with id 452 found
Warning: Unknown metadata with id 453 found
Warning: Unknown metadata with id 504 found
Warning: Unknown metadata with id 405 found
Warning: Unknown metadata with id 406 found
Warning: Unknown metadata with id 407 found
Warning: Unknown metadata with id 403 found
MetaData from EXTH
{'ISBN': ['978-0-307-90706-6'], '450 (hex)': ['00000000'], 'Title': ['Turing&#x2019;s Cathedral: The Origins of the Digital Universe'], 'Clipping Limit': ['0'], 'Fake Cover': ['0'], 'sample': ['1'], 'Codec': ['utf-8'], 'Font Signature (hex)': ['8700008050000000000000000000008000200000000000000000000000000000edbbfde501bd01f801e701e102d307c64094409340994098409c409d40a240a640b240b340c442d8449e'], 'K8(129)_Masthead/Cover_Image': ['kindle:embed:0001'], 'ASIN': ['B005IEGK5C'], 'Language': ['en-us'], 'Watermark': ['atv:kin:1:e+ilN1YvGlfTYxKEpQ+nVyAtCJ25P7eAI016TPM0E2CyqOi+JBAC8IT4B1fZJb/5xbbW1Js0Pi7R8GfttVlALys7cjmnlad/5WHXjHHDYDIiwuQ2OfpX2S7OpUReCTUKdmIhbJ8DXHUGwjVVbRucJF4EvecYIrN+lZ+U/GPeUOc=:pAS3rR9WtVUcbXsGs/7UPtGGv9M='], 'Rights': ['Copyright &#x00A9; 2012 by George Dyson'], '451 (hex)': ['001e1bb0'], '452 (hex)': ['00000000'], '407 (hex)': ['0000000000000000'], 'StartOffset': ['9368'], 'ThumbOffset': ['89'], 'CDE Type': ['EBSP'], 'Publisher': ['Random House, Inc.'], '504 (hex)': ['423030354945474b3543'], 'Updated Title': ["Turing's Cathedral: The Origins of the Digital Universe"], 'Creator Software': ['301'], 'Tamper Proof Keys (hex)': ['01000000d000000001940000000191000000019500000001960000000197'], 'Published': ['2012-03-06'], 'Text to Speech Disabled': ['0'], 'Creator Major Version': ['2'], 'Creator': ['Dyson, George'], '403 (hex)': ['00'], '406 (hex)': ['0000000000000000'], '405 (hex)': ['00'], '453 (hex)': ['0001edab'], 'Creator Build Number': ['35749'], 'CoverOffset': ['7'], 'K8(125)_Count_of_Resources_Fonts_Images': ['9'], 'UniqueID': ['2698837455'], 'Creator Minor Version': ['2'], 'K8(131)_Unidentified_Count': ['4294967295'], 'Drm Ebookbase Book Id': ['0000000501507']}
Unpack raw markup language
Unpacking images, resources, fonts, etc
firstaddl is  44
num_sections is  136
K8Boundary is  -1
Section is  44
    extracting resource:  resc00001.dat
Section is  45
    extracting image:  image00002.jpeg
Section is  46


Error: Unpacking Failed

kovidgoyal · 03-13-2012, 02:05 AM

@wallcraft: See https://bugs.launchpad.net/bugs/953260

DiapDealer · 03-13-2012, 09:56 AM

Perhaps some fonts are being compressed with something other than zlib? Or different zlib compression flags. Easy enough to work around, but I'd like to discover the reason they're failing to uncompress in the long run.

kovidgoyal · 03-13-2012, 11:26 AM

In their publishing guidelines amazon claim they obfuscate fonts in kf8, that would likely be the cause.

pdurrant · 03-13-2012, 02:14 PM

Quote:

Originally Posted by kovidgoyal

In their publishing guidelines amazon claim they obfuscate fonts in kf8, that would likely be the cause.

Oh, that sounds very likely. I wonder what algorithm they're using. I guess they must do it at their end, as fonts aren't obfuscated in the output of Kindlegen.

DiapDealer · 03-13-2012, 02:48 PM

Quote:

Oh, that sounds very likely. I wonder what algorithm they're using. I guess they must do it at their end, as fonts aren't obfuscated in the output of Kindlegen.

I think it's done in the ePub, actually. But maybe it's different in this case—since even obfuscated fonts continue to work in epub readers, you just can't extract them and use them for anything else (regardless of DRM).

Hmm... never mind. It must be something else (or different obfuscation). ePubs using either Adobe's or IDPF's obfuscation (applied with Sigil) result in embedded fonts not working at all when run through the latest kindlegen.

kovidgoyal · 03-14-2012, 03:06 AM

The first 1040 bytes of the *compressed* font are XORed with a (seemingly randomnly generated) key. I have committed code to calibre to de-obfuscate them. Given that the MOBI format itself is a much harder form of obfuscation, I have no idea why amazon felt the need to implement a XOR on top of it, but then I have no idea why Amazon makes have the technical decisions it does...

pdurrant · 03-14-2012, 03:25 AM

Quote:

Originally Posted by kovidgoyal

The first 1040 bytes of the *compressed* font are XORed with a (seemingly randomnly generated) key. I have committed code to calibre to de-obfuscate them. Given that the MOBI format itself is a much harder form of obfuscation, I have no idea why amazon felt the need to implement a XOR on top of it, but then I have no idea why Amazon makes have the technical decisions it does...

1040 bytes sounds like the IDPF font obfuscation algorithm.
http://idpf.org/epub/20/spec/FontMan....0.1_draft.htm
http://idpf.org/epub/30/spec/epub30-...nt-obfuscation

In which case, the key might also be the one specified in those documents. But since the XOR key is given in the file, I suppose it doesn't matter whether it is or not.

I'm not sure that de-obfustication code should go in calibre, unless any output is re-obfusticated. Simple though it is, I think it could be argued that the obfuscation is a digital rights management method, and that removing it might make calibre an illegal bit of software under the DMCA.

kovidgoyal · 03-14-2012, 03:55 AM

There is no way to obfuscate a font in an open source application (for the same reason that it is impossible to implement DRM in an open source application). If someone wishes to argue that therefore open source applications must never be allowed to access embedded fonts in any document, let them make that argument, we'll see how far they can take it.

jhowell · 03-23-2012, 04:42 PM

Any chance for an update to MobiUnpack to handle KF8 files with embedded obfuscated fonts?

About 10% of the KF8 files I have tried from Amazon error out when I try to unpack them using version 0.45. Almost all of them display properly in the latest versions of Calibre.

pdurrant · 03-23-2012, 04:50 PM

I beleive one is in the works.

KevinH · 03-23-2012, 07:34 PM

Hi All,

Attached is the very latest version of Mobi_Unpack.py (v047) which adds support for obfuscated fonts and opentype fonts plus a number of other bug fixes.

Have Fun!

KevinH

lizcastro · 04-09-2012, 01:47 PM

Hi you all. Thank you so much for developing this tool. I have found it very useful.

One doubt. When I double-click the Mobi_Unpack.pyw file on Mac, it asks me for the app I want to open it with. So instead, I go to Terminal, navigate to the MU folder, type python and then type Mobi_Unpack.pyw. And then I get the GUI.

What am I not doing to be able to double-click?

(I'm about to write up my workaround, but I hate feeling like an idiot for not knowing the normal way!

thanks again,
Liz

DiapDealer · 04-09-2012, 01:55 PM

I think you just need to associate .PYW files with the Python launcher... which may be what it's prompting you to do. Someone with more Mac experience will have to provide the details, but I'm certain it's fairly trivial.

lizcastro · 04-09-2012, 02:05 PM

Python Launcher! Didn't know it existed. Google and this page helped me find it:

http://mail.python.org/pipermail/pyt...ry/579790.html

Once I drag Mobi_Unpack_v047.pyw to it the first time, it remembers it for the next time. Excellent!

Thanks. Sometimes you just need to know the right question to ask...

best,
Liz

04-09-2012, 01:47 PM	#328
lizcastro Member Posts: 16 Karma: 148 Join Date: Apr 2010 Device: iPad, NOOK, Kindle, Kobo	Mobi Unpack usage Hi you all. Thank you so much for developing this tool. I have found it very useful. One doubt. When I double-click the Mobi_Unpack.pyw file on Mac, it asks me for the app I want to open it with. So instead, I go to Terminal, navigate to the MU folder, type python and then type Mobi_Unpack.pyw. And then I get the GUI. What am I not doing to be able to double-click? (I'm about to write up my workaround, but I hate feeling like an idiot for not knowing the normal way! thanks again, Liz

04-09-2012, 01:55 PM	#329
DiapDealer Grand Sorcerer Posts: 27,546 Karma: 193191846 Join Date: Jan 2010 Device: Nexus 7, Kindle Fire HD	I think you just need to associate .PYW files with the Python launcher... which may be what it's prompting you to do. Someone with more Mac experience will have to provide the details, but I'm certain it's fairly trivial. Last edited by DiapDealer; 04-09-2012 at 02:18 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Can i rotate text and insert images in Mobi and EPUB?	JanGLi	Kindle Formats	5	02-02-2013 04:16 PM
PDF to Mobi with text and images	pocketsprocket	Kindle Formats	7	05-21-2012 07:06 AM
Mobi files - images	DWC	Introduce Yourself	5	07-06-2011 01:43 AM
pdf to mobi... creating images rather than text	Dumhed	Calibre	5	11-06-2010 12:08 PM
Transfer of images on text files	anirudh215	PDF	2	06-22-2009 09:28 AM

03-13-2012, 02:05 AM	#317
kovidgoyal creator of calibre Posts: 43,842 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	@wallcraft: See https://bugs.launchpad.net/bugs/953260

03-13-2012, 09:56 AM	#318
DiapDealer Grand Sorcerer Posts: 27,546 Karma: 193191846 Join Date: Jan 2010 Device: Nexus 7, Kindle Fire HD	Perhaps some fonts are being compressed with something other than zlib? Or different zlib compression flags. Easy enough to work around, but I'd like to discover the reason they're failing to uncompress in the long run.

03-13-2012, 11:26 AM	#319
kovidgoyal creator of calibre Posts: 43,842 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	In their publishing guidelines amazon claim they obfuscate fonts in kf8, that would likely be the cause.

03-14-2012, 03:06 AM	#322
kovidgoyal creator of calibre Posts: 43,842 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	The first 1040 bytes of the compressed font are XORed with a (seemingly randomnly generated) key. I have committed code to calibre to de-obfuscate them. Given that the MOBI format itself is a much harder form of obfuscation, I have no idea why amazon felt the need to implement a XOR on top of it, but then I have no idea why Amazon makes have the technical decisions it does...

03-14-2012, 03:55 AM	#324
kovidgoyal creator of calibre Posts: 43,842 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	There is no way to obfuscate a font in an open source application (for the same reason that it is impossible to implement DRM in an open source application). If someone wishes to argue that therefore open source applications must never be allowed to access embedded fonts in any document, let them make that argument, we'll see how far they can take it.

03-23-2012, 04:42 PM	#325
jhowell Grand Sorcerer Posts: 6,496 Karma: 84420419 Join Date: Nov 2011 Location: Tampa Bay, Florida Device: Kindles	Any chance for an update to MobiUnpack to handle KF8 files with embedded obfuscated fonts? About 10% of the KF8 files I have tried from Amazon error out when I try to unpack them using version 0.45. Almost all of them display properly in the latest versions of Calibre.

03-23-2012, 04:50 PM	#326
pdurrant The Grand Mouse 高貴的老鼠 Posts: 71,495 Karma: 306214458 Join Date: Jul 2007 Location: Norfolk, England Device: Kindle Voyage	I beleive one is in the works.

04-09-2012, 02:05 PM	#330
lizcastro Member Posts: 16 Karma: 148 Join Date: Apr 2010 Device: iPad, NOOK, Kindle, Kobo	Python Launcher! Didn't know it existed. Google and this page helped me find it: http://mail.python.org/pipermail/pyt...ry/579790.html Once I drag Mobi_Unpack_v047.pyw to it the first time, it remembers it for the next time. Excellent! Thanks. Sometimes you just need to know the right question to ask... best, Liz

Advert

Advert