Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Formats > Kindle Formats

Notices

Reply
 
Thread Tools Search this Thread
Old 06-10-2014, 04:06 PM   #61
KevinH
Guru
KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.
 
Posts: 790
Karma: 308454
Join Date: Nov 2009
Device: many
Hi,

Yes, that script was designed with Kindlegen and Calibre in mind. Kindlegen allocates a full section of up to 8192 (or 4096?) bytes of nulls when it creates its headers. Calibre does the same to match what Kindlegen does. Simply run DumpMobiHeader_v14.py or later to see all of the extra null bytes that pad out the header section after the full title field.

As I am away from my workstation please run DumpMobiHeader on your sample files to verify if the headers are properly sized and padded.

The reason kindlegen does this is to make sure there is enough space in the header section to add in the DRM key info (and there can be multiple keys) and the extra EXTH metadata needed to store the DRM key info all on the fly when downloading.

The dual-meta mobi editor uses this fact to add and remove EXTH info but keeps the entire section size exactly the same. By doing it this way, you do not need to update the palm section table offsets at all nor change the size of the section table at all. I can just replace the entire header section with the new one.

As a safety check, I verify that all I am replacing at the end of the header are these extra padding null bytes and that I have not grown the header section in any way. I did this because some very very old mobis left no extra space. Everything since Kindlegen 1 adds this padding.

So I think that any metadata editing program should be careful to keep the required extra padding (ie size the header to allow additions without the need to resize.) just like Kindlegen and Calibre do.

Hope this helps.

KevinH

Last edited by KevinH; 06-10-2014 at 04:08 PM.
KevinH is offline   Reply With Quote
Old 06-10-2014, 11:36 PM   #62
KevinH
Guru
KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.
 
Posts: 790
Karma: 308454
Join Date: Nov 2009
Device: many
Hi,

I ran DumpMobiHeader_v015.py on your sample files. It seems the Mobi Meta Editor does improperly trim the padding placed there by kindlegen and calibre which is what causes the problem:

Here is the original section map:
Code:
Map of Palm DB Sections
    Dec  - Hex : Description
    ---- - ----  -----------
    0000 - 0000: HEADER 6 [8736]
    0001 - 0001: Text Record 0 [866]
    0002 - 0002: 0000  [2]
    0003 - 0003: NCX Index 0 [236]
    0004 - 0004: NCX Index 1 [208]
    0005 - 0005: NCX Index CNX [8]
    0006 - 0006: Image jpeg [8956]
    0007 - 0007: RESC [4112]
    0008 - 0008: FLIS [36]
    0009 - 0009: FCIS [44]
    0010 - 000a: Source Archive 0 [1612]
    0011 - 000b: BOUNDARY [8]
    0012 - 000c: HEADER 8 [8736]
    0013 - 000d: Text Record 0 [1587]
    0014 - 000e: 00  [1]
    0015 - 000f: Fragment Index 0 [248]
    0016 - 0010: Fragment Index 1 [220]
    0017 - 0011: Fragment Index CNX [16]
    0018 - 0012: Skeleton Index 0 [244]
    0019 - 0013: Skeleton Index_Index 1 [224]
    0020 - 0014: NCX Index 0 [240]
    0021 - 0015: NCX Index 1 [212]
    0022 - 0016: NCX Index CNX [8]
    0023 - 0017: FDST [28]
    0024 - 0018: FLIS [36]
    0025 - 0019: FCIS [44]
    0026 - 001a: DATP [52]
    0027 - 001b: EOF_RECORD [4]
Notice the size of each header is 8736 bytes long. This is made up of 16 bytes of extra info at the start followed by 264 bytes (0x108) bytes of header information plus the EXTH info (264 bytes) leaving 8192 (0x2000) bytes of padding for the DRM Keys, extra metadata, etc.

Here it is again after passing through MME
Code:
Map of Palm DB Sections
    Dec  - Hex : Description
    ---- - ----  -----------
    0000 - 0000: HEADER 6 [572]
    0001 - 0001: Text Record 0 [866]
    0002 - 0002: 0000  [2]
    0003 - 0003: NCX Index 0 [236]
    0004 - 0004: NCX Index 1 [208]
    0005 - 0005: NCX Index CNX [8]
    0006 - 0006: Image jpeg [8956]
    0007 - 0007: RESC [4112]
    0008 - 0008: FLIS [36]
    0009 - 0009: FCIS [44]
    0010 - 000a: Source Archive 0 [1612]
    0011 - 000b: BOUNDARY [8]
    0012 - 000c: HEADER 8 [8736]
    0013 - 000d: Text Record 0 [1587]
    0014 - 000e: 00  [1]
    0015 - 000f: Fragment Index 0 [248]
    0016 - 0010: Fragment Index 1 [220]
    0017 - 0011: Fragment Index CNX [16]
    0018 - 0012: Skeleton Index 0 [244]
    0019 - 0013: Skeleton Index_Index 1 [224]
    0020 - 0014: NCX Index 0 [240]
    0021 - 0015: NCX Index 1 [212]
    0022 - 0016: NCX Index CNX [8]
    0023 - 0017: FDST [28]
    0024 - 0018: FLIS [36]
    0025 - 0019: FCIS [44]
    0026 - 001a: DATP [52]
    0027 - 001b: EOF_RECORD [4]
Notice that the Mobi 6 header is shrunk greatly in size to just 572 bytes (all of the extra padding is removed). Also it is clear MME does not understand how to deal with the Mobi 8 (azw3) part and ignores it.

So this is definitely an MME bug. It really needs to not remove all of the padding added to the end of the header information which is used to store the DRM Keys, and provides room for all of the extra DRM related Meta information to be added on the fly (as Amazon does).


Hope this helps,

KevinH
KevinH is offline   Reply With Quote
 
Enthusiast
Old 06-11-2014, 01:22 AM   #63
Doitsu
Wizard
Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.
 
Doitsu's Avatar
 
Posts: 1,850
Karma: 4630359
Join Date: Dec 2010
Device: Kindle PW2
Quote:
Originally Posted by KevinH View Post
I ran DumpMobiHeader_v015.py on your sample files. It seems the Mobi Meta Editor does improperly trim the padding placed there by kindlegen and calibre which is what causes the problem.
Thanks for testing this.



I'd already suspected that Mobi Meta Editor was to blame, because files directly converted from the source worked just fine.

If you ever officially release this tool, maybe you could mention in the release notes that it'll most likely fail with .mobi files whose metadata was modified with Mobi Meta Editor.

(Since the author of Mobi Meta Editor seems to no longer visit MR, chances are slim that he'll fix this bug.)
Doitsu is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Recommended settings to convert dual-column PDF to useable MOBI format Cephas Atheos Conversion 7 09-18-2012 07:32 AM
Insert metadata as page at start of book adds does not replace (mobi to mobi) linusnc Calibre 2 07-19-2012 03:54 PM
Update Mobi header/file metadata without doing a Mobi to Mobi conversion RecQuery Conversion 2 06-30-2012 11:43 AM
EPUB (CSS) tweaker app Loccy Conversion 9 01-23-2011 10:22 PM
Firefox Tweaker: Flexbeta FireTweaker XP Alexander Turcic Lounge 0 08-16-2004 04:51 AM


All times are GMT -4. The time now is 09:58 AM.


MobileRead.com is a privately owned, operated and funded community.