06-10-2014, 04:06 PM | #61 |
Sigil Developer
Posts: 7,506
Karma: 5433350
Join Date: Nov 2009
Device: many
|
Hi,
Yes, that script was designed with Kindlegen and Calibre in mind. Kindlegen allocates a full section of up to 8192 (or 4096?) bytes of nulls when it creates its headers. Calibre does the same to match what Kindlegen does. Simply run DumpMobiHeader_v14.py or later to see all of the extra null bytes that pad out the header section after the full title field. As I am away from my workstation please run DumpMobiHeader on your sample files to verify if the headers are properly sized and padded. The reason kindlegen does this is to make sure there is enough space in the header section to add in the DRM key info (and there can be multiple keys) and the extra EXTH metadata needed to store the DRM key info all on the fly when downloading. The dual-meta mobi editor uses this fact to add and remove EXTH info but keeps the entire section size exactly the same. By doing it this way, you do not need to update the palm section table offsets at all nor change the size of the section table at all. I can just replace the entire header section with the new one. As a safety check, I verify that all I am replacing at the end of the header are these extra padding null bytes and that I have not grown the header section in any way. I did this because some very very old mobis left no extra space. Everything since Kindlegen 1 adds this padding. So I think that any metadata editing program should be careful to keep the required extra padding (ie size the header to allow additions without the need to resize.) just like Kindlegen and Calibre do. Hope this helps. KevinH Last edited by KevinH; 06-10-2014 at 04:08 PM. |
06-10-2014, 11:36 PM | #62 |
Sigil Developer
Posts: 7,506
Karma: 5433350
Join Date: Nov 2009
Device: many
|
Hi,
I ran DumpMobiHeader_v015.py on your sample files. It seems the Mobi Meta Editor does improperly trim the padding placed there by kindlegen and calibre which is what causes the problem: Here is the original section map: Code:
Map of Palm DB Sections Dec - Hex : Description ---- - ---- ----------- 0000 - 0000: HEADER 6 [8736] 0001 - 0001: Text Record 0 [866] 0002 - 0002: 0000 [2] 0003 - 0003: NCX Index 0 [236] 0004 - 0004: NCX Index 1 [208] 0005 - 0005: NCX Index CNX [8] 0006 - 0006: Image jpeg [8956] 0007 - 0007: RESC [4112] 0008 - 0008: FLIS [36] 0009 - 0009: FCIS [44] 0010 - 000a: Source Archive 0 [1612] 0011 - 000b: BOUNDARY [8] 0012 - 000c: HEADER 8 [8736] 0013 - 000d: Text Record 0 [1587] 0014 - 000e: 00 [1] 0015 - 000f: Fragment Index 0 [248] 0016 - 0010: Fragment Index 1 [220] 0017 - 0011: Fragment Index CNX [16] 0018 - 0012: Skeleton Index 0 [244] 0019 - 0013: Skeleton Index_Index 1 [224] 0020 - 0014: NCX Index 0 [240] 0021 - 0015: NCX Index 1 [212] 0022 - 0016: NCX Index CNX [8] 0023 - 0017: FDST [28] 0024 - 0018: FLIS [36] 0025 - 0019: FCIS [44] 0026 - 001a: DATP [52] 0027 - 001b: EOF_RECORD [4] Here it is again after passing through MME Code:
Map of Palm DB Sections Dec - Hex : Description ---- - ---- ----------- 0000 - 0000: HEADER 6 [572] 0001 - 0001: Text Record 0 [866] 0002 - 0002: 0000 [2] 0003 - 0003: NCX Index 0 [236] 0004 - 0004: NCX Index 1 [208] 0005 - 0005: NCX Index CNX [8] 0006 - 0006: Image jpeg [8956] 0007 - 0007: RESC [4112] 0008 - 0008: FLIS [36] 0009 - 0009: FCIS [44] 0010 - 000a: Source Archive 0 [1612] 0011 - 000b: BOUNDARY [8] 0012 - 000c: HEADER 8 [8736] 0013 - 000d: Text Record 0 [1587] 0014 - 000e: 00 [1] 0015 - 000f: Fragment Index 0 [248] 0016 - 0010: Fragment Index 1 [220] 0017 - 0011: Fragment Index CNX [16] 0018 - 0012: Skeleton Index 0 [244] 0019 - 0013: Skeleton Index_Index 1 [224] 0020 - 0014: NCX Index 0 [240] 0021 - 0015: NCX Index 1 [212] 0022 - 0016: NCX Index CNX [8] 0023 - 0017: FDST [28] 0024 - 0018: FLIS [36] 0025 - 0019: FCIS [44] 0026 - 001a: DATP [52] 0027 - 001b: EOF_RECORD [4] So this is definitely an MME bug. It really needs to not remove all of the padding added to the end of the header information which is used to store the DRM Keys, and provides room for all of the extra DRM related Meta information to be added on the fly (as Amazon does). Hope this helps, KevinH |
06-11-2014, 01:22 AM | #63 | |
Grand Sorcerer
Posts: 5,582
Karma: 22735033
Join Date: Dec 2010
Device: Kindle PW2
|
Quote:
I'd already suspected that Mobi Meta Editor was to blame, because files directly converted from the source worked just fine. If you ever officially release this tool, maybe you could mention in the release notes that it'll most likely fail with .mobi files whose metadata was modified with Mobi Meta Editor. (Since the author of Mobi Meta Editor seems to no longer visit MR, chances are slim that he'll fix this bug.) |
|
09-05-2015, 07:04 PM | #64 |
Sigil Developer
Posts: 7,506
Karma: 5433350
Join Date: Nov 2009
Device: many
|
dualmetafix.py for python 2 and 3
Hi,
Sorry to resurrect an old thread but ... I have taken the original version of dualmetafix.py and modified it to work with both python 2.7 and python 3.4 in case anyone wants it. It is attached. See dualmetafix_new.py.zip Hope this helps, KevinH ps. New zip updated September 8, 2015 with a bug fix Last edited by KevinH; 09-08-2015 at 09:39 AM. |
09-08-2015, 09:40 AM | #65 |
Sigil Developer
Posts: 7,506
Karma: 5433350
Join Date: Nov 2009
Device: many
|
Hi,
Fixed a bug in dualmetafix_new.py see the previous post for the updated attachment. KevinH |
09-08-2015, 10:35 AM | #66 |
eBook Enthusiast
Posts: 85,544
Karma: 93383043
Join Date: Nov 2006
Location: UK
Device: Kindle Oasis 2, iPad Pro 10.5", iPhone 6
|
Hi Kevin,
This sounds very useful. Is there any chance it could be incorporated into calibre? The lack of ability to edit the metadata in dual Mobis in that program has always been a nuisance. |
09-08-2015, 10:57 AM | #67 |
Sigil Developer
Posts: 7,506
Karma: 5433350
Join Date: Nov 2009
Device: many
|
Hi Harry,
This tool is not a general purpose metadata editor. It simply fixes the common issue of missing ASIN and cdetype not being set to "EBOK" in both parts of a joint mobi like kindlegen generates. It was also thrown together quite quickly to fit that exact need. That said, someone could easily expand on it to make a more general purpose metadata editor for .mobi (azw, azw3, and joint mobis) if they so desired. I am pretty sure that calibre metadata handling activity is quite a bit more advanced than this code. Kovid and/or eschwartz could easily add the ability to calibre to make sure the metadata they have is written into both pieces of a joint mobi instead of just the first half. Kovid rarely turns down patches. Alternatively, someone (DiapDealer?) could write a calibre plugin to accomplish the same thing. They (and anyone else) are free to use whatever code I write in any way they want. Hope this helps, Kevin |
09-08-2015, 11:19 AM | #68 |
creator of calibre
Posts: 43,778
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
This code doesn't address the hard part of metadata writing in mobi files (no offense to KevinH) -- dealing with when the replacement metadata is longer than the original. MOBI files are palm databases and when you change the size of an individual record in the database, you then have to adjust various structures in the database header to reflect the fact that now the offsets for all subsequent records have changed. That is tricky to do robustly, especially when the record being changed is not the first record, as is the case for a dual mobi.
|
09-08-2015, 11:28 AM | #69 |
eBook Enthusiast
Posts: 85,544
Karma: 93383043
Join Date: Nov 2006
Location: UK
Device: Kindle Oasis 2, iPad Pro 10.5", iPhone 6
|
Thanks for the explanation, Kovid. I completely understand.
|
09-08-2015, 11:32 AM | #70 |
Sigil Developer
Posts: 7,506
Karma: 5433350
Join Date: Nov 2009
Device: many
|
Hi Kovid,
Very true, but it could do that easily as it is based on the KindleUnpack mobi_split code that does add and remove entire sections and rework things - even in the header fields itself during the split. I did not add that to this simple tool since luckily, kindegen seems to now create huge header sections (approx 8192 chars in length) which typically gives plenty of room for adding an asin and many other things before ever worrying about having to grow a header section length. The issue is that calibre needs a much more robust solution to work with older mobis and mobis not generated by kindlegen (with its large header sections), and even mobis missing EXTH sub-sections completely. That said, calibre already does handle all of that and does create the large header sections like kindlegen (isn't that right Kovid?), so something like dualmetafix could easily be added as a plugin for calibre. Take care, KevinH |
09-08-2015, 11:40 AM | #71 |
creator of calibre
Posts: 43,778
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
The problem is not that it cannot be done in principle -- calibre's current metadata update code already does it (albeit for record 0 not n).
The problem is that there are lots and lots of corner cases, and the only way to test those corner cases is to laboriously upload a file to two actual kindles (one that uses the old mobi and one that uses the new) and see if it works. That makes the effort required to develop a robust solution too large for the gain, at least for me. And I'm afraid that 8192 bytes is not nearly sufficient, some people have rather a lot of metadata, I even remember one user that used put the entire book text into the comments field |
09-08-2015, 11:48 AM | #72 |
Sigil Developer
Posts: 7,506
Karma: 5433350
Join Date: Nov 2009
Device: many
|
Hi Kovid,
Understood. Luckily, for books output by kindlegen, there is always add enough extra room for an asin since they always leave enough room for complete DRM headers and the like as well. Take care, KevinH |
Thread Tools | Search this Thread |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Recommended settings to convert dual-column PDF to useable MOBI format | Cephas Atheos | Conversion | 7 | 09-18-2012 07:32 AM |
Insert metadata as page at start of book adds does not replace (mobi to mobi) | linusnc | Calibre | 2 | 07-19-2012 03:54 PM |
Update Mobi header/file metadata without doing a Mobi to Mobi conversion | RecQuery | Conversion | 2 | 06-30-2012 11:43 AM |
EPUB (CSS) tweaker app | Loccy | Conversion | 9 | 01-23-2011 10:22 PM |
Firefox Tweaker: Flexbeta FireTweaker XP | Alexander Turcic | Lounge | 0 | 08-16-2004 04:51 AM |