Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Formats > Kindle Formats

Notices

Reply
 
Thread Tools Search this Thread
Old 06-10-2014, 04:06 PM   #61
KevinH
Sigil Developer
KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.
 
Posts: 7,506
Karma: 5433350
Join Date: Nov 2009
Device: many
Hi,

Yes, that script was designed with Kindlegen and Calibre in mind. Kindlegen allocates a full section of up to 8192 (or 4096?) bytes of nulls when it creates its headers. Calibre does the same to match what Kindlegen does. Simply run DumpMobiHeader_v14.py or later to see all of the extra null bytes that pad out the header section after the full title field.

As I am away from my workstation please run DumpMobiHeader on your sample files to verify if the headers are properly sized and padded.

The reason kindlegen does this is to make sure there is enough space in the header section to add in the DRM key info (and there can be multiple keys) and the extra EXTH metadata needed to store the DRM key info all on the fly when downloading.

The dual-meta mobi editor uses this fact to add and remove EXTH info but keeps the entire section size exactly the same. By doing it this way, you do not need to update the palm section table offsets at all nor change the size of the section table at all. I can just replace the entire header section with the new one.

As a safety check, I verify that all I am replacing at the end of the header are these extra padding null bytes and that I have not grown the header section in any way. I did this because some very very old mobis left no extra space. Everything since Kindlegen 1 adds this padding.

So I think that any metadata editing program should be careful to keep the required extra padding (ie size the header to allow additions without the need to resize.) just like Kindlegen and Calibre do.

Hope this helps.

KevinH

Last edited by KevinH; 06-10-2014 at 04:08 PM.
KevinH is offline   Reply With Quote
Old 06-10-2014, 11:36 PM   #62
KevinH
Sigil Developer
KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.
 
Posts: 7,506
Karma: 5433350
Join Date: Nov 2009
Device: many
Hi,

I ran DumpMobiHeader_v015.py on your sample files. It seems the Mobi Meta Editor does improperly trim the padding placed there by kindlegen and calibre which is what causes the problem:

Here is the original section map:
Code:
Map of Palm DB Sections
    Dec  - Hex : Description
    ---- - ----  -----------
    0000 - 0000: HEADER 6 [8736]
    0001 - 0001: Text Record 0 [866]
    0002 - 0002: 0000  [2]
    0003 - 0003: NCX Index 0 [236]
    0004 - 0004: NCX Index 1 [208]
    0005 - 0005: NCX Index CNX [8]
    0006 - 0006: Image jpeg [8956]
    0007 - 0007: RESC [4112]
    0008 - 0008: FLIS [36]
    0009 - 0009: FCIS [44]
    0010 - 000a: Source Archive 0 [1612]
    0011 - 000b: BOUNDARY [8]
    0012 - 000c: HEADER 8 [8736]
    0013 - 000d: Text Record 0 [1587]
    0014 - 000e: 00  [1]
    0015 - 000f: Fragment Index 0 [248]
    0016 - 0010: Fragment Index 1 [220]
    0017 - 0011: Fragment Index CNX [16]
    0018 - 0012: Skeleton Index 0 [244]
    0019 - 0013: Skeleton Index_Index 1 [224]
    0020 - 0014: NCX Index 0 [240]
    0021 - 0015: NCX Index 1 [212]
    0022 - 0016: NCX Index CNX [8]
    0023 - 0017: FDST [28]
    0024 - 0018: FLIS [36]
    0025 - 0019: FCIS [44]
    0026 - 001a: DATP [52]
    0027 - 001b: EOF_RECORD [4]
Notice the size of each header is 8736 bytes long. This is made up of 16 bytes of extra info at the start followed by 264 bytes (0x108) bytes of header information plus the EXTH info (264 bytes) leaving 8192 (0x2000) bytes of padding for the DRM Keys, extra metadata, etc.

Here it is again after passing through MME
Code:
Map of Palm DB Sections
    Dec  - Hex : Description
    ---- - ----  -----------
    0000 - 0000: HEADER 6 [572]
    0001 - 0001: Text Record 0 [866]
    0002 - 0002: 0000  [2]
    0003 - 0003: NCX Index 0 [236]
    0004 - 0004: NCX Index 1 [208]
    0005 - 0005: NCX Index CNX [8]
    0006 - 0006: Image jpeg [8956]
    0007 - 0007: RESC [4112]
    0008 - 0008: FLIS [36]
    0009 - 0009: FCIS [44]
    0010 - 000a: Source Archive 0 [1612]
    0011 - 000b: BOUNDARY [8]
    0012 - 000c: HEADER 8 [8736]
    0013 - 000d: Text Record 0 [1587]
    0014 - 000e: 00  [1]
    0015 - 000f: Fragment Index 0 [248]
    0016 - 0010: Fragment Index 1 [220]
    0017 - 0011: Fragment Index CNX [16]
    0018 - 0012: Skeleton Index 0 [244]
    0019 - 0013: Skeleton Index_Index 1 [224]
    0020 - 0014: NCX Index 0 [240]
    0021 - 0015: NCX Index 1 [212]
    0022 - 0016: NCX Index CNX [8]
    0023 - 0017: FDST [28]
    0024 - 0018: FLIS [36]
    0025 - 0019: FCIS [44]
    0026 - 001a: DATP [52]
    0027 - 001b: EOF_RECORD [4]
Notice that the Mobi 6 header is shrunk greatly in size to just 572 bytes (all of the extra padding is removed). Also it is clear MME does not understand how to deal with the Mobi 8 (azw3) part and ignores it.

So this is definitely an MME bug. It really needs to not remove all of the padding added to the end of the header information which is used to store the DRM Keys, and provides room for all of the extra DRM related Meta information to be added on the fly (as Amazon does).


Hope this helps,

KevinH
KevinH is offline   Reply With Quote
Old 06-11-2014, 01:22 AM   #63
Doitsu
Grand Sorcerer
Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.Doitsu ought to be getting tired of karma fortunes by now.
 
Doitsu's Avatar
 
Posts: 5,582
Karma: 22735033
Join Date: Dec 2010
Device: Kindle PW2
Quote:
Originally Posted by KevinH View Post
I ran DumpMobiHeader_v015.py on your sample files. It seems the Mobi Meta Editor does improperly trim the padding placed there by kindlegen and calibre which is what causes the problem.
Thanks for testing this.



I'd already suspected that Mobi Meta Editor was to blame, because files directly converted from the source worked just fine.

If you ever officially release this tool, maybe you could mention in the release notes that it'll most likely fail with .mobi files whose metadata was modified with Mobi Meta Editor.

(Since the author of Mobi Meta Editor seems to no longer visit MR, chances are slim that he'll fix this bug.)
Doitsu is offline   Reply With Quote
Old 09-05-2015, 07:04 PM   #64
KevinH
Sigil Developer
KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.
 
Posts: 7,506
Karma: 5433350
Join Date: Nov 2009
Device: many
dualmetafix.py for python 2 and 3

Hi,

Sorry to resurrect an old thread but ...

I have taken the original version of dualmetafix.py and modified it to work with both python 2.7 and python 3.4 in case anyone wants it.

It is attached. See dualmetafix_new.py.zip

Hope this helps,

KevinH


ps. New zip updated September 8, 2015 with a bug fix
Attached Files
File Type: zip dualmetafix_new.py.zip (3.7 KB, 1102 views)

Last edited by KevinH; 09-08-2015 at 09:39 AM.
KevinH is offline   Reply With Quote
Old 09-08-2015, 09:40 AM   #65
KevinH
Sigil Developer
KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.
 
Posts: 7,506
Karma: 5433350
Join Date: Nov 2009
Device: many
Hi,
Fixed a bug in dualmetafix_new.py see the previous post for the updated attachment.

KevinH
KevinH is offline   Reply With Quote
Old 09-08-2015, 10:35 AM   #66
HarryT
eBook Enthusiast
HarryT ought to be getting tired of karma fortunes by now.HarryT ought to be getting tired of karma fortunes by now.HarryT ought to be getting tired of karma fortunes by now.HarryT ought to be getting tired of karma fortunes by now.HarryT ought to be getting tired of karma fortunes by now.HarryT ought to be getting tired of karma fortunes by now.HarryT ought to be getting tired of karma fortunes by now.HarryT ought to be getting tired of karma fortunes by now.HarryT ought to be getting tired of karma fortunes by now.HarryT ought to be getting tired of karma fortunes by now.HarryT ought to be getting tired of karma fortunes by now.
 
HarryT's Avatar
 
Posts: 85,544
Karma: 93383043
Join Date: Nov 2006
Location: UK
Device: Kindle Oasis 2, iPad Pro 10.5", iPhone 6
Hi Kevin,

This sounds very useful. Is there any chance it could be incorporated into calibre? The lack of ability to edit the metadata in dual Mobis in that program has always been a nuisance.
HarryT is offline   Reply With Quote
Old 09-08-2015, 10:57 AM   #67
KevinH
Sigil Developer
KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.
 
Posts: 7,506
Karma: 5433350
Join Date: Nov 2009
Device: many
Hi Harry,

This tool is not a general purpose metadata editor. It simply fixes the common issue of missing ASIN and cdetype not being set to "EBOK" in both parts of a joint mobi like kindlegen generates. It was also thrown together quite quickly to fit that exact need. That said, someone could easily expand on it to make a more general purpose metadata editor for .mobi (azw, azw3, and joint mobis) if they so desired.

I am pretty sure that calibre metadata handling activity is quite a bit more advanced than this code. Kovid and/or eschwartz could easily add the ability to calibre to make sure the metadata they have is written into both pieces of a joint mobi instead of just the first half. Kovid rarely turns down patches. Alternatively, someone (DiapDealer?) could write a calibre plugin to accomplish the same thing.

They (and anyone else) are free to use whatever code I write in any way they want.

Hope this helps,

Kevin
KevinH is offline   Reply With Quote
Old 09-08-2015, 11:19 AM   #68
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,778
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
This code doesn't address the hard part of metadata writing in mobi files (no offense to KevinH) -- dealing with when the replacement metadata is longer than the original. MOBI files are palm databases and when you change the size of an individual record in the database, you then have to adjust various structures in the database header to reflect the fact that now the offsets for all subsequent records have changed. That is tricky to do robustly, especially when the record being changed is not the first record, as is the case for a dual mobi.
kovidgoyal is offline   Reply With Quote
Old 09-08-2015, 11:28 AM   #69
HarryT
eBook Enthusiast
HarryT ought to be getting tired of karma fortunes by now.HarryT ought to be getting tired of karma fortunes by now.HarryT ought to be getting tired of karma fortunes by now.HarryT ought to be getting tired of karma fortunes by now.HarryT ought to be getting tired of karma fortunes by now.HarryT ought to be getting tired of karma fortunes by now.HarryT ought to be getting tired of karma fortunes by now.HarryT ought to be getting tired of karma fortunes by now.HarryT ought to be getting tired of karma fortunes by now.HarryT ought to be getting tired of karma fortunes by now.HarryT ought to be getting tired of karma fortunes by now.
 
HarryT's Avatar
 
Posts: 85,544
Karma: 93383043
Join Date: Nov 2006
Location: UK
Device: Kindle Oasis 2, iPad Pro 10.5", iPhone 6
Thanks for the explanation, Kovid. I completely understand.
HarryT is offline   Reply With Quote
Old 09-08-2015, 11:32 AM   #70
KevinH
Sigil Developer
KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.
 
Posts: 7,506
Karma: 5433350
Join Date: Nov 2009
Device: many
Hi Kovid,

Very true, but it could do that easily as it is based on the KindleUnpack mobi_split code that does add and remove entire sections and rework things - even in the header fields itself during the split.

I did not add that to this simple tool since luckily, kindegen seems to now create huge header sections (approx 8192 chars in length) which typically gives plenty of room for adding an asin and many other things before ever worrying about having to grow a header section length.

The issue is that calibre needs a much more robust solution to work with older mobis and mobis not generated by kindlegen (with its large header sections), and even mobis missing EXTH sub-sections completely.

That said, calibre already does handle all of that and does create the large header sections like kindlegen (isn't that right Kovid?), so something like dualmetafix could easily be added as a plugin for calibre.

Take care,

KevinH
KevinH is offline   Reply With Quote
Old 09-08-2015, 11:40 AM   #71
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,778
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
The problem is not that it cannot be done in principle -- calibre's current metadata update code already does it (albeit for record 0 not n).

The problem is that there are lots and lots of corner cases, and the only way to test those corner cases is to laboriously upload a file to two actual kindles (one that uses the old mobi and one that uses the new) and see if it works. That makes the effort required to develop a robust solution too large for the gain, at least for me. And I'm afraid that 8192 bytes is not nearly sufficient, some people have rather a lot of metadata, I even remember one user that used put the entire book text into the comments field
kovidgoyal is offline   Reply With Quote
Old 09-08-2015, 11:48 AM   #72
KevinH
Sigil Developer
KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.KevinH ought to be getting tired of karma fortunes by now.
 
Posts: 7,506
Karma: 5433350
Join Date: Nov 2009
Device: many
Hi Kovid,

Understood. Luckily, for books output by kindlegen, there is always add enough extra room for an asin since they always leave enough room for complete DRM headers and the like as well.

Take care,

KevinH
KevinH is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Recommended settings to convert dual-column PDF to useable MOBI format Cephas Atheos Conversion 7 09-18-2012 07:32 AM
Insert metadata as page at start of book adds does not replace (mobi to mobi) linusnc Calibre 2 07-19-2012 03:54 PM
Update Mobi header/file metadata without doing a Mobi to Mobi conversion RecQuery Conversion 2 06-30-2012 11:43 AM
EPUB (CSS) tweaker app Loccy Conversion 9 01-23-2011 10:22 PM
Firefox Tweaker: Flexbeta FireTweaker XP Alexander Turcic Lounge 0 08-16-2004 04:51 AM


All times are GMT -4. The time now is 07:21 AM.


MobileRead.com is a privately owned, operated and funded community.