Need for a dual mobi metadata tweaker

[KevinH]

Originally Posted by AcidWeb

I will add it to file header and KCC README/Credits.
This script will not output any text when it will be part of KCC.

That's fine. Speaking for myself only, I do this for fun and to keep my programming skills from getting any rustier with age (brains past 50 seem to rot faster ;-). I have written/forgotten/abandoned so much code in my lifetime that as long as it is used by someone, I am happy.

Take care,

KevinH

[AcidWeb]

Yep. When you will you will be pleased with standalone version make topic on Kindle sub-forum. I'm quite sure that not only me need this type of tool. There is no other script that correctly edit both header sets - you are first :-)

I just test it. ASIN cover system is working correctly. Personal tag removing is... not. For some reason Personal tag is still there. I'm investigating that now.

[KevinH]

Hi,

Perhaps the 501 tag is in each exth header more than once. I have seen some mobis that have duplicate exth numbers in the same header. I could easily modify the del_exth to keep trying until no more 501 elements exist in each header.

Also, it could just be that the del_exth method is broken since I only tested it with kindlegen 2.9 generated mobis which had no 501 metadata elements.

Please let me know what you find out and I would be happy to try and fix the code.

Take care,

KevinH

[AcidWeb]

My all test are conducted on MOBI created by KindleGen so that is not a cause of problem.

BTW this how I inject field 501 in current version of KCC:
https://github.com/ciromattia/kcc/bl...esplit.py#L300

EDIT:
That is not del_exth fault. After commenting it tag is still here.

[KevinH]

Hi,
I am confused. Your mobi split modified code and mine are very close (same parent code). Yours does the splitting whereas mine does not split and updates the header records in place with the new exth information for both single and combination headers.

But the basic idea is exactly the same except for the splitting part.

Is CDE_TYPE (501) metadata being properly changed in the header or not? Perhaps the only difference is the splitting? Perhaps it needs to be split for the cover to work?

Ideas?


KevinH

[KevinH]

Perhaps some other field in the header needs to be changed.

Here is the output of DumpMobiHeader_v012.py for the first header:

First Header Dump from Section 0
Header Version is: 0x6
Header start position is: 0x0
Header Length is: 0x108
  Field:     compression_type   Offset: 0x000   Width:  2   Value: 0x02
  Field:                fill0   Offset: 0x002   Width:  2   Value: 0x00
  Field:          text_length   Offset: 0x004   Width:  4   Value: 0x7904
  Field:         text_records   Offset: 0x008   Width:  2   Value: 0x08
  Field:     max_section_size   Offset: 0x00a   Width:  2   Value: 0x1000
  Field:          crypto_type   Offset: 0x00c   Width:  2   Value: 0x00
  Field:                fill1   Offset: 0x00e   Width:  2   Value: 0x00
  Field:                magic   Offset: 0x010   Width:  4   Value: MOBI
  Field:        header_length   Offset: 0x014   Width:  4   Value: 0x0108
  Field:                 type   Offset: 0x018   Width:  4   Value: 0x0002
  Field:             codepage   Offset: 0x01c   Width:  4   Value: 0xfde9

And here is the same piece for the second header:

Second Header Dump from Section 22
Header Version is: 0x8
Header start position is: 0x16
Header Length is: 0x108
  Field:     compression_type   Offset: 0x000   Width:  2   Value: 0x02
  Field:                fill0   Offset: 0x002   Width:  2   Value: 0x00
  Field:          text_length   Offset: 0x004   Width:  4   Value: 0x8732
  Field:         text_records   Offset: 0x008   Width:  2   Value: 0x0a
  Field:     max_section_size   Offset: 0x00a   Width:  2   Value: 0x1000
  Field:          crypto_type   Offset: 0x00c   Width:  2   Value: 0x00
  Field:                fill1   Offset: 0x00e   Width:  2   Value: 0x00
  Field:                magic   Offset: 0x010   Width:  4   Value: MOBI
  Field:        header_length   Offset: 0x014   Width:  4   Value: 0x0108
  Field:                 type   Offset: 0x018   Width:  4   Value: 0x0002
  Field:             codepage   Offset: 0x01c   Width:  4   Value: 0xfde9

Notice the mobi type field which has value 2. In the Mobileread Wiki under mobi, there are many other possible values:

The kind of Mobipocket file this is
2 Mobipocket Book
3 PalmDoc Book
4 Audio
232 mobipocket? generated by kindlegen1.2
248 KF8: generated by kindlegen2
257 News
258 News_Feed
259 News_Magazine
513 PICS
514 WORD
515 XLS
516 PPT
517 TEXT
518 HTML

Perhaps the mobi type needs to be changed as well?

Sorry, but I am really fishing here as I can not recreate your issue since I do not own a PW or even a standalone Kindle device of any sort.

KevinH

[AcidWeb]

I'm also confused because I don't really see where it can fail. Changing 501 is enough.

But... it is quite possible that is my fault. I just given my test subject(:-)) a file made by old code. I'm sure it don't have Personal tag on PW1. But he use PW2 and tag is here.

I have to do more tests because I'm quite sure that more than one PW2 user already told me that tag is removed correctly.

Developing this type of stuff without access to hardware is "fun" :-)

[KevinH]

Hi AcidWeb,

One other thing to check - perhaps ASIN keys or more generally any strings are padded to some power of two or multiple of 4 in length? Then by writing out the asins last we may be misaligning the 501 metadata entry which was written previously.

Please try changing the order of adding in the new exth metadata info by writting exth 504 first, 113, next, and finally 501 last (so it will appear first in the metadata list and alignment should not be an issue) for both headers and see if that helps.

If so, we can try to figure out what type of alignment for odd string data like the asins is needed.

[KevinH]

Hi AcidWeb,

I have to run a few Sunday errands for my wife and kids so I will be awy form the computer for a few hours. Just post here if you can figure out what needs to be changed and I will check for things when I get back.

Good luck tracking it down!

KevinH

[AcidWeb]

I just need to test it on more than one Paperwhite. I will make next post when I get more feedback.

I don't think that your script is writing headers incorrectly. When I write only field 501 it don't work too. With or without ASIN all headers are intact according to DumpMobiHeader.

[DiapDealer]

It's possible that the dual-format mobi may never be seen as anything other than a personal document, regardless of the EXTH records. I'm assuming since the split-out and modified azw3 file seems to work as you expect it to, that you've tried simply renaming the kindlegen output from .mobi to .azw3 or .prc (in addition to the header modifications) to see if that affects how the device classifies it?

[AcidWeb]

OK. I admit that I screwed this one. On PaperWhite models field 501 is simply not enough to remove personal tag. Script is working correctly. There was some confusion between me and people who provide feedback. Lack of access to real hardware is killing me.

[KevinH]

Hi,

If you can find a couple of examples of mobi files that work (show as non-personal when *not* a commerical mobi on a PW) then we can run DumpMobiHeader_v012.py on them to see if they differ in any other way to what we are generating (except for the splitting part)

I have attached a zip of the latest version of DumpMobiHeader_v012.py in case you would like to use it for your testing.

It has a few additions for determining the section maps on more recent Kindlegen 2.9 made mobis and other metadata fields.

Notice in Kindle 2.9 generated mobis there is now a second BOUNDARY section followed by a CONT section with its own EXTH which as the following string in it (HD_CONTAINER) which seems to have an HD copy of the cover image stored in a CRES section in a large format.

Using vi or emacs in hexl-mode can show you an awful lot of goodies they are packing into their kindlegen output now. They are basically including everything but the kitchen sink (they have the complete kindlegen build log stored as well)

Hope you can track down what else needs to change.

Take care,

KevinH

[AcidWeb]

Oh I know how to fix it for some time as Calibre do that. I just don't like the answer.

When ASIN is set, 501 set to EBOK and cover image is separately side-loaded to kindle cache then everything is looking perfectly.

Few hours ago I was sure that there is method to do that without side-loading cover.

Currently I'm dissecting some MOBI files - Maybe I will find something.

[AcidWeb]

OK. Finished my tests.

There is no way to remove Personal tag without injecting fake ASIN.
So there is no problem with the script - It is working correctly.

Summing up the situation with hybrid MOBIs. There are two scenarios.
A - There is fake ASIN number + EBOK tag
B - There is only EBOK tag

If file A will be uploaded by Calibre.
No Personal tag.
HQ cover will be automatically side-loaded to Kindle cache.

If file A will be uploaded manually.
No Personal tag.
No cover at all. Kindle will try to load it from cache - and fail.

If file B will be uploaded by Calibre.
Fake ASIN will be injected by Calibre. But it will only edit first MOBI7 header.
Personal tag. Due to lack of ASIN in MOBI8 header.
HQ cover will be automatically side-loaded to Kindle cache. But it will not work as MOBI8 header don't have ASIN.

If file B will be uploaded manually.
Personal tag.
Kindle will display LQ cover from book file.

-----

So it is lose-lose situation. File A work only correctly with Calibre. And file B will always have LQ cover and Personal tag.

Split files are simply better. Because you can make MOBI8 only file. And only change 501 to EBOK. Then this file will work fully correctly when uploaded by Calibre and still have cover when uploaded manually.

In this state it looks more like Calibre need a patch to support hybrid MOBI.