Problems with multibyte data flags

[pdurrant]

Mobipocket files have a method of coping with multi-byte (UTF8) characters that get broken over a block boundary. They have a flag at the end of the block indicating that this has happened, and also supplying the missing bytes (that are at the start of the next block).

See

https://wiki.mobileread.com/wiki/MOBI...racter_overlap

Now, the problem I'm having is that I can't find a way to determine if these multibyte flags are present in a Mobipocket file or not. I used to think that a test of mobipocket version (>5) and header length (>=0xE8) was necessary and sufficient. But now I've come across a Mobipocket version 5 file (with header length 0xE8) that /does/ have the multibyte flags. And yet I have another file, also version 5 and header length 0xE8, that /doesn't/ have the multibyte flags (despite the bytes at offset 242 seeming to indicate that it should).

Does anyone have any idea of how to reliably determine whether the data blocks in a Mobipocket file have or don't have the multibyte flags at the end of each block?

In hope...

[pdurrant]

Originally Posted by pdurrant

Now, the problem I'm having is that I can't find a way to determine if these multibyte flags are present in a Mobipocket file or not.

That wasn't my problem. My problem was that I didn't understand that although trailing data entries and multibyes do not get compressed, the trailing multibytes in a utf8 encoded file do get encrypted (although the other trailing data entries do not). So it's vital not to remove the count of trailing multibytes before trying to decrypt the file.

So it seems the test for whether the trailing data flags is valid is actually quote simple - version 5 or greater, and header length 0xE4 or greater. Although since I've never seen a version 5 Mobipocket file with a header length less than 0xE4, the latter condition might be redundant.

I've updated the wiki to clarify the situation for anyone else fool enough to poke about in Mobipocket file innards.

[KevinH]

Hi Paul,

Nice that you found it! Will you please add your fixes to our Decoder tool google source code site extraction routines?

I will then use hg to sync the changes and include them.

Thanks!

KevinH

[pdurrant]

Originally Posted by KevinH

Nice that you found it! Will you please add your fixes to our Decoder tool google source code site extraction routines?

Ok, done.

[JSWolf]

Originally Posted by pdurrant

Ok, done.

What about changes to MobiDeDRM.py?

[pdurrant]

Originally Posted by JSWolf

What about changes to MobiDeDRM.py?

I leave that as an exercise for some random apprentice