Test oeb2mobi output for me?

[tompe]

Originally Posted by llasram

Done .

Concerning multibyte character overlap. How do you know which byte is the size byte?

Are these characters and the trailing data part of the record size or are they outside the specified record size?

[tompe]

Originally Posted by tompe

I looked at the output from --rawhtml but could not find any UTF-8 characters... But there is null characters in the file. But that is the data directly from the Perl module unpacking the compressed data so this is probably releated to something else. UTF-8 ought not to produce null characters.

The extra data flag is set to 0x31 for this file. So the extra characters are probably something from the unpacking of the data. The unpacking does not know about the extra data.

[Hadrien]

Originally Posted by nrapallo

Most Feedbooks.com Mobipocket/Kindle offerings have this problem (as they usually are UTF-8 encoded).

We use UTF-8 on 100% of our files actually.

[llasram]

Originally Posted by tompe

The extra data flag is set to 0x31 for this file. So the extra characters are probably something from the unpacking of the data. The unpacking does not know about the extra data.

If the '1' bit is set, and there are no actual multibyte characters in the text, then each record will end with a NUL byte indicating 0 overlaping bytes. (Well, unless bits one of bits 4-8 is set on the "size & flags" byte.)

Originally Posted by tompe

Concerning multibyte character overlap. How do you know which byte is the size byte?

It's the last byte of that trailing entry.

Originally Posted by tompe

Are these characters and the trailing data part of the record size or are they outside the specified record size?

As I understood it, the "record size" was just the distance to the next record. In which case yes, they are part of the record they follow.

[llasram]

Originally Posted by tompe

The extra data flag is set to 0x31 for this file. So the extra characters are probably something from the unpacking of the data. The unpacking does not know about the extra data.

Wait, where are you getting 0x31 from? I see it as 0x1 (offset 0x58c in the file).

[tompe]

Originally Posted by llasram

Wait, where are you getting 0x31 from? I see it as 0x1 (offset 0x58c in the file).

Ah, it is 0x1. My routine converting to hex is not working properly...

But that explains all the extra null characters.

[tompe]

Originally Posted by llasram

If the '1' bit is set, and there are no actual multibyte characters in the text, then each record will end with a NUL byte indicating 0 overlaping bytes. (Well, unless bits one of bits 4-8 is set on the "size & flags" byte.)

I am not sure I get it totally. If bit "1" is set is then the last byte in the record always realated to multibyte characters?

My code now is the following and I wondered if this is a correct understanding of it:

                eval {
                    sub min { return ($_[0]<$_[1]) ? $_[0] : $_[1] }
                    my $maxi = min($#$recs, $header->{'records'});
                    for( my $i = 1; $i <= $maxi; $i ++ ) {
                        my $data = $recs->[$i]->{'data'};
                        my $len = length($data);
                        my $overlap = "";
                        if ($self->{multibyteoverlap}) {
                            my $c = chop $data;
                            print STDERR "I:$i - $len - ", int($c), "\n";
                            my $n = $c & 7;
                            foreach (0..$n-1) {
                                $overlap .= chop $data;
                            }
                        }

                        $body .= _decompress_record( $header->{'version'},
                                                     $data );
                        $body .= $overlap;
                    }
                };

Why is three bits used for the size if the maximum size is 3? (I see now that I have reversed the order in $overlap).

[llasram]

Originally Posted by tompe

I am not sure I get it totally. If bit "1" is set is then the last byte in the record always realated to multibyte characters?

Almost. It's the *first* trailing entry, which means it immediately follows the text, but may be followed by other trailing entries. If bit 1 is set, plus another bits, you'll have:

<trailing multibyte bytes><multibyte size & flags><trailing data><size>

Originally Posted by tompe

My code now is the following and I wondered if this is a correct understanding of it:

My Perl is pretty rusty, but I think mostly... Except instead of needing to preserve the overlap, you actually need to just chop it off -- it appears again at the beginning of the next record.

Originally Posted by tompe

Why is three bits used for the size if the maximum size is 3? (I see now that I have reversed the order in $overlap).

My error. I did byte & 3 to get the size, and for some reason when I was translating the info into the wiki I turned that into 3 bits. It is only 2 bits (which I have updated the wiki to reflect).

[nrapallo]

Originally Posted by nrapallo

Most Feedbooks.com Mobipocket/Kindle offerings have this problem (as they usually are UTF-8 encoded).

...

I've always seen this behaviour with Feedbooks.com .prc/.mobi ebooks.

I'm happy to report that all previous "issues" I've had with Feedbooks.com .mobi ebook conversions using my Mobi2IMP have now been resolved by the recent update to tompe's mobi2html (which I have incorporated into a beta Mobi2IMP).

[tompe]

Originally Posted by llasram

Almost. It's the *first* trailing entry, which means it immediately follows the text, but may be followed by other trailing entries. If bit 1 is set, plus another bits, you'll have:

<trailing multibyte bytes><multibyte size & flags><trailing data><size>

But how do I then detect how many bytes there are in the trailing multibyte bytes? How can I know for sure which byte is the one giving the number of bytes? Or can you parse it in reverse order and it is not ambigious?

[llasram]

Originally Posted by tompe

But how do I then detect how many bytes there are in the trailing multibyte bytes? How can I know for sure which byte is the one giving the number of bytes? Or can you parse it in reverse order and it is not ambigious?

Right. You parse each trailing entry backwards. So if all 16 were present, you'd parse #16 at the end of the record, then #15, etc etc on through #1 last. I may have complicated understanding by on the Wiki leaving out the distinction between what I'm calling "forwards-encoded" variable-width integers and "backwards-encoded" ones. The sizes of trailing entries 2-16 are backwards-encoded variable-width integers, encoded with only the high (first) byte having bit 8 set, which means you can most easily read them backwards. So yeah -- start from the end and work backwards .

This is Calibre's current code for find the total size of the trailing entries:

def sizeof_trailing_entries(self, data):
    def sizeof_trailing_entry(ptr, psize):
        bitpos, result = 0, 0
        while True:
            v = ord(ptr[psize-1])
            result |= (v & 0x7F) << bitpos
            bitpos += 7
            psize -= 1
            if (v & 0x80) != 0 or (bitpos >= 28) or (psize == 0):
                return result
    
    num = 0
    size = len(data)
    flags = self.book_header.extra_flags >> 1
    while flags:
        if flags & 1:
            num += sizeof_trailing_entry(data, size - num)
        flags >>= 1
    if self.book_header.extra_flags & 1:
        num += (ord(data[size - num - 1]) & 0x3) + 1
    return num

HTH!

[tompe]

Thanks, it was as complicated as I suspected then... These kind of complications seems very odd and I suspect that a specification of the MobiPocket format is not released because either it does not exist or they do not want to show the world how bad the format really is.

Is there any test file available somewhere were the extraflags is something else than 0x1?

[llasram]

Originally Posted by tompe

Thanks, it was as complicated as I suspected then... These kind of complications seems very odd and I suspect that a specification of the MobiPocket format is not released because either it does not exist or they do not want to show the world how bad the format really is.

This really isn't that complicated compared to LIT's internal indices -- hash tables and multi-level look-up tables and tree lists oh my. At least Mobipocket realized they needed a backwards-compatible way to specify new trailing entries before they added more than two.

Originally Posted by tompe

Is there any test file available somewhere were the extraflags is something else than 0x1?

Attached is one I've generated with mobigen.

[pdurrant]

Oh - very useful stuff. And it turns out that the Mobipocket decoder will need some fixes for cases where bit position 1 is set. I can only suppose that very few commercial DRMed eBooks are out there with that bit set.

Happily easy to fix given this code, once such a book turns up.

Originally Posted by llasram

This is Calibre's current code for find the total size of the trailing entries:

def sizeof_trailing_entries(self, data):
    def sizeof_trailing_entry(ptr, psize):
        bitpos, result = 0, 0
        while True:
            v = ord(ptr[psize-1])
            result |= (v & 0x7F) << bitpos
            bitpos += 7
            psize -= 1
            if (v & 0x80) != 0 or (bitpos >= 28) or (psize == 0):
                return result
    
    num = 0
    size = len(data)
    flags = self.book_header.extra_flags >> 1
    while flags:
        if flags & 1:
            num += sizeof_trailing_entry(data, size - num)
        flags >>= 1
    if self.book_header.extra_flags & 1:
        num += (ord(data[size - num - 1]) & 0x3) + 1
    return num

HTH!

[Jellby]

How do you deal with "font-variant: small-caps"? Do you convert <span class="small-caps">Foo Bar</span> into F<font size="-1">OO</font> B<font size="-1">AR</font> ?

I guess "text-transform: uppercase" is easier... (I once found an HTML book where many capital letters were "created" with this property, which meant that copy-pasting gave lowercase letters, it was a pain...)