what the ’ ??? (mobi conversion woes)

[zelda_pinwheel]

something very strange...

i tried to convert this book to imp, via html, using Nick's conversion tool (version 9.1).

but i got some very strange text as a result...

’ instead of curly apostrophe, “ for a double curly left quotation mark, and � for a double curly right quotation mark...

in the html source, the codes are respectively :
’ & acirc ; & #128 ; & #153 ; (i added the spaces)
“ & acirc ; & #128 ; & #156 ;
� & acirc ; & #128 ; & #157 ;

in the mobi version, the characters display correctly (apostrophe, open quotation mark, close quotation mark).

anybody know why this would happen ? there must be something strange in the mobi code, but i've never seen this result before... and is there any easy way to fix it (besides search and replace, i mean, because for the moment i have identified 3 charactes, but maybe there are a lot more in the rest of the book...) ?

[EDIT yes there are more, a LOT more. it looks like every single special punctuation mark is incorrectly coded. i hope somebody knows an easy way to fix this... or this conversion is going to be a lot more work than i expected...]

[tompe]

I would guess that UTF-8 is used in the MobiPocket file. Try mobi2html from MobiPerl and see if the resulting html file displays correctly in a browser. If that works then it must be the translation to imp that is wrong (and not my problem )

[zelda_pinwheel]

i will try mobi2html if you want (if i can figure it out...) but i think the problem is BEFORE the conversion to imp, because the html file which is left behind (it's the source for the imp file) contains these errors.

so, either the problem is in the original mobi code (seems likely), or in the conversion from mobi to html (never had this problem before).

[EDIT : i was looking at the html in a browser already. i have attached the page for you.]

[tompe]

Originally Posted by zelda_pinwheel

i will try mobi2html if you want (if i can figure it out...) but i think the problem is BEFORE the conversion to imp, because the html file which is left behind (it's the source for the imp file) contains these errors.

so, either the problem is in the original mobi code (seems likely), or in the conversion from mobi to html (never had this problem before).

[EDIT : i was looking at the html in a browser already. i have attached the page for you.]

I looked at the HTML file but hard to tell without looking at the MobiPocket file. If you give me the file I will test mobi2html and fix any possible problems with mobi2html (give me a link or email to tpe@ida.liu.se). But since it is 2 o'clock in the night here I might not do it until tomorrow...

[zelda_pinwheel]

heh, right, we are in the same time zone...

well, the mobipocket file is here, if you want to look at it tomorrow.

in the meantime, good night !

[JSWolf]

The problem is with mobi2imp. The HTML direct from mobi2html has the quotes just fine. I tried it with 9.2 and it had the same problem you did describe.

[tompe]

Originally Posted by zelda_pinwheel

heh, right, we are in the same time zone...

well, the mobipocket file is here, if you want to look at it tomorrow.

in the meantime, good night !

Oh, not only me having strange times...

mobi2html did work properly. It said:

Adding name attributes
UTF-8 detected: convert before TreBuilder parse

This is a recent fix so it might be something that Nick have not yet merged into his program.

[slayda]

Originally Posted by zelda_pinwheel

heh, right, we are in the same time zone...

well, the mobipocket file is here, if you want to look at it tomorrow.

in the meantime, good night !

Zelda,
mobi2html yields the same erroneous HTML markings. Book Designer will not "unpack" the .prc file. ABC Palm Converter just yields total garbage.

Sorry. Not a lot of help. Tompe seems to have answered. Maybe I need to get an updated version of Mobiperl.

[JSWolf]

Originally Posted by slayda

Zelda,
mobi2html yields the same erroneous HTML markings. Book Designer will not "unpack" the .prc file. ABC Palm Converter just yields total garbage.

Sorry. Not a lot of help. Tompe seems to have answered. Maybe I need to get an updated version of Mobiperl.

0.0.37 expands the PRC correctly so this error is not the fault of mobi2html. The fault lies in mobi2imp not handling UTF-8 correctly.

[nrapallo]

Originally Posted by tompe

Oh, not only me having strange times...

mobi2html did work properly. It said:


This is a recent fix so it might be something that Nick have not yet merged into his program.

Yup! Missed this, even though I recently went looking for improvements you have made to the mobiperl code. I found 'longtitle' (yeah!), but saw the test UTF-8 that was commented out and I 'blanked-out'; didn't even see your improvements!

I responded here with the exploration of this issue and that to fix the above strange behaviour, I best be getting the UTF-8 improvements implemented in Mobi2IMP!

[zelda_pinwheel]

amazing, i go to sleep, and when i wake up everything is fixed and the problem explained !!

brilliant !!

thanks everybody for your help ! i'm glad that the problem was so easily fixed, and i'm particularly glad there was an explanation for it so i understand (i hate those mysterious bugs that make no sense and you can't figure out where they come from, and even when you fix them you don't know why or how...).

have a drink on me, just tell the barman to put it on my tab

[jgray]

This issue of quotes, mdashes and such turning into strange characters is why I use the numeric representations in my HTML markup. For example, for a left-double-curly-quote, & # 8220 ; and & # 8221 ; for the right. This is the only way to guarantee that non-ASCII characters will display properly on different systems.

This is especially true with XHTML, as the only such characters defined by name are & lt ;, & gt ; & amp ; (I think there are a few more, I just don't remember them right now). Not even the & nbsp ; is defined for XHTML.

Note that I had to insert spaces on each of those tags. The BBS software shows the character and not the tag that I entered, even if I wrap them in CODE tags.

[DaleDe]

Originally Posted by jgray

This issue of quotes, mdashes and such turning into strange characters is why I use the numeric representations in my HTML markup. For example, for a left-double-curly-quote, & # 8220 ; and & # 8221 ; for the right. This is the only way to guarantee that non-ASCII characters will display properly on different systems.

This is especially true with XHTML, as the only such characters defined by name are & lt ;, & gt ; & amp ; (I think there are a few more, I just don't remember them right now). Not even the & nbsp ; is defined for XHTML.

Note that I had to insert spaces on each of those tags. The BBS software shows the character and not the tag that I entered, even if I wrap them in CODE tags.

The problem was the code was not in word or character coding, they were raw UTM-8. To understand this take the number 8220 from your example and convert it to binary values. In hexadecimal this would be 201C or 0010000000011100 and probably gets byte swapped. This attempts to get converted into 3 very different characters as shown in the problem. The original source possbile or even probably used characters like you do that was fine but it was compiled into Mobi internal format and then disassembled back into html and then converted to imp. The problem has been fixed by recognizing the UTM-8 character set.

Dale

[nrapallo]

Originally Posted by jgray

The BBS software shows the character and not the tag that I entered, even if I wrap them in CODE tags.

BBS - You're dating yourself with this acronym (and so am I).

Ever use 300 baud (I did)?

[kovidgoyal]

Originally Posted by jgray

This issue of quotes, mdashes and such turning into strange characters is why I use the numeric representations in my HTML markup. For example, for a left-double-curly-quote, & # 8220 ; and & # 8221 ; for the right. This is the only way to guarantee that non-ASCII characters will display properly on different systems.

This is especially true with XHTML, as the only such characters defined by name are & lt ;, & gt ; & amp ; (I think there are a few more, I just don't remember them right now). Not even the & nbsp ; is defined for XHTML.

Note that I had to insert spaces on each of those tags. The BBS software shows the character and not the tag that I entered, even if I wrap them in CODE tags.

Actually using numeric entities doesn't guarantee anything. After all you have to adopt some convention for how to decode the numbers into characters. I've seen HTML files that use the convention that the character encoding for the numeric entities is the same as that declared in the meta charset tag.