oh, criminy... (lit to imp)

[zelda_pinwheel]

well it seems this is my week for bizarre conversion headaches, but who knows, maybe i will be really lucky, and this one will be as easy to fix as the last one...

i bought the book The Professor and the Madman in LIT format to convert to imp using the LIT2SB utility (thanks Nick !). in the past i have had very good results with this (including on DRM lit files, like this one) however this time...

- the html file looks perfect when displayed in a browser.

- the html file looks ok with minor fixes when opened in Word and saved as a .doc file, except that the first time i did that i found there was a page break after EVERY SINGLE PARAGRAPH (including one-line paragraphs, like in the TOC...). once i removed those it works ok.

- the imp file generated from the html file (automatically by LIT2SB or using eBook Publisher) looks like crap !!! the text is aligned sometimes to the left but usually to the right with a strange, large and variable left margin, the whole first chapter (and maybe more...) mysteriously switches to bold text for no reason, many special characters are not displayed ("Legal unicode value encountered for a glyph we do not support. Replacing with '?' ")...

there is an external css file as long as my arm, and the problem must be somewhere in there, but i have not found it yet (still looking...), because NONE of the classes applied to the paragraphs in question mention "text-align : right", or specify margins. i added my own "text-align:left;" and defined 2% margins on the body : no joy. but if i remove the css file completely, the text reverts to correctly left-aligned with no margins.

last night, in desperation, i decided to give up on trying to make my imp correctly, and cheat by using the .doc file, in case this actually worked.

well, miraculously, it did, rather well, except for some "legal unicode characters" which are still replaced by "?" (like & # 601 ; which makes a small upside-down e as you can see here, or look at the html capture in the "mysterious" phonetic spelling.)

Question 1 : is there some way (other than making images...) to get this stupid thing to display the proper glyphs ? (if not, maybe i will make images, but it's not a really good solution especially since i fear there will be dozens or even hundreds of them)

since this is a copyrighted book i suppose i cannot attach any of my files here, so i don't know whether anyone will be able to help me. i have made some screen captures of the hmtl and imp versions, just in case anyone might have an idea.

Question 2 : is this a problem which is known, like the UTF-8 problem ? is there an easy solution to it ??

[nrapallo]

Originally Posted by zelda_pinwheel

...many special characters are not displayed ("Legal unicode value encountered for a glyph we do not support. Replacing with '?' ")...

That's a current limitation that you are up against! It recognizes the glyph, but doesn't render (support) it. Making images to replace common unsupported glyphs is a practical solution; just search and replace

"& # 601"

with

<img src="601image.gif" align="absbottom">

Question 1 : is there some way (other than making images...) to get this stupid thing to display the proper glyphs ? (if not, maybe i will make images, but it's not a really good solution especially since i fear there will be dozens or even hundreds of them)

I think this is your only choice. Just make ONE image for each unsupported legal gylph and search and replace as indicated above. For say a hundred "& # 601" you will only need one 601 image!

Question 2 : is this a problem which is known, like the UTF-8 problem ? is there an easy solution to it ??

Usually 'Tidy' used by 'LIT2SB' handles the conversions quite well, but I had it fail miserably recently for a test case of converting Esperanto. Maybe your html is similar.

p.s. I also got terrible results from Tidy when from converting UTF-8 (see post#44 here), however, I think your situation is different.

EDIT: I will get you started with some sample character images in the attached .zip!

[zelda_pinwheel]

Originally Posted by nrapallo

That's a current limitation that you are up against! It recognizes the glyph, but doesn't render (support) it. Making images to replace common unsupported glyphs is a practical solution; just search and replace

"& # 601"

with

<img src="601image.gif">

I think this is your only choice. Just make ONE image for each unsupported legal gylph and search and replace as indicated above. For say a hundred "& # 601" you will only need one 601 image!

heh, sorry, i should have been more clear. obviously, i will only need one image for the 601 glyph. but i meant, in this book there are a lot of *other* glyphs which also would need to be replaced (there are a lot of phonetic spellings of words, using wacky phonetic glyphs, because it's a book about the creation of the dictionary. quite a few of them are not supported) ; potentially dozens or hundreds of *different* glyphs and each time i have to make an image (god help me...).

well, since there is no good way of doing this, i think i will go along to the eBook Publisher bug thread, and mention that it would be really nice if they could support all the special glyphs... (and some different fonts would be nice as well ! sheesh, if they just added Arial and Times New Roman i think we would probably have no more trouble with any wacky glyphs) i really don't like putting text in images if i can possibly avoid it, for many reasons. just an example, if i resize the text, the image will stay the same...

Usually 'Tidy' used by 'LIT2SB' handles the conversions quite well, but I had it fail miserably recently for a test case of converting Esperanto. Maybe your html is similar.

p.s. I also got terrible results from Tidy when from converting UTF-8 (see post#44 here), however, I think your situation is different.

well, the meta tag *does* specify that this is UTF-8 character set, however i don't think that is the problem, since i can't imagine how the character set could affect the text alignment... could it ?

i'm getting close to a presentable text right now, even if i did cheat and use the .doc as my starting point. i suppose i probably will have to make those blasted glyph images, if i want to really make a nice text (well, "nice" as long as no-one looks at the code !!).

stupid drm lit... i swear, problems like this make me want to stop buying books altogether.

thanks for your help, even if there's not really anything to be done...

[nrapallo]

Originally Posted by zelda_pinwheel

...well, the meta tag *does* specify that this is UTF-8 character set, however i don't think that is the problem, since i can't imagine how the character set could affect the text alignment... could it ?

Here's a radical idea. Why not use your exploded .html (or .opf) in (free)Mobipocket Creator to make a .prc, then use Mobi2IMP to create a new .html that has been UTF-8 "fixed".

You will still need to supply the missing gylphs (using images as indicated above), but if it was UTF-8, then a lot of them should now appear better.

[jbenny]

If you look around on the ETI site, they have a list of all the extended characters supported by their readers. I used to have a copy, but can't seem to find it.

[zelda_pinwheel]

Originally Posted by jbenny

If you look around on the ETI site, they have a list of all the extended characters supported by their readers. I used to have a copy, but can't seem to find it.

thanks for that info, i will try to find it, it interests me.

@ nrapallo : well, since i made every other possible version of this book already, why not mobi too ? if only for the sake of science... so i installed MobiPocket Creator and first i tried making a book from the html, then from the word files.

there are 2 separate problems with this particular file :

1. the unsupported glyphs. only possible solution (barring an update to the imp firmware and eBook Publisher programme from ETI, which i am hoping for but hardly seems likely to appear tomorrow, or me buying a Gen3 *especially* to read this book...) : make images. not feeling zealous enough to do that today.

2. the inexplicable messed-up text formatting and page layout. this is clearly a problem from *somewhere* in the css, because if i remove the css file and build an imp (in eBook Publisher), the text reverts to default of left-aligned, no bold, etc. *however* i still have not figured out where, because the css file (which is REALLY long) does not seem to contain the corresponding rules (text align right, huge left margin, etc. ; when i search for these rules in the css, they only appear on classes which are not actually used anywhere in the document, and the classes applied to the text have the right rules).

so, back to my trials (and tribulations ) when i put the html file through MobiPocket creator, i wasn't sure how to attach the css file, and it turns out it is not imported automatically (first time using MPC : it was quite easy, but i imagine there are some details i don't understand). the glyphs are supported in mobi files, however without the css file there was no formatting at *all*, so the book was not very nice to look at. i didn't bother to make an imp there, because i knew the result would be the same as when i tried to make an imp directly from the html without css.

then i tried using the doc file, which was formatted (it was the original html file, opened in word, and saved as .doc, with some reformatting done by me afterwards, notably to remove the rather surprising page breaks after every paragraph return).

this file of course gave much better results, about the same as when i made the imp directly from the doc file, although i lost some features which i like in imp (how do you make a running header in mobi ? also, how do you specify a serif font for the body text ?).

Conclusion of the experiment : using mobipocket creator as an intermediate version only adds additional steps to the conversion, with no apparent improvement, and in fact loss of some features (header, serif font). however, making a mobi file as the *destination* file is an improvement over the equivalent imp file in this case, in that the mobi format correctly supports the special characters.

Conclusion part 2 : experimentation overall inconclusive, as i still don't understand why this blasted file interprets the css in this ridiculous manner (especially since firefox can interpret it perfectly, and word interprets it with also some strange things but not the SAME strange things). it is the second time it has happened to me on a drm lit book (2 out of a total of 4 books converted... not a very good score...), and *never* on a non-drm conversion from mobi format or creation from zero of a PG text. it is really infuriating and i will definitely hesitate before buying another drm book.

[DaleDe]

Originally Posted by jbenny

If you look around on the ETI site, they have a list of all the extended characters supported by their readers. I used to have a copy, but can't seem to find it.

It is all in the wiki - check special characters.

Dale

[nrapallo]

Originally Posted by DaleDe

It is all in the wiki - check special characters.

Dale

Attached is a copy of the Special Characters Wiki in .IMP formats for viewing on the ebook reader or PC viewer.

Please note that the filename ending in '_1200.imp' is for the REB 1200; the other '.imp' for the EBW 1150.

EDIT: an updated Special Characters Wiki .IMP replaces the files below.

[DaleDe]

Originally Posted by nrapallo

Attached is a copy of the Special Characters Wiki in .IMP formats for viewing on the ebook reader or PC viewer.

Please note that the filename ending in '_1200.imp' is for the REB 1200; the other '.imp' for the EBW 1150.

Interesting, but some of the characters are lost in the conversion process. I am not sure where the errors were introduced but the display of the document is clearly missing some characters.

If you download the Gemstar formatting standard from the wiki on the eBook Publisher page you will find that I added the changes and special characters specific to the 1150. I believe there are other IMP sources for this as well including at least one that you generated and some others by me.

Dale

[DaleDe]

For those interested in special characters there is a web site that shows most of them. http://webdesign.about.com/library/bl_htmlcodes.htm

There is even a grouping for the specialized http://webdesign.about.com/od/locali...lcodes-pro.htm Pronunciation codes.

Dale

[zelda_pinwheel]

at this point, i'm not so much interested in special characters, as interested in getting an update to the eBook Publisher / eb1150 firmware / imp format / whatever, so they will be correctly supported.

[jgray]

I think it is a matter of the firmware in the reader not supporting some characters.

[zelda_pinwheel]

Originally Posted by jgray

I think it is a matter of the firmware in the reader not supporting some characters.

ok, well then we need a firmware update. what do you think, should i get my hopes up ?

[DaleDe]

Originally Posted by zelda_pinwheel

ok, well then we need a firmware update. what do you think, should i get my hopes up ?

Probably not, and there is probably not room for the entire UTF-8 characters sets anyway. I suspect there are quite a few characters needed more than a pronouncing dictionary set but then I could be wrong.

Dale

[zelda_pinwheel]

Originally Posted by DaleDe

Probably not, and there is probably not room for the entire UTF-8 characters sets anyway. I suspect there are quite a few characters needed more than a pronouncing dictionary set but then I could be wrong.

Dale

not enough room in what ? do you mean the internal memory ? would the special characters really take up that much space ? wouldn't it just be a matter of integrating a more complete font ? Times New Roman and Arial contain probably every glyph you could imagine (and more !) but the font file is only a few kb ; shouldn't that be enough ? what else do you need to support the glyphs, apart from a fontface which contains them ?

anyway if that's the case and it's impossible to update the eb1150 / 1200 to support these glyphs, then i really hope they make these changes in the next versions, and that the next versions become available soon.

it would be nice if they could support unconverted html and /or mobi as well in the next versions (although i don't get the impression this is necessarily in their plans) ; both of these had no trouble displaying this file, and the html got the formatting right on the first try, too. not to mention how much easier it would be to just slide an html file onto the memory card, without worrying about conversion, end formats, etc... and, when you found an ocr mistake in the PG book you were reading, you could just correct it in the original html file, and transfer the new copy over ! so simple.