Mobipocket output

[kovidgoyal]

calibre 0.5 does not have support for MOBI indexing (that's what is responsible for the waypoints and joystick based navigation on the Kindle). calibre 0.6 however does, thanks to some nifty reverse engineering by Gregory Riker.

As for support for special mobipocket markup, it should be possible to add support for it to calibre, however, it's not really a priority for me, since most calibre users will never use it. Still if someone wants to take it on, I am always happy to accept patches.

[sirbruce]

Originally Posted by kovidgoyal

calibre 0.5 does not have support for MOBI indexing (that's what is responsible for the waypoints and joystick based navigation on the Kindle). calibre 0.6 however does, thanks to some nifty reverse engineering by Gregory Riker.

But it's still not turned on in 0.6b6, right?

[kovidgoyal]

Originally Posted by sirbruce

But it's still not turned on in 0.6b6, right?

It is (it's what causing most of the bugs you reported :-) Try downloading news using b6 and you will see it in action

[sirbruce]

Originally Posted by kovidgoyal

It is (it's what causing most of the bugs you reported :-) Try downloading news using b6 and you will see it in action

Oh cool, I hadn't tested it yet. It works!

[Sunlite]

I have tried to convert html to mobi with the any2mobi tool using a intermediated opf-file. On most parts the conversion works very well. But in one of my sources there is a text part that has a line through it. That is converted to normal text.

The original uses the <strike> tag, but I did tried <span style="text-decoration: line-through">, too. Both are lost when using any2mobi and both are retained by mobigen.

<p>strike through:<br />
<strike>This is a sentence with a strike through.</strike><br />
<span style="text-decoration: line-through">This is a sentence with a strike through.</span><br />
</p>

I extracted the raw html from the two mobi versions with mobi2html from the mobiperl tools. In the any2mobi version the text adornment is just gone. In the mobigen version both lines now use the <strike> tag.

html any2mobi:

<p align="justify" height="1em" width="0pt">strike through:<br />
This is a sentence with a strike through.<br />
This is a sentence with a strike through.<br />

html mobigen:

<p>strike through:<br />
<strike>This is a sentence with a strike through.</strike><br />
<span><strike>This is a sentence with a strike through.</strike></span><br />

Attached are my test html file (cleverly named test.html) and the two mobi versions.

Am I doing something wrong? Is there a way to conserve strike through text with any2mobi?

[tompe]

Originally Posted by kovidgoyal

calibre 0.5 does not have support for MOBI indexing (that's what is responsible for the waypoints and joystick based navigation on the Kindle). calibre 0.6 however does, thanks to some nifty reverse engineering by Gregory Riker.

Is this format information documented anywhere? In the Wiki?

[kovidgoyal]

@Sunlite: Open a ticket for strike through support, IIRC, calibre's MOBI output doesn't implement it

@tompe: In the source code, but, GRiker who did the reverse engineering plans to release some documentation of it of soon

[llasram]

Originally Posted by kindleformatting

As I outline in my Kindle Formatting book, there is a lot of HTML formatting that is not valid in Mobi/Kindle, and some specific Kindle formatting that is not necessarily valid HTML. An example of this problem is the lack of support for outlines made from list tags.

Can you point to an example of this Mobipocket/Kindle-optimized markup? I haven't been active in calibre lately, but my underlying principle when I initially wrote the Mobipocket generation code was to accept standards-compliant markup and CSS as input and produce Mobipocket-optimized markup as output. So for me the ideal solution would be to figure out what XHTML+CSS most closely represents the output you want, and figure out how to produce the Mobipocket-optimized markup from that input.

[chorpler]

Is this something that will allow ebook-convert's Mobipocket output to create navigation entries? The ones you use by adding <reference> tags under the <guide> section of the .opf file? Most of my custom Mobipocket books include things like Table of Contents, Copyright Page, Dedication, Notes, etc. When I create the book with mobigen, these entries show up in the Contents menu (in the desktop Reader app) or in the Navigation menu (in the Windows Mobile app). But when I create them with ebook-convert, which is of course the only way to get larger images to store properly in the books, only the Table of Contents shows up. I've tried searching to see if I'm missing something, but I haven't been able to find out.

[kindleformatting]

Originally Posted by llasram

Can you point to an example of this Mobipocket/Kindle-optimized markup? I haven't been active in calibre lately, but my underlying principle when I initially wrote the Mobipocket generation code was to accept standards-compliant markup and CSS as input and produce Mobipocket-optimized markup as output. So for me the ideal solution would be to figure out what XHTML+CSS most closely represents the output you want, and figure out how to produce the Mobipocket-optimized markup from that input.

The outlines issue I mentioned is a pretty common problem. While you can make a nested list in HTML and give list-style-type properties to the lists, those outline-type lists do not translate into Mobipocket. I have had to create a hack to make nested lists work properly in Mobi, and they look especially good on the Kindle.

Another issue is that calibre does not support the <mbpagebreak/> tag. I know it is not valid HTML, but it is valid for Mobi files so it would be nice to see calibre use it.

calibre also requires that any attached NCX file be added to the end of the document, even when there is a TOC in the HTML, and that NCX-generated TOC takes the focus of the TOC guide element.

I have also noticed some differences in how calibre interprets and displays some basic formatting, like line heights and heading font sizes.

I have attached a sample book with these issues all shown. Source files are included, as well as a mobi file created by calibre and a prc created by Mobipocket Creator.

- Joshua

[llasram]

Originally Posted by kindleformatting

The outlines issue I mentioned is a pretty common problem. While you can make a nested list in HTML and give list-style-type properties to the lists, those outline-type lists do not translate into Mobipocket. I have had to create a hack to make nested lists work properly in Mobi, and they look especially good on the Kindle.

I'll give the samples you attached a look this evening. It would certainly be possible to translate 'display: list-item' elements into something other than Mobipocket HTML <li/> elements if it yields better results across most Mobipocket renderers.

Originally Posted by kindleformatting

Another issue is that calibre does not support the <mbpagebreak/> tag. I know it is not valid HTML, but it is valid for Mobi files so it would be nice to see calibre use it.

I'm not too keen on adding support for Mobipocket-specific markup when there is an exactly comparable standards-compliant mechanism available. In this case, CSS 'page-break-before' and 'page-break-after' will let you do exactly the same thing. That said, the OPS specification provides a mechanism for allowing markup to contain XML from foreign namespaces that not all processing applications understand. Adding support for that to calibre could potentially allow it to understand Mobipocket-specific markup it could ignore when generating any other output types.

Originally Posted by kindleformatting

calibre also requires that any attached NCX file be added to the end of the document, even when there is a TOC in the HTML, and that NCX-generated TOC takes the focus of the TOC guide element.

If you generate the Mobipocket book from an OPF file instead of a directly from the HTML, and in the OPF specify a //guide/reference of @type="toc", then calibre won't generate an HTML TOC from the NCX file and will use the provided TOC guide reference in the Mobipocket output.

Originally Posted by kindleformatting

I have also noticed some differences in how calibre interprets and displays some basic formatting, like line heights and heading font sizes.

Line-height I'm a little confused on, as I don't recall a way to control line-height in Mobipocket markup. Does mobigen do special processing for blocks with e.g. a line-height greater than 2x the default line-height?

Headers will definitely be a little different though -- instead of passing through <hn> tags, calibre just groks the (possibly default) CSS and produces pure-formatting output as-similar-as-possible to the supplied styling.

-Marshall

[llasram]

Originally Posted by chorpler

Is this something that will allow ebook-convert's Mobipocket output to create navigation entries? The ones you use by adding <reference> tags under the <guide> section of the .opf file?

Create an OPF file, add //guide/reference tags, and convert the OPF file instead of the source HTML directly? That is supposed to work, although I don't have the code open in front of me ATM.

[chorpler]

Originally Posted by llasram

Create an OPF file, add //guide/reference tags, and convert the OPF file instead of the source HTML directly? That is supposed to work, although I don't have the code open in front of me ATM.

That's what I'm talking about -- I do all my conversions that way, because you get much better control over the final output if you work from the OPF file manually. mobigen reads all the //guide/reference tags in the OPF file and creates the corresponding //head/guide/reference entries in the Mobipocket markup in the output file, such as converting the OPF file's entries:

<guide>
     <reference type="title-page" title="Title Page" href="generic_file.xhtml" />
     <reference type="toc" title="Table of Contents" href="generic_file.xhtml#toc" />
     <reference type="copyright-page" title="Copyright Page" href="generic_file.xhtml#copyright" />
</guide>

into the following in the Mobipocket markup in the .mobi file:

<reference title="Title Page" type="title-page" filepos=0000000681 /><reference title="Table of Contents" type="toc" filepos=0000002434 /><reference title="Copyright Page" type="copyright-page" filepos=0000001140 />

But with ebook-convert, the //guide/reference tags seem to be ignored if they aren't of type toc, because the only //head/guide/reference entity is Table of Contents.

Edited: Hmm, I've actually been misinterpreting it. I just noticed that the //head/guide/reference entry that ebook-convert creates is to the automatically generated Table of Contents link at the end of the file, rather than to the existing Table of Contents link in the OPF file. So it's really completely ignoring the //guide/reference tags in the source OPF.

Edited again:: I just noticed that calibre 0.5.14's oeb2mobi command does, in fact, translate the //guide/reference tags properly. So the problem must be with ebook-convert not realizing it should read and use those tags instead of generating its own Table of Contents. Or maybe there's a way to disable Table of Contents generation that I haven't figured out? I tried using the --chapter=/ option to disable chapter detection, but that didn't work.

[llasram]

Originally Posted by chorpler

That's what I'm talking about -- I do all my conversions that way, because you get much better control over the final output if you work from the OPF file manually.

As you noticed, that support has accidentally gone away ATM in 0.6 due to some internal changes. I'll see if I can make the changes necessary to get it working again.

[chorpler]

Originally Posted by llasram

As you noticed, that support has accidentally gone away ATM in 0.6 due to some internal changes. I'll see if I can make the changes necessary to get it working again.

Awesome. I was trying to look at it myself but I'm having major trouble getting a Calibre development environment set up -- the VMWare machine won't load for me and I can't install some of the necessary tools because Visual Studio 2008 won't install because some weird Windows error is making it impossible to install the .NET 3.5 framework. I'm going to just set up a virtual machine and run it from there.

But thanks, you'll probably do a much better job of it than I could, since as I understand it you wrote the Mobipocket output in the first place, right?