New MOBI output plugin, testing needed

[kovidgoyal]

Hi all,

calibre version 0.8.16, released today, has a new MOBI output plugin. This is based on a month long effort to further reverse engineer the parts of the MOBI format that were still opaque.

The major bug that the new plugin fixes is that periodicals (news/magazines) created with calibre now have fully working navigation on the Kindle 3. When you click on View articles and sections while reading an article, the Kindle now returns to the current article in the list of articles. See https://bugs.launchpad.net/calibre/+bug/748741 for details

In addition various small things were figured out that should make the MOBI files produced by calibre have a binary structure closer to those produced by Amazon's tools. Note that the "binary structure" refers to internals of the MOBI file, not the text formatting, which remains the same as before.

Since the new plugin is a complete rewrite, it may well have bugs, so I would appreciate it if a few of you turned it on and tested it over the next week or two, before I release it widely.

To enable the new plugin, go to Preferences->Tweaks, click the "Plugin Tweaks" button and type

new_mobi_writer = True

into the box and then click OK and Apply. This will enable the new MOBI output plugin.

Report any problems you encounter here.

Thanks.

[Joseph R]

I volunteer..

Thank you for your hard work!

[kovidgoyal]

You're welcome

[user_none]

@Kovid,

Are you going to update the MOBI format documentation file in the format_docs folder with all you've learned? Having a better understanding of the structure of the MOBI format would benefit a lot of people. Not everyone is comfortable reading Python code to understand the format.

[kovidgoyal]

Nope, I much prefer my documentation to be at source level. The new MOBI writer is heavily commented, but that's about what I'm willing to do documentation wise.

EDIT: calibre also includes a tool to decompile MOBI files into a nice ASCII representation, which should really help with understanding the MOBI format as well, calibre-debug --inspect-mobi filename.mobi (corresponds to the --extract-to argument to ebook-convert)

[Bratzzo]

Hi,

Originally Posted by kovidgoyal

The major bug that the new plugin fixes is that periodicals (news/magazines) created with calibre now have fully working navigation on the Kindle 3. When you click on View articles and sections while reading an article, the Kindle now returns to the current article in the list of articles. See https://bugs.launchpad.net/calibre/+bug/748741 for details

this is great!
Until now, I not even knew that I would miss it!

Thank you!


But one thing, I don't like.
And I don't know if this is a bug or a feature.

At the beginning of every new section, it starts with a small section-TOC just above the first article of the section. But only if I use the pageturn buttons.
If I use the Kindle 5-way, it only shows the article.

Maybe the images explain what I mean.
The first image shows the first article of a new section when I use the 5-way to change between articles. It looks OK.
The second image shows the first article of a new section when I use the pageturn button. The red box shows the small TOC.

I don't like the small TOC above the new article. I would either prefer a seperate page at the beginning of a new section, or no seperate TOC at the beginning of a new section.



Bratzzo

[kovidgoyal]

That small TOC is necessary to make the Kindle's View Articles and Sections feature work, I agree it's annoying, but if you dont have it, the feature doesn;t work...don't ask me to explain why.

[Bratzzo]

Thanks for your answer.

But why it shows the TOC only if I turn the page, but doesn't shows the TOC if I change the article?

And now, I discovered this:

last page last article section 1 --> 5-way to the next article (first article section 2) --> article without TOC --> pageturn backwards --> same article with TOC --> pageturn backwards --> last page last article section 1.

This is a litte annoying. But thanks god, I seldom turn backwards.

While the benefit is much bigger than the "pain", I think, I have to accept this.



Bratzzo

[kovidgoyal]

changing the article goes from article to article and skips content in between, its like the chapter to chapter jumps in a book.

[bobb40]

Originally Posted by kovidgoyal

To enable the new plugin, go to Preferences->Tweaks, click the "Plugin Tweaks" button and type

new_mobi_writer = True

into the box and then click OK and Apply. This will enable the new MOBI output plugin.

Does that enable it for CLI?

[kovidgoyal]

yes, it does.

[bobb40]

Originally Posted by kovidgoyal

Report any problems you encounter here.

Using ebook-convert from CLI, after enabling new o/p module, converting text to mobi, source is part html, part markdown.
Paragraphs beginning with:
<p style="text-indent: 0; margin-left: 3em; text-align: left;">
were placed oddly; ran with --debug-pipeline, and found the style in the
"processed" directory stylesheet.css but with text-align set to "justify"; edited the stylesheet, zipped directory, ran ebook-convert on zipfile, and paragraph placement is as expected. So the inline style text-align got lost somewhere along the way.

[kovidgoyal]

That has nothing to do with MOBI output, its a txt input issue. You should open a bug report and attach a txt file demonstrating the problem.

[siebert]

Hi,

I've noticed that with the new output plugin calibre pretends to be kindlegen by setting the EXTH records 204-207.

Unless this is necessary for some features to work (which would surprise me), I'd lobby to either use a unique value for record 204 which would identify calibre and set records 205-207 according to the current calibre version or, as we can't predict which values for record 204 amazon will use in the future and the calibre version is already available in the contributor record, don't set the records 204-207 at all.

As the calibre output can't be 100% identical to the kindlegen, no matter how much effort we put into it, pretending that kindlegen created the mobi file might break tools which depend on the kindlegen version to be correct.

I've also seen that calibre has a kindlegen option which when enabled uses the kindlegen binary to create the mobi file instead of the calibre python code.

It looks like a debugging option and I couldn't find a way to enable it in the calibre gui, but as this option is the best of both worlds for some users, having the epub conversion done by calibre, which is far superior than the epub conversion from kindlegen, together with a mobi binary created by the official amazon tool kindlegen, it would be great if this would be a user selectable option on the mobi output option screen of the gui.

With that option the AZW Conversion Output Plugin (https://www.mobileread.com/forums/sho...80&postcount=1) should be obsolete, too.

Ciao,
Steffen

[kovidgoyal]

As you say, the fact that calibre created the MOBI file is documented in the creator record. So if some tool needed to differentiate between calibre generated and kindlegen generated files (highly unlikely since there are multiple versions of kindlegen as well as non kindlegen tools amazon uses to produce MOBI files) that tool can check for the creator EXTH record containing calibre. I have just about zero motivation to generate the half a dozen different types of MOBI files and check that everything still works on a Kindle without the 204-207 records. Incidentally, does such a tool actually exist?

--kindlegen is a debug option and as such is not suitable for inclusion in the GUI. I do not intend to maintain it/make it robust. Maintaining one set of mappings from epub to mobiml is quite enough work, thank you. Remember that running kindlegen does not just create the binary wrapper it also re-processes the markup. In most cases this is harmless, but I highly doubt it will be harmless in all cases. And then kindlegen is a moving target, one over which I have no control.