[GUI Plugin] KindleUnpack - The Plugin

[DNSB]

Originally Posted by m_pluto

Feature request: With the new kepub support in calibre 8.0 release, would it be possible to add an option to add a kf8/azw3 unpack to kepubs in addition to the kf8/azw3 to epub?

That is well beyond what this plugin is designed to do. Asides, the difference between a kepub and epub is only noticeable when using a Kobo ereader. Since calibre now has a built-in conversion to kepub when sending to a Kobo ereader that takes care of that possibility.

[m_pluto]

I love this Plugin because of how it completely preserves how the HTML & CSS is within the file, particularly if you have multiiple selectors for a tag, rather than what Calibre does which is truncate them into one single selector. (It's not my preference to have multiple sectors for how I buld my books, but rather that most of the ebooks I read are formated this way.)

Having said that, if it is out of the scope/abilities of the Plugin, that is fair and I understand! Thanks for the quick reply!

[DNSB]

Originally Posted by m_pluto

I live this Plugin because of how it completely preserves how the HTML & CSS is within the file, particularly if you have multiiple selectors for a tag, rather than what Calibre does which is truncate them into one single selector.

That is one of my favourite items about this plugin. If the Amazon azw3/KF8 format ebook was generated from an ePub, the ePub it generates comes very close to the original. In many ways, I prefer this to the calibre conversion approach which often bears little resemblance to the CSS of the original ePub or even the CSS you see when opening the azw3/KF8 with calibre's ebook-editor.

Admittedly, I have a set of saved searches in Sigil that remove much of the complexity from Vellum generated ePubs but does require something very close to the original ePub to be useful.

[DiapDealer]

I know absolutely nothing about Kobo's keypub format. Nor do I care to learn at this point. I don't mean that rudely. I simply don't have the time nor the interest.

[m_pluto]

Originally Posted by DiapDealer

I know absolutely nothing about Kobo's keypub format. Nor do I care to learn at this point. I don't mean that rudely. I simply don't have the time nor the interest.

Which is a fair and valid response! Thanks for the response!

[mallomar]

I've been using the Kindle Unpack plugin to convert some fixed-layout children's books from AZW3 to EPUB, and I'm running into some formatting issues.

When I convert the AZW3 files of "The Berenstain Bears Go to Camp" and "Noah's Ark" by Peter Spier using the KF8 to EPUB option, the resulting EPUB has formatting problems when viewed in Apple Books, Readest, and BookFusion. The issues are consistent across all non-Kindle reading apps.

The AZW3 files display perfectly on a Kindle device (I sideloaded the files I bought on my account, with DRM stripped, to my daughter's Colorsoft). When I open the DRM-stripped AZW3s directly in an app that supports it (like Readest), the same formatting issues appear as in the converted EPUB.

This suggests there's some Kindle-specific formatting in the AZW3s that isn't being correctly interpreted or converted to a standard EPUB format that other readers can understand.

I have the DRM-stripped AZW3 files for both books that I am providing scrambled for your analysis. This might help in debugging the conversion process for these types of fixed-layout books.

Thank you for your time and for this very useful plugin.

[DiapDealer]

It could also mean that the Kindlebooks weren't created from epubs. Kindleunpack does the best it can in such situations, but somethings could easily get shoehorned into the resulting epub. I'm not entirely certain KindleUnpack fully supports unpacking fixed layout Kindlebooks to begin with.

Thank you for the scrambled samples. They should prove useful. Also keep in mind that the inability to properly render the drm-free azw3 could simply be the fault of non-Kindle apps/renderers such as Readest

[mallomar]

Thanks so much for the quick and detailed response. That all makes sense, and I appreciate you clarifying the complexities. Fortunately Kindle Unpack has worked without issue for all other fixed layout azw3 I’ve used it with, which is why I thought something strange is going on with these two books.

I'm glad the samples might be useful. Please let me know if you end up needing the original, non-scrambled files to fully replicate the issue. I'd be happy to provide them privately

[DiapDealer]

Any particular rendering issues you can help target for us? I have Kindle devices I can view the originals (scrambled drm-free azw3s) on, but knowing exactly where to look will help a lot.

This is probably an issue for the original KindleUnpack (this plugin is primarily a wrapper around that), by the way.

https://github.com/kevinhendricks/KindleUnpack

[mallomar]

Originally Posted by DiapDealer

Any particular rendering issues you can help target for us? I have Kindle devices I can view the originals (scrambled drm-free azw3s) on, but knowing exactly where to look will help a lot.

This is probably an issue for the original KindleUnpack (this plugin is primarily a wrapper around that), by the way.

https://github.com/kevinhendricks/KindleUnpack

Hi, thanks for looking into this and for clarifying that this is likely an issue for the core KindleUnpack tool. I'm happy to help target the issues.

Here are two specific examples with supporting details from other developers who have looked at the unpacked EPUBs.

In The Berenstain Bears Go to Camp, the text on the title page is not contained in its bounding box.

This is visible in multiple readers (Apple Books, Readest, BookFusion). The developer of Readest confirmed this is an issue in the EPUB's code. You can see his analysis in the ticket I filed with him here, where he notes the bounding box is missing. BookFusion support also confirmed it was an issue with the EPUB file itself.

In Noah's Ark, there is overset text on the third and fourth pages.

The developers at BookFusion investigated this and confirmed it's an issue with the EPUB file that is present across different reading apps. Here is what they said:

We looked at this EPUB and it appears to be an issue with that page. We also quickly checked other apps including Apple Books, Readest that you suggested and others seem to also have the same rendering issue for this page.

I am attaching screenshots showing a comparison for each book below. Let me know if you need anything else!

[DiapDealer]

Ok. I have it on good authority (thanks KevinH!) that KindleUnpack had no real issues with these two fixed layout Kindlebooks. In general, there should be no problem unpacking KindleBooks of this nature. This is why you've rarely run into any issues when extracting fixed layout books in the past.

The problem with these two books is that bounding boxes weren't really used. The book creators used a bit of a hackish solution to make sure text fit within a certain area of the image. They relied upon font sizing and metrics in order to size the text to fit exactly where they wanted it. The difference is that official Kindle devices are using different font layouts, metrics and kerning that simply don't hold up after being extracted into an epub. Even with an extracted embedded font. That's also why even the DeDRMed Kindlebook was showing the same rendering problems when viewed in a non-Kindle app (pre-unpacking).

When I open the DRM-stripped AZW3s directly in an app that supports it (like Readest), the same formatting issues appear as in the converted EPUB.

Unfortunately, there's really nothing we can do with KindleUnpack to accommodate this.

[KevinH]

As DiapDealer said, there is nothing KindleUnpack can do. But with some very slight css modifications to your Berenstain epub, you can fix the issue.

If you modify OEBPS/Styles/style0001.css

And after modifying the following two css rules, it all appears to work at least for some font layout engines:

p.copyright {
  font-size: 28%;
  line-height: 75%; /* was 118% */
  margin-bottom: 0;
  margin-top: 0;
  text-align: center;
  text-indent: 0;
  letter-spacing: -0.5px;  /* added */
}



div.page3_item1 {
  left: 49.22%;
  position: absolute;
  right: 0;
  top: 43.5%; /* was 44.5% */
}

So you can compensate for the difference between the Kindle font layout engine and Calibre's or Sigil's QtWebEngine font engine with just very slight css modifications.

I am sure any other anomalies inside that epub can be fixed in a similar manner. And I am sure the same thing holds for the other Noah epub but since I did not have a version of it, I could not given you hints as to where to fix them.

Unfortunately the scrambled versions of the azw3s even scrambled the images and thus were next to useless especially given the problem was not in KindleUnpack.

Hope this helps.

[KevinH]

Update: Those changes worked with Sigil and PageEdit but the problem was still there in Books on my iPad and in the BlueFire reader on my iPad. So different font engines will need different CSS which is the whole problem with the idea of Fixed Layout epubs that try to position text absolutely.

Getting a PDF / Kindle Print Replica version is simply better than trying to fight with fixed layout epubs different across devices.

[mallomar]

DiapDealer and KevinH,

Thank you both so much for your time and the incredibly detailed analysis. Your explanations were extremely helpful and have shed a lot of light on why these particular books are problematic.

Given that even the proposed fix doesn't work consistently across all readers, as KevinH pointed out, I agree that it's not worth the effort to try and fix them.

This has been a valuable lesson. In the future, I will definitely take your advice and seek out PDF or Print Replica versions if these kinds of fixed-layout books have similiar issues.

Thanks again for all the help!