kindlegen plugin for Sigil
 

[Plugin] KindleGen - Simple KindleGen wrapper

Updated: December 5, 2021
Current Version: "0.6.0"

This plugin is a very simple kindlegen wrapper for converting epubs to mobi files with kindlegen. (If you're a macOS user or using Sigil 0.9.8 or higher, please download KindleGenQt_v0.6.0.zip, otherwise download KindleGen_v0.5.5.zip.)

Disclaimer: Even though it's unlikely that the plugin will somehow corrupt the epub that you're working on or crash Sigil, you might want to save your ePub in Sigil before running this plugin.

Credits: This plugin uses code originally written by KevinH, nickredding and other KindleUnpack developers. It also uses jhowell's Calibre KFX Output plugin.

System requirements

This plugin requires the Kindlegen binary or Kindle Previewer, which comes with KindleGen. If you uncheck the Use Bundled Python option, you'll also need to install Python, bs4, lxml and Pillow.

Linux users will need to download the kindlegen binary from the Internet Archive and unzip it to a /bin folder.

Installation

1. Select Manage Plugins from the Plugins menu. Select Use Bundled Python to use the bundled interpreter.
If you haven't updated to Sigil 0.9.0 (or higher) click one of the Auto buttons to detect the path or Set to manually select the Python interpreter path. You also might need to install bs4, Pillow and lxml.
2. Click Add Plugin and select KindleGen_v0.6.0.zip. This will install the kindlegen plugin, which you can select via Plugins > Output > KindleGenQt.
If you want to generate KFX files, you'll also need to install Kindle Previewer and jhowell's Calibre KFX Output plugin.
Since the Calibre KFX Output plugin doesn't support Linux, you can't generate KFX files with the Linux version of Sigil.

When you run the plugin for the first time, you'll need to select the kindlegen location, unless you have installed Kindle Previewer in the default folder. Select either kindlegen.exe (Windows) or kindlegen (macOS, Linux).

Settings

By default, the plugin will generate .mobi files using the same parameters as Kindle Previewer (master mobi file, no compression, attached source files).

There are two groups of settings:

1. KindleGen settings that are passed on to the KindleGen binary and allow you to:

• Prevent KindleGen from attaching the source files.
• Change the compression factor (0 = no compression, 2 = maximum compression).
• Display verbose messages.
• Force Windows-1252 output.
• Convert .jpg images to .gif files.
• Change the language used to display KindleGen messages.

2. Postprocessing options that allow you to:

• Add a fake ASIN number (using DualMetaFix).
• Split the master mobi file into azw3 and mobi7 components (using KindleUnpack).
• Generate a KFX file with jhowell's Calibre KFX Output plugin (Windows und macOS only).
• Generate a thumbnail image for eink Kindles.
• Change the output folder (default = Desktop).

If you select any of the post-processing options, you can't upload the AZW3/MOBI7/KFX files generated by the plugin to KDP!

The plugin will also check for the presence of the following recommended guide/landmarks/metadata items:

• SRL = Start Reading Location*
• TOC = HTML TOC
• Cover = cover metadata entry

Advanced Settings

* According to the Kindle Publishing Guidelines, the SRL has been deprecated. (If you don't use the SRL, edit KindleGen.json and change "check_srl": true to "check_srl": false.)

License: GNU General Public License v3 (GPL-3)

Hi Doitsu,

Already 19 downloads .... Great Work.

Thank you!

KevinH

I am trying this out on my linux mint 17.1 rebecca x64 system, but unfortunately am getting errors:
================================================== ==
Status: success

Running KindleGen ... please wait

************************************************** ***********
Amazon kindlegen(Linux) V2.9 build 1028-0897292
A command line e-book compiler
Copyright Amazon.com and its Affiliates 2014
************************************************** ***********

Info:I9006:option: -c1: Standard DOC compression
Info:I9014:option: -verbose: Verbose output
Info(prcgen):I1047: Added metadata dc:Title ""Sweet Young Things""
Info(prcgen):I1047: Added metadata dc:Date "2015-11-03"
Info(prcgen):I1047: Added metadata dc:Creator "Kristine Kathryn Rusch"
Info(prcgen):I1047: Added metadata dc:Contributor "calibre (1.25.0) [http://calibre-ebook.com]"
Info(prcgen):I1047: Added metadata dc:Source "Free Fiction Monday 2014/11/17"
Info(prcgen):I1052: Kindle support cover images but does not support cover HTML. Hence using the cover image specified and suppressing cover HTML in content. URL: /OEBPS/Text/cover.xhtml
Info(prcgen):I1002: Parsing files 0000001
Info(prcgen):I1003: Parsing file URL: part0000_split_002.html
Info(cssparser):I10004: @rules other than @import, @charset and @font-face are not supported.
Info(prcgen):I1015: Building PRC file
Info(prcgen):I1006: Resolving hyperlinks
Info(prcgen):I1010: Writing hyperlinks
Info(prcgen):I1049: Building table of content URL: /OEBPS/toc.ncx
Error(prcgen):E24010: Hyperlink not resolved in toc (One possible reason can be that the link points to a tag with style display:none):/OEBPS/Text/titlepage.xhtml#
Error(prcgen):E24001: The table of content could not be built.
Info(prcgen):I1016: Building enhanced PRC file
Info(prcgen):I1007: Resolving mediaidlinks
Info(prcgen):I1011: Writing mediaidlinks
Info(prcgen):I1009: Resolving guide items
Info(prcgen):I1038: MOBI file could not be generated because of errors!

Kindlegen failed.

Please click OK to close the Plugin Runner window.

================================================== ==

What do I need to do to get this working?

Dave

The error seems pretty clear; there's an invalid hyperlink in the TOC.

I never created a TOC for this epub. It is a short-story and as such a TOC is unnecessary. All I really want in this epub is the cover and the text.

I ran the validator and get the following errors:

OEBPS/Text/titlepage.xhtml N/A This OPS document is reachable but not present in the OPF <spine>. "Reachable" means that a reference of some kind that points to this resource exists in the epub.

OEBPS/Text/titlepage.xhtml N/A This resource is reachable but not present in the OPF <manifest>. "Reachable" means that a reference of some kind that points to this resource exists in the epub.

OEBPS/toc.ncx 17 This <content> element's "src" attribute value is "Text/titlepage.xhtml", but that file does not exist.

I don't see a way to remove the TOC or to modify the OPF anywhere.

I was able to use ebook-convert from calibre to create a .mobi from this file, so I am surprised that kindlegen has problems with it.

Dave

All those errors seem to be saying the same thing: that you're referencing a file called "titlepage.xhtml", but the file doesn't exist.

calibre is a better piece of software than kindlegen -- and you're surprised to hear that? :rofl:

Correct, a file that is not needed for a short story does not exist. How can I change whatever needs to be changed to remove these references to an unnecessary file?

Dave

Regenerate your Toc.ncx (semantic ToC) -- or manually edit it -- to get rid of the reference to that nonexistent file...

Duh! Somehow I never expected it to be that easy. I was able to use kindlegen without problems after doing as you suggested.

The .mobi kindlegen made is substantially larger than the one ebook-convert made, over twice the size. Perhaps I need to tweak the kindlegen.ini settings to deal with this. I have both a K3 WiFi and a KT2, so I need a file that both can read, so I would appreciate some pointers.

Dave

kindlegen is creating a dual-MOBI.
I assume you mean it is bigger than a dual-MOBI that calibre creates. (Rather than being bigger than a standalone AZW3 which is just duh. ;))



Try using the undocumented switch to kindlegen -- by default it creates a MOBI that has both mobi7 && KF8... and also the original source files, because reasons. ;)
For kindlegen.ini that would be the "donotaddsource = False True" mentioned in the OP.

...

Also consider creating the standalone AZW3 and MOBI, and sideloading each one as and when needed.
Unless you intend to use Amazon PDOCs of course.

I changed the donotaddsource and azw3_only settings and now I get a .mobi file smaller than that generated by ebook-convert and a .azw3 file that is larger. I will confirm that these look good on my kindles and then will start using the plugin.

On a related topic, is there a way to improve the file name created? I left the mobi_file setting blank, so it used the title as a starting point. As this was a short story, the title was in double-quotes which were translated to underscores. The embedded blanks in the title were kept in the file name. My preference in this matter is to elide the double-quotes and translate the embedded spaces to underscores in the file name. It would be nice if this could be automated by the plug-in. I also normally convert embedded apostrophes to hyphens in the file name. I imagine a more general translate table might be needed for foreign languages titles to file names.

At this point I will simply rename the files manually, but it would be nice to automate it.

Dave

I don't know, doesn't sound like it.


I usually use calibre's metadata template language for that. In calibre. ;)

Calibre has extremely robust conversion settings that allow users to convert invalid ePubs to invalid Mobi files. These files can be opened with Kindles and Kindle apps, but some links/styles might not work.
You may want to make sure that all books that you want to convert with KindleGen pass ePubCheck and/or FlightCrew, because books that fail validation usually also fail KindleGen. Also note that azw3 and mobi7 files generated by this plugin can't be used for KDP uploads.

If you want to convert multiple epubs you're better off using Calibre.

If you're familiar with regular expressions, you can change the generated file name by editing line #285 in plugin.py.

You could also press F8 in Sigil and modify the title before running the KindleGen plugin. (You don't have to save the ePub after the change; the plugin will always use the current version of the files as they're displayed in Sigil.)

Thank you for the pointer to the appropriate code. I have tweaked it a bit to suit my preferences as follows:;

I edited the comment to be a little more accurate, added the apostrophe and white space to the list to be replaced with underscores and stripped leading/trailing underscores to deal with titles in quotes.

There might even be a few more unwanted characters that ought to be added to the replacement list that could cause problems in a file name. I will see if I run into any and add them as I encounter them.

The white space is a preference of mine to avoid the need of quoting filenames or using escape codes while entering file names. I find the underscores an acceptable replacement avoiding that problem.

Dave

Question
 

Would installing X11 (iMac running El Capitan, Sigil 0.9.0) affect the ability to run this plugin?

I downloaded the plugin to my downloads folder, and followed the instructions above to the letter...all was well, and the plugin worked as advertised.

I then separately downloaded Kindle Previewer to view the results, but in order to get Kindle Previewer to run, I had to update Java, and download X11, per the instructions here - http://www.amazon.com/forum/kindle%2...x2KDW3X9NV83MC

Again, all was good - Kindle Previewer ran as advertised.

I then uninstalled the plugin, and moved the zip file to its final resting place (in a folder in the Applications folder, rather than in the downloads folder) and reinstalled it, but now I can't get it to run.

Plugins-->Output-->KindleGen brings up the Plugin Runner window as it should. Clicking Start brings up the window (I notice the "Application" menu at the top of the screen changes from Sigil to Python) the to select the Kindlegen location (I want to select the Kindlegen_v0.2.2.zip file here, correct, or am I being an idiot?). I select this, click on "Open" and nothing happens (I left it for 30 mins just in case things were running incredibly slow). This window stays open. Selecting it a second time, and clicking Open again results in some text flashing in the Plugin Runner window, but much too fast for me to be able to read it, and then
Status: success
KindleGen not selected
Press OK to close the Plugin Runner Window

I checked the ini file and there has been no path to the Kindlegen executable written (I don't know if running the plugin the first time writes the path in here or not), and I also checked the permissions for the folder to ensure it was able to be written to.

The only thing that changed between the first time I ran it (when it was in the downloads folder) and the second time was the location of the Kindlegen.zip, and the fact that I updated Java and installed X11.

I have tried starting from scratch a few times now, removing all plugins (I have FlightCrew as well, which runs OK), quitting out of Sigil, restarting my machine, and trying to install again, all with the same outcome.

Am I being an idiot somewhere, or do I have genuine issue? If the former, I apologise. If the latter does anyone have any idea where, and how to fix it please?

In order run KindleGen, you only need the kindlegen binary, which comes with KindlePreviewer and is also available as a separate download. kindlegen itself doesn't require Java; it's only needed for KindlePreviewer

By default the Plugin looks for the KindleGen binary (kindlegen) in:

Of course the kindlegen binary can be in any folder.

The plugin zip file doesn't contain the kindlegen binary. You can delete it from your Mac after you've installed the plugin.

No, you'll need to select a file called kindlegen (without any extensions), which should be in /Applications/Kindle Previewer.app/Contents/MacOS/lib/kindlegen if you've installed Kindle Previewer in the default folder.

I'd recommend the following:

1. Select Plugins > Manage Plugins and delete the Kindlegen plugin and reinstall it.

2. Download the standalone OSX kindlegen version and unzip it to a folder of your choice.
(You only need to extract one file: kindlegen.)

3. Run the KindleGen plugin and select the kindlegen file that you extracted in step 2.

Worked a treat.

Awesome.

Thank you Doitsu.

I have now used the kindlegen plugin on over 100 documents and would up tweaking the plugin.py just a little more to remove more characters from the title for the filename:

# replace unwanted characters in book title
title = re.sub('[/|\u2019|\u201C|\u201D|\u2024|\u2025|\u2026|\?|<|>| \\\\|:|;|\.|,|\+|=|!|&|\*|\||\"|\^|\'|\s]+', '_', dc_title.group(1))
title = title.strip("_")

You might want to consider updating your script as well, and you can probably improve on my tweaks. I was surprised by the unicode in some of the document titles and think there might be a better way to deal with this, perhaps using ranges to remove non-alphabetics? In fact, I am starting to wonder if perhaps the best bet might be to invert the sense and define a list of permitted characters and translate/trim anything else.

Dave

Thanks for your suggestions. When I originally designed the plugin I had a look at lists of invalid characters in Windows, Linux and OSX file names and I believe that the plugin as it is right now generates valid file names for all three operating systems.

I'll have a closer look at your suggestions and might add a couple of characters to be removed from file names. For example, even though spaces are valid in Windows/Linux/OSX file names they usually need to be escaped.
OTOH, typographical quotation marks shouldn't cause any problems.

You don't want to use FlightCrew to validate. It doesn't catch enough errors. FlightCrew has needed updating for some time.

JSWolf,

This is really getting old.

It certainly can not hurt to run anything through Flightcrew. Please stop injecting this comment each time after flightcrew is mentioned anyplace in Sigil. In fact FlightCrew will catch many types of common errors and provide a nicer error code many times to help the user.

Then instead of just complaining about things ... file an actual bug report at the new Flightcrew github site and include a clear test case that Flightcrew calls wrong when it is not, or a clear test case of something it misses completely that it should catch that epubcheck 2 actually catches.

KevinH

Ok, I'll have a go at FC and see what it missed that ePubcheck 4 catches.

Do us a favor and don't bring your findings back here. It's not really relevant to this discussion. Start your own thread or file an issue with Flightcrew's issue tracker on github please.

I have a puzzle related to this plugin I am trying to sort out. I used this plugin to create a mobi file from an epub and loaded that file onto my Kindle 3 WiFi. I am having no problems reading the book, but if I hit "Goto" there is no table-of-contents available despite there being one in the original epub. In order to see if this was somehow left out by kindlegen I used the companion kindleimport plugin on the mobi file and was a bit surprised to see the table of contents was imported!

So it seems as if there is something peculiar about the generated mobi file that my K3 doesn't like even though the plugins have no problems with it. Obviously I can read the book without the table of contents, but it is annoying.

Dave

You'll need to mark the HTML TOC via Add Semantics > Table of Contents.

This will add the following line to the opf <guide> section:

Question
 

Doitsu,

Just a question.

Each time I run the Kindlegen plugin, I notice this error - "Info(pagemap):I8000: No Page map found in the book" - in the output, no matter what the book is or what I have done to it..

It's not a major, because as far as I can see, it doesn't seem to affect anything, but just wondering what this pertains to, and whether there is something else I should be doing in Sigil. I can't see any page map generation option in Sigil.

I am asking purely for my edification.

That message comes directly from KindleGen; you can ignore it, unless you're trying to generate a Kindle book with "real page numbers" from an epub with either an Adobe pagemap or an NCX pagelist in it.

Understand. I HAVE been ignoring it - but then I started wondering what it was.

Makes sense.

Thanks.

I'm encountering a problem with the output of this plugin on my Kindle (KT2) running 5.6.5 firmware. Now that the jailbreak is available for that release I have done so and installed the screensaver hack. This is working fine with the .mobi output of the plugin, but the .azw3 output is not operating properly.

The author of the screensaver hack has determined the problem, but it will need to be addressed here as it involves missing information in the file. Please see the thread:

https://www.mobileread.com/forums/sho...postcount=1829

Help.

Dave

Umm, all the plugin does is invoke kindlegen.

kindlegen is responsible for creating output that is sane according to Amazon's standards.

I am not sure that is the problem, as the .mobi file works properly. It is only the .azw3 being split out by the plugin that has the problem. Is it possble that something is being missed in the split process?

Dave

As NiLuJe said, the fault lies further on, with KindleUnpack.

Which is also something that the plugin is not responsible for -- go ask the KindleUnpack folks: https://www.mobileread.com/forums/showthread.php?t=61986

Unless I am missing something, the problem is the lack of an ASIN in the .azw3(KF8) file. These books do have an ASIN, but I guess it gets lost somewhere in the KindleImport/edit/Kindlegen process. How do I add the ASIN back in after the work is done? Did I miss a setting?

Dave

PS: looks like I did, I just set add_asin = True

This plugin is not designed to be used with ePubs generated by KindleImport.

However, you can have it automatically add an ASIN, by changing the following kindlegen.ini setting:

The plugin will then look for a metadata entry with the following format in the metadata section of the .opf file:

or

If it doesn't find one of these entries it'll generate a fake ASIN.

BTW, KindleUnpack will write the original ASIN as a comment to the re-generated .ePub. This commented-out ASIN will NOT be found by the plugin. You'll need to manually add it to the metadata section.

If this setting is enabled, it'll also generate a cover thumbnail for the .azw3 book that you'll need to copy to the /system/thumbnails/ folder.

BTW, if you just want to touch up some Kindle books, Calibre Editor is much better suited for this task, because it'll automatically unpack and reassemble AZW3 books.

Of course the ASIN gets lost in that process. Think about it:
KindleImport makes an azw3 an epub. Epubs don't have ASINs. Then you run that ASIN-less EPUB through Kindlegen (via the kindlegen plugin) to make it a dual format mobi. There's still no ASIN because kindlegen doesn't give its output ASINs. That happens later in the KDP process of making your book available for purchase on Amazon (or by using the fake/existing asin generation method Doitsu mentioned above).

There is no ASIN in the KF8/azw3 after splitting with KindleUnpack because the Kindlegen output didn't have an ASIN. It couldn't (again: unless you use Doitsu's method). The KindleUnpack split feature could make something up for an ASIN, but quite frankly, KU isn't in the business of manufacturing data that's not present.

Has anyone checked to see if KindleUnpack's split feature will honor the supplied (or faked) ASIN that kindlegen can be coerced into adding? If it doesn't, that would be the only way I could see that KU was doing anything "wrong." Otherwise it's a case of KU's splitter choosing to not make-up data that doesn't exist in the kindlegen output in the first place.

DiapDealer, the problem is that kindlegen generates dual-MOBIs which on the Kindle get an old-style hash for the cdeKey, and then KindleUnpack gives that a null value.

As far as the Kindle is concerned, a MOBI/AZW3 should either have an ASIN (calibre can create this, as can KevinH's dualmetafix) OR it should be a PDOC with a hash, but the KindleUnpack-split book is somehow breaking PDOCs, because they have neither.
Or:
It definitely appears to be a problem ;) but it also has nothing to do with either this plugin or ASINs. :shrug:



And now, I really think this discussion should move on over to the KindleUnpack thread, if anyone wants to continue discussing it.

Late to the party (well, over here anyway ^^), but I definitely agree with @DiapDealer that KU should absolutely not add stuff that wasn't there in the first place.

The fact that the Kindle framework can't deal with an ASIN-less KF8 properly is a bummer, but, well, out of KU's scope, I think? (KU is just doing its job, if there's an ASIN in the source file, there'll be one in the output ones).

Because after all, why the framework just craps out in fun and interesting ways with ASIN-less KF8 files instead of choosing to generate a md5 hash like it does for ASIN-less M7 or dual files is kind of a mystery...

It's just one of those quirks that one should be aware of when dealing with that kind of stuff. Which is why I'm happy to see that this plugin *does* have a way to handle it seamlessly ;).

So, yeah, maybe add a spiel to the relevant setting somewhere about how brokenly the Kindle (well, FW 5.x at least) handles KF8 files if there's no ASIN set, but that's about it. Can't make too many assumptions about what people will do with these files, after all ;).

Be that as it may; I was primarily addressing dhdurgee's statement concerning ASINs getting "lost" in his process involving this plugin. Which they most certainly are.

And if Kindlegen can be coerced into providing real (or fake) ASINs (and if KU's splitter keeps them), then I would say that ASINs are extremely relevant to the discussion. I mean, there must be a reason calibre is creating fake ASINs for its stand-alone KF8s, no? If it didn't, wouldn't those .azw3s suffer from the same problem (under NiLuJe's scenario)?

Hey NiLuJe. Nice to see you. :)

@DiapDealer: From a (very) quick glance at the plugin code, KindleGen doesn't really enter the equation, the plugin's using the dualmetafix python script to add a 12 character one from a cropped random uuid (I'm guessing to more closely match the length of a real ASIN) to the metadata of the KG output.

Since it's mobi header metadata, KU properly preserves that when splitting. So, all's well ;).

It's just that the option is disabled by default, and one may not be aware that trying to use such KF8 files directly will risk upsetting the Kindle's framework ;).

And while doing it by default makes sense for Calibre, I'd argue that @Doitsu's choice was the right one here? He documented most of the underlying issues fairly well, and even I had forgotten how deeply messy it could be, so ;p. (That, or it got 'worse' since FW 5.5.x. But I'm way too lazy to check on an older device ^^).