[Plugin] KindleImport Sigil plugin

[Hitch]

Originally Posted by DiapDealer

I can't take all the credit. KindleUnpack does a lot of the work, and then perhaps you remember an old python script called mobiml2html? KevinH has helped me tweak that old script so that it can get the old mobiml at least closer to xhtml. It's not perfect, but it's responsible for generating the css from the mobi markup. I just gather it all up into an epub for Sigil to open.

The AZW3 import stuff is all KindleUnpack. I just have to pipe KU's epub output to Sigil. If the AZW3 still has the source package attached from kindlegen (and the source is an epub), KindleImport opens that source epub rather than unpacking the AZW3.

So handsome, and so modest, too!

Thanks for the info.

Hitch

[DiapDealer]

Updated to v0.80.3 in the first post.

Changes are fairly minor, but they include:

• Updated to the latest developer version of the underlying KindleUnpack engine to incorporate a few bug fixes/improvements.
  
• Utilize the autostart/autoclose plugin features (preparing for their availability in Sigil v0.9.3)
  
• If using the bundled Python Interpreter (or if cssutils is installed), the css in a mobi7 import will be "pretty-printed."
  
• The addition of three configurable preference variables to fine-tune behavior.

[DiapDealer]

Updated to v0.80.4 in the first post.

[DiapDealer]

Well so much for that update!

Reverting back to v0.80.3 until I work out a module dependency issue.

Sorry for any confusion.

[DiapDealer]

New version 0.80.5 in the the first post.
(replaces faulty v0.80.4)

[dhdurgee]

I am seeing a problem importing an azw3 book, only part of the table of contents is being imported. Are there limits to the tool? I don't see any errors that appear related in the plugin runner window:

Status: success

Entering KindleUnpack input plugin's run() routine
Path to Kindlebook /perm/home/dhdurgee/Downloads/Kindle/Startide_Rising_original.azw3
Palm DB type: BOOKMOBI, 343 sections.
Warning: Bad key, size, value combination detected in EXTH 406 16 0000000000000000
Unpacking a KF8 book...
Processing K8 section of book...
Mobi Version: 8
Codec: utf-8
Title: Startide Rising
EXTH Title: Startide Rising (Uplift Trilogy Book 2)
Huffdic compression
Unpacking images, resources, fonts, etc
Extracting font: font00323
Extracting font: font00324
Extracting font: font00325
Extracting font: font00326
Extracting image: image00327.jpeg from section 327
Extracting image: cover00328.jpeg from section 328
Extracting image: image00329.jpeg from section 329
Extracting image: image00330.jpeg from section 330
Extracting font: font00331
Extracting font: font00332
Extracting image: image00333.jpeg from section 333
Extracting image: image00334.jpeg from section 334
Extracting image: image00336.jpeg from section 336
Unpacking raw markup language
Processing ncx / toc
Building an epub-like structure
Building proper xhtml for each file
Building a cover page.
Building an opf for mobi8 using epub version: 2
Write K8 ncx
Creating an epub-like file
Path to epub or src /tmp/tmpvIbD8q/mobi8/Startide_Rising_original.epub

The imported table of contents contains seven initial entries, a part one entry and eleven sub-entries under part one. The last sub-entry is incomplete, lacking a target. The remainder of the table of contents is missing.

I can generate a new complete table of contents, but it lacks the structure of the imported one and duplicates entries as there are both numbers and titles listed for chapters.

Dave

[DiapDealer]

You'd probably have to try that books with the stand-alone version of KindleUnpack (or at the very-least, the KindleUnpack calibre plugin). If both of those produce the same result, there could be a bug in KindleUnpack or something weird with the book.

If you'd care to try the ScrambleEbook calibre plugin, you could scramble the contents of the the original AZW3 and then share it here without any copyright concerns.

Also important: were you presented with a message about the book containing errors upon import? If so, did you answer yes or no to the offer to fix them automatically?

And what are your Sigil General Settings in the preferences. The markup in Kindlebooks can be notoriously bad, so I can't recommend importing any Kindlebooks with the "Mend XHTML Source Code" turned off (nor declining any offers to fix problems).

EDIT: Judging by the messages, I believe I own the exact same book--same fonts and images anyway. The only thing different is the filenames. I get all of the ToC (HTML and NCX) when I import it. Are you using the very latest version of KindleImport? Or was the AZW3 modified/edited by another program before trying to import it?

[dhdurgee]

I am using the version that came with the KindleImport 0.8.2 download, inspecting the kindleunpack.py in the KindleImport plugin directory I see the last comment identifying it as 0.80 there.

The only modification prior to using the plugin was to decrypt the .azw3 file using the version 5.2 k4mobidedrm.py script.

Dave

[DiapDealer]

Originally Posted by dhdurgee

I am using the version that came with the KindleImport 0.8.2 download, inspecting the kindleunpack.py in the KindleImport plugin directory I see the last comment identifying it as 0.80 there.

The only modification prior to using the plugin was to decrypt the .azw3 file using the version 5.2 k4mobidedrm.py script.

Dave

There is a newer version of the KindleImport plugin (v0.80.5) in the first post of this thread that does include some changes/bug fixes to the underlying KindleUnpack engine. I'd try that first.

I'm not permitted to discuss drm decryption in any real detail, but I do know that there are several options out there that attempt to "clean" way too much and actually end up breaking some things. KindleUnpack has had trouble with the output of some of those in the past. Apprentice Alf seems to be the de facto standard for this sort of thing.

As I mentioned: the book unpacked/imported OK for me. And I doubt I'm up to date at all with Alf's tools, to tell the truth.

[dhdurgee]

I updated to the newer plugin and got the same results. Thinking it might yield different results, I downloaded the book from Amazon to my system for USB transfer. It was different alright, "Book is encrypted" message. I guess I will have to see if there is an update to the drm removal script. I will check for a newer version. The one I am using shows this in the top:

# ignobleepub.pyw, version 3.6
# Copyright © 2009-2012 by DiapDealer et al.

So perhaps I am behind the times on this. I might boot my W7 vbox and let Calibre take a shot at it and see if the results differ.

Dave

[dhdurgee]

I downloaded the latest version of the DeDRM_tools_6.3.6a and installed them in place of the release I was using. Unfortunately the results are still the same. Later I will see if anything works better with W7 Calibre doing the work. If not, I guess I will have to look into your Scrambler Plugin and let wiser heads look into this.

I am having no problem reading the book, but the cover page is the ugly Ballantine Books generic cover as opposed to the proper one. My main reason for using Sigil on it is to put the proper cover on it for use by the screen saver.

Dave

[dhdurgee]

Actually just had a thought, could I post the encrypted .azw3 file here? Without knowing the key information it would take a major effort to open it. I could then private message the key data to the maintainer who needed it for analysis.

Dave

[eschwartz]

If the tools do not work on that book, ask on Apprentice Alf's website, not here.

If the tools work, but the standalone version of KindleUnpack doesn't work, post on the KindleUnpack support thread, and wait for instructions.

This thread is only for issues with the pluginization itself. If the standalone KindleUnpack works, but the plugin doesn't, then it is time to post here.

[dhdurgee]

Originally Posted by eschwartz

If the tools do not work on that book, ask on Apprentice Alf's website, not here.

If the tools work, but the standalone version of KindleUnpack doesn't work, post on the KindleUnpack support thread, and wait for instructions.

This thread is only for issues with the pluginization itself. If the standalone KindleUnpack works, but the plugin doesn't, then it is time to post here.

The tools do appear to work, I am able to open the decrypted books. The problem is that when I use the KindleImport plugin the ToC is truncated in Sigil. I guess I will need to be sure I have the latest standalone KindleUnpack and see if the extracted ToC is correct or truncated first.

Dave

[DiapDealer]

Originally Posted by dhdurgee

Actually just had a thought, could I post the encrypted .azw3 file here? Without knowing the key information it would take a major effort to open it. I could then private message the key data to the maintainer who needed it for analysis.

That's a no-go, I'm afraid. Even though I'm certain the devs would act responsibly with such info, it doesn't change the fact that a key plus a copy of the entire book the key will work on could be construed as copyright violation. Some could argue debugging ebook software and then deleting might fall under fair use, but MobileRead isn't willing to test that theory.

Plus, as eschwartz mentioned, DRM removal is out of the scope of this particular thread. I purchased the exact same ebook from Amazon, and once the DRM was gone the KindleImport plugin worked without any data being lost. So there must be something else going on.

If you post a scrambled version of the drm-free azw3 (that shows a full working toc before and after scrambling), I will be happy to see if the toc gets truncated by KindleImport. Likewise, I can share with you, the scrambled version of the azw3 that works for me so you can test on your end.

Other than that, I'm afraid there's not much we can do.