[Plugin] KindleImport Sigil plugin - Page 9

dhdurgee · 02-03-2016, 03:57 PM

I just ran the standalone KindleUnpack on the DeDRM'ed .azw3 and the extracted toc.ncx is NOT truncated. I just ran the KindleImport plugin again on the same file to confirm I was still getting the truncated one with the updated plugin and it is.

Here is the end of the truncated toc.ncx from Sigil:

Quote:

<navLabel>
<text>Chapter 10 - Metz</text>
</navLabel>
<content src="Text/part0019.xhtml"/>
</navPoint>
<navPoint id="np_19" playOrder="19">
<navLabel>
<text>Chapter 11 - Creideiki </text></navLabel></navPoint></navPoint></navMap></ncx>

Here is the same section from the KindleUnpack extract:

Quote:

<navLabel>
<text>Chapter 10 - Metz</text>
</navLabel>
<content src="Text/part0019.xhtml"/>
</navPoint>
<navPoint id="np_19" playOrder="19">
<navLabel>
<text>Chapter 11 - Creideiki & Orley</text>
</navLabel>
<content src="Text/part0020.xhtml"/>
</navPoint>
<navPoint id="np_20" playOrder="20">
<navLabel>
<text>Chapter 12 - Galactics</text>
</navLabel>
<content src="Text/part0021.xhtml"/>
</navPoint>
<navPoint id="np_21" playOrder="21">

From what I see here it seems perhaps the "&" is causing a problem with the plugin?

Let me know if you need more information.

Dave

HOLD ON! It may not be a plugin problem! It may be a Sigil problem! I noticed that KindleUnpack created a .epub file in the mobi8 directory. Looking at the I found that it had the non-truncated toc.ncx file. Given this I thought to open it with Sigil and found that I WAS STILL SEEING THE TRUNCATED TOC! This is Sigil 0.8.6 on Linux Mint x64 here.

Where do we go from here?

Dave

DiapDealer · 02-03-2016, 04:32 PM

It's a bad book, for one thing. You can't have ampersands in a toc.ncx file. It needs to be an entity. But with recent versions of Sigil, the ampersand just get tossed out and things move on. That's why I wasn't seeing the truncation you were. I had no idea you were still using an old 0.8.x version of Sigil. Is it one of my deb packages or did you compile it yourself?

Either way, you're right; this isn't a plugin issue, per se. I'll see if we want to make KindleUnpack accountable for heading off ampersands in ncx files in the future.

I might be able to cobble together a work-around for this in KindleImport, for the time being, though. I'll look into it.

Thanks for the persistence!

EDIT: actually it won't be as easy to kludge together a workaround for this in the KindleImport plugin as I thought. Might have to see about bouncing this upstream to KindleUnpack.

dhdurgee · 02-03-2016, 04:40 PM

I am using your latest available .deb here. Am I to assume then that this is addressed in the 0.9.x version? Will you be able to make a .deb available for the newer release?

Dave

DiapDealer · 02-03-2016, 05:08 PM

Quote:

Originally Posted by dhdurgee

I am using your latest available .deb here. Am I to assume then that this is addressed in the 0.9.x version? Will you be able to make a .deb available for the newer release?

Dave

Not really addressed, no. But it would no longer truncate everything after an error like that in the ncx. It just seems to toss the '&' altogether. Sigil's never been particularly great at handling things that aren't supposed to be in the opf/ncx, but it's getting better, I think, at not dying when it does encounter errors like that.

As for a 0.9.x .deb. That may not be in the cards. But I still hold out hope that I'll be able to provide a standalone Linux installer of some kind in the near(ish) future.

In the meantime, see if this attached test version of the KindleImport plugin will allow you to import that azw3 with the full ncx intact. Fix now incorporated in v0.80.6 in first post

dhdurgee · 02-03-2016, 08:04 PM

That did the trick, even the "&" in the ToC is still there. The F7 check still shows some problems, but hopefully I can deal with them.

Dave

PS: I just thought of something else, fixing this plugin does not deal with Sigil opening a .epub that has such an erroneous toc.ncx in it. So now I can import the .azw3 with such a toc.ncx, but after I save it as a .epub I will be unable to edit it further as the toc.ncx will be truncated when I reload it.

DiapDealer · 02-03-2016, 08:14 PM

Not surprising. Many Kindle books don't result in 100% valid epubs when unpacked (it all depends on how valid the epub was before it was fed to Kindlegen). Glad it worked for you. I'll see about updating the first post with this version of the plugin, but I think I'm going to see about addressing this in KindleUnpack.

dhdurgee · 02-04-2016, 09:25 AM

Thanks for your assistance in this matter. Now that I know what the problem is I was able to edit the ToC and replace the "&" with "&" such that Sigil no longer has a problem loading the ToC from a saved .epub document. Should I worry about entity codes for any other characters? There are also single and double quotes in the ToC, which could be replaced if they can also cause problems.

Dave

DiapDealer · 02-04-2016, 09:58 AM

The & character is the only one I'd worry about in an ncx <text></text> element.

dhdurgee · 02-04-2016, 06:55 PM

This must be my time to hit books with problems. I used KindleImport on another book and found that the ToC looked too small. So I did a KindleUnpack on it to see what it looked like there.

My surprise was that I found there were many more Text parts unpacked than were loaded by the KindleImport plugin. I am not 100% sure of why, but it seems suspicious to me that the ones the plugin loaded were only the ones in the unpacked content.opf, which is missing a LOT of the Text parts!

So what needs to be done to fix this? Perhaps I can start with the .epub generated by KindleUnpack instead?

Dave

DiapDealer · 02-04-2016, 07:18 PM

I'm not following. You're saying the OPF file of an unpacked book doesn't have all the Text files in its manifest?

I can guarantee that Sigil won't load any resources that aren't properly manifested in the OPF, but the question is, why would any text files exist that weren't manifested?

Keep in mind that there is no guarantee that the import of old-fashioned mobi files will be successful. Are we talking about an old mobi here or a KF8 Kindlebook? The former is always going to be a crap shoot when trying to import, but that latter should definitely not have data missing.

KindleImport is producing the exact same thing as KindleUnpack (it's actually mostly the same under the hood, in fact). But Sigil is particular about things. So like with the other book, starting with the unpacked epub is unlikely to produce a different result (when opened in Sigil) than using KindleImport will.

The real problem is that non-epub things can still be used to make Kindlebooks. So Unpacking them doesn't always create spec-compliant epubs. If you find yourself running into a lot of these problem books, you may find that unpacking them and importing the html/css files may be the way to go.

dhdurgee · 02-04-2016, 09:18 PM

I ran KindleUnpack on the .azw3 file:

Quote:

dhdurgee@DHD-Z560 ~/Downloads/Kindle $ ~/bin/KindleUnpack_v075/lib/kindleunpack.py Infinity-s_Shore.azw3 /var/tmp/kindle
KindleUnpack v0.75
Based on initial mobipocket version Copyright © 2009 Charles M. Hannum <root@ihack.net>
Extensive Extensions and Improvements Copyright © 2009-2014
by: P. Durrant, K. Hendricks, S. Siebert, fandrieu, DiapDealer, nickredding, tkeo.
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, version 3.
Unpacking Book...
Palm DB type: BOOKMOBI, 437 sections.
Warning: Bad key, size, value combination detected in EXTH 406 16 0000000000000000
Unpacking a KF8 book...
Processing K8 section of book...
Mobi Version: 8
Codec: utf-8
Title: Infinity’s Shore
EXTH Title: Infinity's Shore (Uplift Trilogy Book 2)
Huffdic compression
Unpacking images, resources, fonts, etc
Extracting font: font00417
Extracting font: font00418
Extracting font: font00419
Extracting font: font00420
Extracting image: image00421.jpeg from section 421
Extracting image: cover00422.jpeg from section 422
Extracting image: image00423.jpeg from section 423
Extracting image: image00424.jpeg from section 424
Extracting image: image00425.jpeg from section 425
Extracting image: image00426.jpeg from section 426
Extracting image: image00427.jpeg from section 427
Extracting image: image00428.jpeg from section 428
Extracting image: image00430.jpeg from section 430
Unpacking raw markup language
Processing ncx / toc
Building an epub-like structure
Building proper xhtml for each file
Building a cover page.
Warning: cover_page.xhtml already exists.
Building an opf for mobi8 using epub version: 2
Write K8 ncx
Creating an epub-like file
Completed
dhdurgee@DHD-Z560 ~/Downloads/Kindle $

The .epub file created contains 149 files in the OEBPS/Text directory, but the manifest in the content.opf does not list many of them:

Quote:

I assume that the content.opf was generated by KindleUnpack, if so is this a bug?

Dave

DiapDealer · 02-05-2016, 07:29 AM

Let me take a look. I may split this off to the KindleUnpack thread if it looks like that's where the problem lies. I'll let you know.

DiapDealer · 02-05-2016, 08:08 AM

Quote:

Originally Posted by dhdurgee

The .epub file created contains 149 files in the OEBPS/Text directory, but the manifest in the content.opf does not list many of them:

...

I assume that the content.opf was generated by KindleUnpack, if so is this a bug?

Dave

Are you sure you're not unpacking this book into a folder that already contains remnants of a previously unpacked (and much larger) book?

I purchased this book from Amazon and when I unpack it using the standalone KindleUnpack, I only see 46 files in the resulting mobi8/OEBPS/Text folder (cover_page.xhtml and part0000.xhtml - part0044.xhtml)--all of which are properly manifested in the opf file.

The resulting epub in the mobi8 folder also only has 46 files in the OEBPS/Text folder (all manifested). When using KindleImport/Sigil, I get the exact same result as I do with KindleUnpack (I've even made sure to use an older version of KindleUnpack to make sure my previous patch wasn't affecting things).

However, when I unpack it into a folder containing the contents of a previously unpacked larger book. I see the same thing you do: extra, unmanifested files in the mobi8/OEBPS/Text folder and in the resulting epub. You should always unpack to an empty directory.

I see no reason to suspect that KindleUnpack or KindleImport or Sigil has truncated any data or left any files out of this book.

dhdurgee · 02-05-2016, 08:58 AM

Feel free to delete the past few posts if you like. You are absolutely correct, I had not cleared the working folder and it appears that the files from a previously unpacked book were mixed in. I was unaware that KindleUnpack expected a "clean" working area, not having made much use of it.

Sorry for the false alarm.

Dave

DiapDealer · 02-05-2016, 09:09 AM

Quote:

Originally Posted by dhdurgee

Feel free to delete the past few posts if you like. You are absolutely correct, I had not cleared the working folder and it appears that the files from a previously unpacked book were mixed in. I was unaware that KindleUnpack expected a "clean" working area, not having made much use of it.

Sorry for the false alarm.

Dave

No problem. But I'll leave the posts if you don't mind. It may help someone else who notices discrepancies between a KindleImported azw3 and a KindleUnpacked azw3. I don't think it's explicitly stated anywhere that books should be unpacked to an empty directory, it's just that KindleUnpack isn't going to be responsible for deleting important files if someone should happen to unpack into the wrong folder or something. And it always creates the same folder structure, so it's not always readily apparent that something was already there.

Don't feel bad about making mistakes. It's not as if this stuff comes with a current manual or anything.

02-03-2016, 04:32 PM	#122
DiapDealer Grand Sorcerer Posts: 28,635 Karma: 204624552 Join Date: Jan 2010 Device: Nexus 7, Kindle Fire HD	It's a bad book, for one thing. You can't have ampersands in a toc.ncx file. It needs to be an entity. But with recent versions of Sigil, the ampersand just get tossed out and things move on. That's why I wasn't seeing the truncation you were. I had no idea you were still using an old 0.8.x version of Sigil. Is it one of my deb packages or did you compile it yourself? Either way, you're right; this isn't a plugin issue, per se. I'll see if we want to make KindleUnpack accountable for heading off ampersands in ncx files in the future. I might be able to cobble together a work-around for this in KindleImport, for the time being, though. I'll look into it. Thanks for the persistence! EDIT: actually it won't be as easy to kludge together a workaround for this in the KindleImport plugin as I thought. Might have to see about bouncing this upstream to KindleUnpack. Last edited by DiapDealer; 02-03-2016 at 04:37 PM.

02-03-2016, 08:04 PM	#125
dhdurgee Guru Posts: 910 Karma: 3000002 Join Date: Jun 2010 Device: K3W, PW4	That did the trick, even the "&" in the ToC is still there. The F7 check still shows some problems, but hopefully I can deal with them. Dave PS: I just thought of something else, fixing this plugin does not deal with Sigil opening a .epub that has such an erroneous toc.ncx in it. So now I can import the .azw3 with such a toc.ncx, but after I save it as a .epub I will be unable to edit it further as the toc.ncx will be truncated when I reload it. Last edited by dhdurgee; 02-03-2016 at 08:17 PM. Reason: further thoughts required a PS

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[Plugin] PunctuationSmarten Sigil plugin	DiapDealer	Plugins	143	07-30-2025 06:58 AM
[Plugin] ePub3-itizer - epub3 output plugin for Sigil	KevinH	Plugins	457	05-28-2025 06:14 PM
epubcheck plugin for Sigil	Doitsu	Plugins	539	04-19-2025 08:45 AM
kindlegen plugin for Sigil	Doitsu	Plugins	173	10-15-2024 02:51 AM
smoothRemove_v010 plugin for Sigil	kbanelas	Plugins	15	01-27-2017 05:51 PM

02-03-2016, 04:40 PM	#123
dhdurgee Guru Posts: 910 Karma: 3000002 Join Date: Jun 2010 Device: K3W, PW4	I am using your latest available .deb here. Am I to assume then that this is addressed in the 0.9.x version? Will you be able to make a .deb available for the newer release? Dave

02-03-2016, 08:14 PM	#126
DiapDealer Grand Sorcerer Posts: 28,635 Karma: 204624552 Join Date: Jan 2010 Device: Nexus 7, Kindle Fire HD	Not surprising. Many Kindle books don't result in 100% valid epubs when unpacked (it all depends on how valid the epub was before it was fed to Kindlegen). Glad it worked for you. I'll see about updating the first post with this version of the plugin, but I think I'm going to see about addressing this in KindleUnpack.

02-04-2016, 09:25 AM	#127
dhdurgee Guru Posts: 910 Karma: 3000002 Join Date: Jun 2010 Device: K3W, PW4	Thanks for your assistance in this matter. Now that I know what the problem is I was able to edit the ToC and replace the "&" with "&" such that Sigil no longer has a problem loading the ToC from a saved .epub document. Should I worry about entity codes for any other characters? There are also single and double quotes in the ToC, which could be replaced if they can also cause problems. Dave

02-04-2016, 09:58 AM	#128
DiapDealer Grand Sorcerer Posts: 28,635 Karma: 204624552 Join Date: Jan 2010 Device: Nexus 7, Kindle Fire HD	The & character is the only one I'd worry about in an ncx <text></text> element.

02-04-2016, 06:55 PM	#129
dhdurgee Guru Posts: 910 Karma: 3000002 Join Date: Jun 2010 Device: K3W, PW4	This must be my time to hit books with problems. I used KindleImport on another book and found that the ToC looked too small. So I did a KindleUnpack on it to see what it looked like there. My surprise was that I found there were many more Text parts unpacked than were loaded by the KindleImport plugin. I am not 100% sure of why, but it seems suspicious to me that the ones the plugin loaded were only the ones in the unpacked content.opf, which is missing a LOT of the Text parts! So what needs to be done to fix this? Perhaps I can start with the .epub generated by KindleUnpack instead? Dave

02-04-2016, 07:18 PM	#130
DiapDealer Grand Sorcerer Posts: 28,635 Karma: 204624552 Join Date: Jan 2010 Device: Nexus 7, Kindle Fire HD	I'm not following. You're saying the OPF file of an unpacked book doesn't have all the Text files in its manifest? I can guarantee that Sigil won't load any resources that aren't properly manifested in the OPF, but the question is, why would any text files exist that weren't manifested? Keep in mind that there is no guarantee that the import of old-fashioned mobi files will be successful. Are we talking about an old mobi here or a KF8 Kindlebook? The former is always going to be a crap shoot when trying to import, but that latter should definitely not have data missing. KindleImport is producing the exact same thing as KindleUnpack (it's actually mostly the same under the hood, in fact). But Sigil is particular about things. So like with the other book, starting with the unpacked epub is unlikely to produce a different result (when opened in Sigil) than using KindleImport will. The real problem is that non-epub things can still be used to make Kindlebooks. So Unpacking them doesn't always create spec-compliant epubs. If you find yourself running into a lot of these problem books, you may find that unpacking them and importing the html/css files may be the way to go.

02-05-2016, 07:29 AM	#132
DiapDealer Grand Sorcerer Posts: 28,635 Karma: 204624552 Join Date: Jan 2010 Device: Nexus 7, Kindle Fire HD	Let me take a look. I may split this off to the KindleUnpack thread if it looks like that's where the problem lies. I'll let you know.

02-05-2016, 08:58 AM	#134
dhdurgee Guru Posts: 910 Karma: 3000002 Join Date: Jun 2010 Device: K3W, PW4	Feel free to delete the past few posts if you like. You are absolutely correct, I had not cleared the working folder and it appears that the files from a previously unpacked book were mixed in. I was unaware that KindleUnpack expected a "clean" working area, not having made much use of it. Sorry for the false alarm. Dave

Advert

Advert