KindleUnpack (MobiUnpack): Extracts text, images and metadata from Kindle/Mobi files

[DiapDealer]

I'm still seeing a slight issue with AZW4/Print Replica in v0.72.

Line 454 (in processPrintReplica) of kindleunpack.py attempts to initiate the class OPFProcessor (mobi_opf.py) with an insufficient number of parameters. Should not OPFProcessor's 'pagemapxml' parameter be made optional?

From (mobi_opf.py line 11):

def __init__(self, files, metadata, filenames, imgnames, isNCX, mh, usedmap, pagemapxml, guidetext=False, k8resc=None):

to:

def __init__(self, files, metadata, filenames, imgnames, isNCX, mh, usedmap, pagemapxml='', guidetext=False, k8resc=None):

Other than that, I've had great luck plopping the refactored code into my plugin's framework.

I just have to figure out what to do about letting users select an apnx file from the plugin. I'm leaning toward not worrying about it at all. There's no way to store an apnx file WITH the book in calibre anyway, so I'm thinking of just leaving that as a feature they'll need to use the standalone version of KindleUnpack for. We'll see.

[KevinH]

Hi DiapDealer,

Yes, most definitely. I made pagemapxml a null string in Mobi7 but this approach is better.

I will make the change and release a 0.72a immediately. Hopefully the last bug!

Edited: removed KindleUnpack_v072.zip and changed it to KindleUnpack_v072a.zip in previous post


Yes, I would just leave out the apnx feature in the plugin. The apnx files generated by calibre even with the recent changes are mainly fake offsets which would just confuse the hell out of things for KindleUnpack. Also many of the apnx files in Kindle ebooks are just plain fake offsets as well. So yes, I would leave that for more advanced standalone use.

Also, one question: Is there any real limit on image size or resolution in epubs? In other words, can we simply unpack the HDImages and replace the smaller kindlegen generated images on the fly. That will give the "best" image for later use in any conversion.

If so, users can convert with an epub to epub conversion in Calibre if they need to downsample the images for their particular epub device.

Thanks,

KevinH

Originally Posted by DiapDealer

I'm still seeing a slight issue with AZW4/Print Replica in v0.72.

Line 454 (in processPrintReplica) of kindleunpack.py attempts to initiate the class OPFProcessor (mobi_opf.py) with an insufficient number of parameters. Should not OPFProcessor's 'pagemapxml' parameter be made optional?

From (mobi_opf.py line 11):

def __init__(self, files, metadata, filenames, imgnames, isNCX, mh, usedmap, pagemapxml, guidetext=False, k8resc=None):

to:

def __init__(self, files, metadata, filenames, imgnames, isNCX, mh, usedmap, pagemapxml='', guidetext=False, k8resc=None):

Other than that, I've had great luck plopping the refactored code into my plugin's framework.

I just have to figure out what to do about letting users select an apnx file from the plugin. I'm leaning toward not worrying about it at all. There's no way to store an apnx file WITH the book in calibre anyway, so I'm thinking of just leaving that as a feature they'll need to use the standalone version of KindleUnpack for. We'll see.

[DiapDealer]

Originally Posted by JSWolf

But do we really need or even want ePub 3 support given that most devices we use don't support ePub 3. Will it still generate ePub 2 as well as ePub 3 or would be be stuck with ePub 3 only?

I can understand the concern, but it's not really a question of "wanting" epub 3 support. (Mobi|Kindle)Unpack's goal has always been to try and recreate the original source material used to create the kindlebook as accurately as possible. The fact is ...kindlegen now accepts ePub 3 (and html5 and css3) as input for building kindlebooks. How silly (or presumptuous) would it be for KindleUnpack to attempt to force ePub 3 content into an ePub 2 box (without giving the user a choice to do so anyway)?

[DiapDealer]

Originally Posted by KevinH

Yes, I would just leave out the apnx feature in the plugin. The apnx files generated by calibre even with the recent changes are mainly fake offsets which would just confuse the hell out of things for KindleUnpack. Also many of the apnx files in Kindle ebooks are just plain fake offsets as well. So yes, I would leave that for more advanced standalone use.

My thoughts, as well. Besides... I believe calibre's apnx files are actually created "on the fly" when sending books to a device.

Also, one question: Is there any real limit on image size or resolution in epubs? In other words, can we simply unpack the HDImages and replace the smaller kindlegen generated images on the fly. That will give the "best" image for later use in any conversion.


If so, users can convert with an epub to epub conversion in Calibre if they need to downsample the images for their particular epub device.

I know of no "official" ePub image size limit. I do know of devices/apps that have limits. Any chance it could be (hopefully somewhat easily) made a user-choice?

"Use HD images whenever present?"

I've come to realize that many use the KindleUnpack plugin precisely because they want to avoid having to do a calibre epub to epub conversion.

[JSWolf]

Originally Posted by DiapDealer

I just have to figure out what to do about letting users select an apnx file from the plugin. I'm leaning toward not worrying about it at all. There's no way to store an apnx file WITH the book in calibre anyway, so I'm thinking of just leaving that as a feature they'll need to use the standalone version of KindleUnpack for. We'll see.

What I would like to have is the number of pages pulled out of the apnx so the plugin can populate the page number field for Calibre's apnx generator to use. May not match up perfectly, but at least it will give the same number of total pages and to me, that's good enough.

[DiapDealer]

Originally Posted by JSWolf

What I would like to have is the number of pages pulled out of the apnx so the plugin can populate the page number field for Calibre's apnx generator to use. May not match up perfectly, but at least it will give the same number of total pages and to me, that's good enough.

Pulled out of what APNX file? The plugin only operates on whichever book is highlighted in the library. And while I understand that calibre CAN generate an apnx file based on a user-defined page-count field and send it to devices along with the Kindlebook, I don't see that as falling within KindleUnpack's purview.

Regardless, I don't think I'm interested in populating user-defined calibre fields from externally selected and parsed apnx files.

[Doitsu]

Originally Posted by DiapDealer

And while I understand that calibre CAN generate an apnx file based on a user-defined page-count field and send it to devices along with the Kindlebook, I don't see that as falling within KindleUnpack's purview.

It certainly doesn't fall within KindleUnpack's purview, but it would be a nice-to-have feature to be able to generate an .apnx file with "real page numbers" from a KindleGen generated .mobi file with an embedded pagemap or pagelist since KindleGen wont generate one, and, AFAIK, Amazon won't generate .apnx files for KDP published books with pagemaps or pagelists either.

[tkeo]

Hi Kevin,

If you don't like storing them in the mobi7 folder first then, I guess I don't understand why we can't simply write the files to a neutral location as we read them. Perhaps a base Images/ and HDImages/ and then in processMobiX put them in the proper location?

Also for the mobi 8 we want to create both an epub file and leave it unpacked in place so that users can see what is there more easily.

I prefer this approch.

Originally Posted by KevinH

Hi tkeo,
I looked at your patch from v067 to add epub3 support and your comments.

Firstly, the posted code is written with thinking, so it may be redundant.

- why do you want to move writing of files back to process_all_mobi_headers?
In general, an object should know how and where it should write itself. Therefore, NAV, OPF, NCX etc should all know where and how to create themselves from the passed in data and the "files" object.

In fact, my inclination is to move the writing of the text/html files out of even processMobiX and out to header specific routines.

I feel easier to understand what are extracted and built when calling of write() are placed in single function( or in single file). In addition, I prefer to avoid reading from written file during processing.

Another, I am considering to make an option to converting to epub only and output no other files. Holding datalist and partlist might be helpful to do so, but not necessary if there are other ways.

- you seem to duplicate information from k8proc to pull into k8resc. It would simply be easier to pass in the k8proc object is and where you need that information.

I will modify it; however, I am not fully understand your intention. Does it mean adding a parameter to K8RESCProcessor?

- your datalist[] simply duplicates much of what imgnames is used for and you never store any raw data in that list anyway. Why is the data offset information there if the data itself is never stored? Why do you need the section number? The directory and type information can easily be deduced from the file name extension, and metadata for (offsets).

As you mentioned, storing actual data makes memory usage lager; therefore secno and offset are stored to extract from sect, except fonts.

- why do we need to know the width and height of the images? Is this needed to create svg based cover pages?

Yes, it is needed to create the svg based cover page.

- Also do we really need to rename the cover image as "cover"?

It makes to find the cover image at a glance when looking in Image folder; but not crucial.

- Also Can't we simply use the EmptyImagePlaceholders and the kindleembed string from the "kind" section and info from CONT metadata to overwrite the the corresponding non-HD image file with the correct HDImage but keep its original name so nothing in the OPF needs to care.

The CONT and following sections up to the container boundary is a simple one-to-one mapping from the first image to the last HDImage with empty place holders being used to indicate images that do not have HD replacements (so they only need to keep track of things up to the last HDImage.

The number in image names are correspond to section numbers currently, so keeping the numbers might help to know where the image is come from; however, personally, I am not oppose to renaming (or write out) HD images to correspond to low resolution images.
It makes easer to switch the images in epub. I prefer to add an option to switch it by user.


For conclusion, I am not stick to the code I have written, but please consider and give feedback to me about adding an option to convert to epub only though I do not plan to this immediately.
And I would like to hear considerations from others too.


Thanks,
tkeo

[KevinH]

Originally Posted by tkeo

Hi Kevin,
I feel easier to understand what are extracted and built when calling of write() are placed in single function( or in single file). In addition, I prefer to avoid reading from written file during processing.

No, that just makes the code harder to modify as everything must be collected and then written out in one code block. Let each object decide how to write itself out, you just pass in the data it needs to build itself properly, and where you want it to go. This encapsulates the all knowledge of the object in the objects own code.

Another, I am considering to make an option to converting to epub only and output no other files. Holding datalist and partlist might be helpful to do so, but not necessary if there are other ways.

We have no guarantees a valid epub2 or epub3 was even passed in. If you give kindlegen a valid old mobi it will create a horrible mobi8 version that looks like one giant file. So the output of KindleUnpack is guaranteed to be only epub-like. Serious editing might be needed. Having the actual epub structure fully unpacked can allow people to easily edit and change things and pass the resulting content.opf right back into kindlegen until they get the results they want. So I don't think we need an option to only create an epub. It is easy enough to extract it fom the output.

I will modify it; however, I am not fully understand your intention. Does it mean adding a parameter to K8RESCProcessor?

Yes, pass in the k8proc object when you create the k8resc object and allow it to store it away and then move most of the processing you are doing in the main KindleUnpack routine back into mobi_k8resc.py to better encapsulate it and prevent the needless copying of data structures to create partslist[].

As you mentioned, storing actual data makes memory usage larger; therefore secno and offset are stored to extract from sect, except fonts.

But everything has already been extracted to that point. You seem to want to run a two path algorithm where the first pass finds out what every section is for and where it should go, and then a second pass to do the extraction itself.

Why make two passes when one will do. Simply intelligently walk the sections once while writing images and fonts out as you go along (they will all be needed so no wasted space) and collecting any data that needs to be collected for later processing of ncx, nav, opf, xhtml,etc
[/QUOTE]

Yes, it is needed to create the svg based cover page.

I found that exact extract image height and width routine on the internet but did not see license information associated with it. Are you sure we have permission to incorporate it into our code?

The number in image names are correspond to section numbers currently, so keeping the numbers might help to know where the image is come from; however, personally, I am not oppose to renaming (or write out) HD images to correspond to low resolution images.
It makes easer to switch the images in epub. I prefer to add an option to switch it by user.

So we add an option to use HDImage if available and if checked we walk the CONT sections in order and replace the corresponding image file in the Images folder. No need for changes in OPF or anything else.

Also, please remove all of the flags you pass in for epub version support. It just means passing around flag bits that no one else will follow. The Tk GUI calls unpackBook directly, and should not need to build up flags. Simply pass in one character based option --epub_version= and use 2 or 3 or A as desired, default to 2 if nothing is passed in but please do not make a flags field, it just makes the code unreadable.

For conclusion, I am not stick to the code I have written, but please consider and give feedback to me about adding an option to convert to epub only though I do not plan to this immediately.
And I would like to hear considerations from others too.

I am not in favour of that. This should be a general unpack tool and should help people see and understand all of the new sections and headers as Amazon changes them. And Amazon has kept changing them and will probably continue to change them to add new features and section types.

Also, it is not a general epub creator as we can only guarantee an epub-like structure and not a valid epub2 or even 3 depending on just what was passed into kindlegen in the first place.

Take care,

KevinH

[KevinH]

Hi,
Given how very close APNX files and PAGE sections are, it should be easy to create a very short program to extract and covert a Mobi 8 PAGE section (which kindlegen generated from the input epub pageMap) into a standalone APNX file.

It remains to be seen if this would work or not but it could be as simple as dumping the PAGE section after removing the first 8 bytes.

When I get a free moment I will look into this to give you something to test with.

At the very least it can easily provide page names and offsets to code that already knows how to generate an APNX. But unfortunately, none of the current APNX generating code groks the use of multiple page naming schemes used in the same document as our unpack code does.


KevinH

Originally Posted by Doitsu

It certainly doesn't fall within KindleUnpack's purview, but it would be a nice-to-have feature to be able to generate an .apnx file with "real page numbers" from a KindleGen generated .mobi file with an embedded pagemap or pagelist since KindleGen wont generate one, and, AFAIK, Amazon won't generate .apnx files for KDP published books with pagemaps or pagelists either.

[tkeo]

Hi Kevin,

Thank you for replying to my dim opinions. I understand why converting only to epub is not acceptable.
I will modify the code based on your suggestions.

I found that exact extract image height and width routine on the internet but did not see license information associated with it. Are you sure we have permission to incorporate it into our code?

Because written 'from draco' in the function, I simply thought it was pulled out from the draco package whose license is the GPL. However, I am not sure it is truly from the draco, so I will not include the function and the cover page creation using svg.

Thanks,

[KevinH]

Hi tkeo,

Originally Posted by tkeo

Hi Kevin,
Thank you for replying to my dim opinions. I understand why converting only to epub is not acceptable.
I will modify the code based on your suggestions.

They were fine opinions, they just differed in approach and philosophy. If you were going to only write out an epub, then a two-pass approach where everything is written out in the main routine and only once, would be an excellent approach. We just can't be sure what we create would be a true epub, and so we use a different approach and philosophy.

Because written 'from draco' in the function, I simply thought it was pulled out from the draco package whose license is the GPL. However, I am not sure it is truly from the draco, so I will not include the function and the cover page creation using svg.

I don't know what "Draco" even is but if we can search and find a repository on the web that shows is is GPL or BSD, then all is fine. So don't remove it yet. Give us a chance to look and find the Draco repository and license to verify all is okay first.

If you want, I would be happy to handle adding the command line switch to use HDImages if available and handle overwriting the Image files into the proper place and name in the mobi8 tree.

That way you can focus on the epub3 changes.

Take care and thanks for all of your hard work!!

KevinH

[tkeo]

Hi All,

Here is the preview version of epub3 supported KindleUnpack, v0.72b.

Most RESC processing in kindleunpack.py is offloaded to mobi_resc.py (still remained a little bit needed for cover creation).
Although I have tested, it might have bugs since I have modified a large amount of code.

This version has three building manifest functions (for mobi7/azw4, for epub2 and for epub3) in order to make further support of epub3 less affect others; however, I think it is possible to use an unified function.

Thanks,

[KevinH]

Hi tkeo,

Nicely done! Thank you!

Once we get enough "its working" or by lack of "bug reports" please release it here as an official KindleUnpack_v073 version!

Take care,

Kevin

Originally Posted by tkeo

Hi All,

Here is the preview version of epub3 supported KindleUnpack, v0.72b.

Most RESC processing in kindleunpack.py is offloaded to mobi_resc.py (still remained a little bit needed for cover creation).
Although I have tested, it might have bugs since I have modified a large amount of code.

This version has three building manifest functions (for mobi7/azw4, for epub2 and for epub3) in order to make further support of epub3 less affect others; however, I think it is possible to use an unified function.

Thanks,

[KevinH]

Hi All,

I have a new experimental version of KindleUnpack that has all of tkeo's latest changes for epub3 support and that will generate an apnx file if the mobi passed in has a PAGE section.

So if have an epub that has either a pagelist in the ncx or a page-map.xml, and you pass it into kindlegen, kindlegen will create a mobi with a PAGE section.

If you then pass that mobi into KindleUnpack, and split it, you should now find a:
- mobi7-BOOKFILENAME.mobi
- mobi7-BOOKFILENAME.apnx
- mobi8-BOOKFILENAME.azw3
- mobi8-BOOKFILENAME.apnx

all in the unpacked output directory.

I don't have access to an actual Kindle, but I am hoping if you side load either version and its associated apnx, you should see actual page numbers that exactly match what what was provided in the original epub.

So if anyone has an actual Kindle they can test with and an epub with real page numbers in either the pagelist in the ncx or a separate page-map.xml file, I would love to have help debugging the apnx generation and getting it to actually work.

Please let me know if you are willing to help and I will upload the experimental version of KindleUnpack for you to try.

Thanks,

KevinH


PS: FWIW, it works perfectly with my Kindle.app on my Mac OS X machine. Page numbers show up just fine when I move the .azw3 and its .apnx into "My Kindle Content" folder.

So I would love to know if this works for standalone Kindles and other Kindle apps.