KindleUnpack (MobiUnpack): Extracts text, images and metadata from Kindle/Mobi files

[DiapDealer]

Originally Posted by JSWolf

What is the deal with posting a full version of 0.72h? It gets posted, I test, if things fail, I post about it and it gets fixed. Simple really. But as it is, I get blocked at all turns about testing the latest version.

Because you already HAVE the latest version. The proposed changes are just that: proposed changes. But whatever. I'm done with you for a bit. Enjoy your super-secret-agent bug.

[JSWolf]

Originally Posted by DiapDealer

Because you already HAVE the latest version. The proposed changes are just that: proposed changes. But whatever. I'm done with you for a bit. Enjoy your super-secret-agent bug.

OK, I just tested things again but this time on the laptop and KindleUnpack 0.72h ran and the AZW3 unpacked without an error. If it's still wanted, I can rerun using 0.72f.

[tkeo]

Hi,

These are the patches to apply PEP recommendation.
All

!= None

are replaced to

is not None

and all

== None

are replaced to

is None

Thanks,

EDIT attached files are removed.

[JSWolf]

Seems KindleUnpack 0.72h didn't like Python 2.6. So I removed it, installed 2.7 along PyCrypto 2.6 for Python 2.7 and things now work on my desktop computer.

[KevinH]

Hi tkeo,

Please hold off on both your g and h versions for now. Let's focus on the resc code changes and then work from there.

FWIW, I am a firm believer in KISS engineering, especially for software. So I always focus on minimalist code and simplicity. So I want to change and simplify greatly the K8RESCProcessor code and remove taglist and make some other changes. The idea was to design something that could keep sequence/order as it parsed the resc data as well as something that only kept the actual information we needed and not try to keep the entire spine.

I have posted the equivalent parseRESC.py program (see attachment) to play with. Just pass it in any RESC file. It handles your test case just fine.

If you look at the main routine you will see how I would like to interface with the resc code. There is a spine_order list that keeps the keys (mainly skeleton file numbers) as strings, and these skelids are also used as keys into spine_idrefs, and etc. The extra metadata is extracted and ready to process as well.

We can change the opf generation code to add in all extra EXTH metadata (ASIN, etc,) inside a commented out section which will be ignored when parsed in the RESC code if you pass the file back through kindlegen after unpacking.

Once we agree on that or similar approach to parsing the resc, we can go back to figuring out the best way to simplify the opf generation code without off-loading code back to kindleunpack.py and to make it closer to what it was originally without all of the setid, regular expressions, taglist code, and all the associated overhead.

So please grab the latest parseRESC.py.zip and look it over and see if it does what we want in a minimal fashion. Please test it with your harder RESCXXXXX.dat files. Please let me know if you think it is missing some key information. Once we agree on how the resc code should work, we can move forward with reworking the opf generation code to take advantage of these changes.

Thanks,

KevinH

[KevinH]

JSWolf,

Originally Posted by JSWolf

Seems KindleUnpack 0.72h didn't like Python 2.6. So I removed it, installed 2.7 along PyCrypto 2.6 for Python 2.7 and things now work on my desktop computer.

As DiapDealer repeatedly advised you .... if you have a bug, post a full bug report!!!!

We would like to keep KindleUnpack working all of the way back to python 2.5 if possible and definitely python 2.6. So the lack of a bug report on this issue just makes it harder to see what is broken under python 2.6 meaning we can't fix it.

The g and h versions are experimental and we are discussing changing them considerably as I write this. Testing against those versions is pretty much meaningless since all of that code is in flux.

So if you have a backtrace or error report that is reproducible and shows what is broken under Python 2.6, we would like to see it in whatever version of KindleUnpack that you have that shows the error.

Exactly as DiapDealer requested many times.

KevinH

[tkeo]

Hi Kevin,

Originally Posted by KevinH

Please hold off on both your g and h versions for now. Let's focus on the resc code changes and then work from there.

EDIT: I have removed the attached files to withdraw them this time.

Originally Posted by KevinH

So I want to change and simplify greatly the K8RESCProcessor code and remove taglist and make some other changes. The idea was to design something that could keep sequence/order as it parsed the resc data as well as something that only kept the actual information we needed and not try to keep the entire spine.

We can change the opf generation code to add in all extra EXTH metadata (ASIN, etc,) inside a commented out section which will be ignored when parsed in the RESC code if you pass the file back through kindlegen after unpacking.

In short, I agree with your ideas.
I will see parseRESC.py. I am very interested in seeing it.

Thanks,

[JSWolf]

Originally Posted by KevinH

JSWolf,



As DiapDealer repeatedly advised you .... if you have a bug, post a full bug report!!!!

We would like to keep KindleUnpack working all of the way back to python 2.5 if possible and definitely python 2.6. So the lack of a bug report on this issue just makes it harder to see what is broken under python 2.6 meaning we can't fix it.

The g and h versions are experimental and we are discussing changing them considerably as I write this. Testing against those versions is pretty much meaningless since all of that code is in flux.

So if you have a backtrace or error report that is reproducible and shows what is broken under Python 2.6, we would like to see it in whatever version of KindleUnpack that you have that shows the error.

Exactly as DiapDealer requested many times.

KevinH

I didn't get any errors running 0.72h under Python 2.6. It just didn't run. But I updated Python to 2.7 and it runs. The AZW3 eBook I had an error with under 0.72f now works under 0.72h and it works under the text plugin as well.

If you like, I can go back to 0.72f and try the eBook again so I can post the error message.

[tkeo]

I thought it was not necessary to say they were proposed and exrimental, since DiapDealer had already said they were PROPOSED, and posted two versions simultaneously...
Anyway, I have removed them.

Please use v0.72f for testing.

Originally Posted by KevinH

JSWolf,
The g and h versions are experimental and we are discussing changing them considerably as I write this. Testing against those versions is pretty much meaningless since all of that code is in flux.

[KevinH]

Hi tkeo,

After looking at what you have done some more, I would like to adopt your idea of passing more information in filenames[] to make creating the opf easier.

So I propose replacing filenames[dir,filename] with something along the lines of:

fileinfo[key, dir, filename]

The "key" will be one of the following:
- skelid (skeleton number/partno converted to string) to match with RESC skelid
- "coverpage" - used when we create a coverpage
- None

This fileinfo will be passed to the opf code along with k8resc (the much simpler version I proposed) and the spine and manifest will be built as it was originally "on the fly", with the key used to access the spine_order, spine_idrefs, spine_properties, as we build it.

I have modified my mobi_k8resc.py version to add a "x_" prefix to the given idrefs from the RESC. I have also offloaded most of the RESC header and extraction processing from kindleunpack.py into the new mobi_k8resc.py and then changed the resc returned to simply be the k8resc object (as it will have all of the other info you stored in the resc[] list.


The code in the opf then does the following:

For the metadata, we use what you have but teach it to grok the new k8resc extra metadata format instead.

For the manifest:

we use the imgnames, fileinfo, and used_map information as before but now we looks up the original idref in the k8resc.spine_idrefs dictionary as needed otherwise we use our itemXXXXX style idrefs.

For the spine:

if k8resc exists and length of k8res.spine_order >= number of parts:

- we can use k8resc.spine_order to create the proper order and get all idrefs, and page properties from the k8resc object for the spine

else

- we build the spine in the order given by the fileinfo array which should match the k8proc.partInfo order as we always did previously.

How does that sound?


To better explain what I am thinking, I have thrown together some changes with a new mobi_k8resc.py and some associated changes in the opf but this is only tested briefly for epub2! It will most likely die under epub3 but I think it illustrates the approach I was thinking about. If you agree, we would then try to reduce the redundancy using this mobi_opf.py and fix it to work with epub3 and also remove all the dependencies on mobi_taglist.py since it should no longer be needed.

So please see KindleUnpack_v072x_test.zip that is attached.

This is not a public release!!!!!

This version is just meant to demonstrate the approach and ideas so we can decide how best to move forward.

Take care,

KevinH

[tkeo]

Hi Kevin,

Originally Posted by KevinH

So I propose replacing filenames[dir,filename] with something along the lines of:
fileinfo[key, dir, filename]

So please see KindleUnpack_v072x_test.zip that is attached.

It's great! We move forward based on your proposal.

Alongside opf, I would like to discuss about insertion of cover page and linear attribute in spine.

1. The insertion of cover page to spine is not implemented in your posted code.
I have modified to insert a cover page if it is not in spine.
It is needed to consider to offload from kindleunpack.py.
Not to implement insertion is one of choices.
Attached file rtl_example2.mobi is an example that the cover page is inserted.

2. I worry about that linear attributes might be needed to be stored in K8RESCProcessor for very special case, like attached files rtl_example1.mobi and rtl_example3.mobi.

Take care,
tkeo

[KevinH]

Hi tkeo,

Originally Posted by tkeo

Hi Kevin,
Alongside opf, I would like to discuss about insertion of cover page and linear attribute in spine.

1. The insertion of cover page to spine is not implemented in your posted code.
I have modified to insert a cover page if it is not in spine.
It is needed to consider to offload from kindleunpack.py.
Not to implement insertion is one of choices.
Attached file rtl_example2.mobi is an example that the cover page is inserted.

Yes, I assumed we only insert a cover page when the RESC spine has an initial entry with no "skelid" because an existing cover page had been removed. In other words, I thought we only include a cover page if the initial epub had a cover page.

Your version always inserts a cover page if the cover_img exists but is not referenced in part 0. Your way may produce a cover page when the original had none. That said, if RESC is missing or does not parse, your version will still create a cover page when my version we will not.

I am okay with either approach, you choose.

The code is short enough we can keep it in kindleunpack.py or offload it to your mobi_cover.py by passing in the k8resc and k8proc to a routine that does the insertions there.

2. I worry about that linear attributes might be needed to be stored in K8RESCProcessor for very special case, like attached files rtl_example1.mobi and rtl_example3.mobi.

So we need to modify mobi_k8resc.py to either:

A) Add spine_pagelinear={} to go with the spine_pageprops

or

B) We combine both linear and properties into one spine_pageattributes dictionary that holds all attributes of that itemref in the spine in its own dictionary.

or

C) we treat spine_pageprops as a dictionary holding a list [linear, properties]

Since you are more up to date on what can appear here, are their any other page attributes of the itemref besides "linear" and "properties" or are they the only ones that exist?

If there are others I say we go with option B as it is more general. If linear and properties are the only ones that exist we can use options A or B your choice.

Just let me know which one you prefer or simply modify mobi_k8resc.py to do what you need.

If we are in agreement on this general approach, then will you please take a shot at fixing the epub3 versions of things in the mobi_opf.py since you know that code better than I do.

Also please remove the dependency on taglist and remove mobi_taglist.py.

Then we can try merging the OPF pieces to remove redundancy while keeping it as simple as possible.

Thanks!

Kevin

[tkeo]

Hi Kevin,

It seems you made tattr = None if it is empty intentionally.
Is it better to keep tattr = {} since None type errors occur?
I have encounterd the errors with <spine> and <dc:format/>.

Thanks,

EDIT
I have nomore time to read your post tonight. I will answer tomorrow.

[KevinH]

Hi,

Yes, we can set empty tattr to {} and not None.
Simply remove this line from near the end of parsetag():

if tattr is not None and len(tattr)== 0: tattr = None

Yes, that will make it simpler not to have to test for tattr not being None all of the time.

Kevin

Originally Posted by tkeo

Hi Kevin,

It seems you made tattr = None if it is empty intentionally.
Is it better to keep tattr = {} since None type errors occur?
I have encounterd the errors with <spine> and <dc:format/>.

Thanks,

EDIT
I have nomore time to read your post tonight. I will answer tomorrow.

PS: Attached is a patch to v072x that will add linear and deal with the coverpage as well as clean up a few nits. Please apply it and see if it deals with the issues you have mentioned and if not we can modify it as needed. I have changed the coverpage code in kindleunpack to let the k8resc object do the necessary prepending to the order so that class data it is better encapsulated.

[tkeo]

Hi Kevin,

Originally Posted by KevinH

A) Add spine_pagelinear={} to go with the spine_pageprops

or

B) We combine both linear and properties into one spine_pageattributes dictionary that holds all attributes of that itemref in the spine in its own dictionary.

or

C) we treat spine_pageprops as a dictionary holding a list [linear, properties]

Since you are more up to date on what can appear here, are their any other page attributes of the itemref besides "linear" and "properties" or are they the only ones that exist?

If there are others I say we go with option B as it is more general. If linear and properties are the only ones that exist we can use options A or B your choice.

Just let me know which one you prefer or simply modify mobi_k8resc.py to do what you need.

I have never seem but id attribute might in an itemref tag.
http://www.idpf.org/epub/301/spec/ep...c-itemref-elem

Besides, it makes simpler to generate itemref tags in mobi_opf.py.
So, I prefer to go with B).

As for properties, it is allowed to have more than two values, ex.

<itemref idref="titlepage" properties="page-spread-right rendition:layout-pre-paginated"/>

This is just note. We can store it as a string in a dict.

I have modified to change from spine_pageprops and spine_linear to spine_pageattributes, prepending 'x_' to cover_name and id attribute.

A little bit irregular way though, I have also modified to insert cover page in the case RESC is not exist nor spine is not in the RESC.
Please see attached patch.

Thanks,