Need for a dual mobi metadata tweaker

[KevinH]

Hi,
Okay in its own thread, the topic is to create a metadata tweaker that will work with dual mobis as created by kindlegen so that people can tweak the metadata but NOT split the mobi 7 and mobi 8 parts.

Right now it would need the following capability:

1) add EXTH 501 element with value EBOK to both headers so that cover display will work better for some devices


There also seems to be some interest in allowing a uuid style ASIN element to be changed to something else but I am not sure why this is needed for something that was a personal document since it would not be an exact match to any official ASIN from Amazon's catalog.

I did not think kindlegen even generated an ASIN field? Does it?

[DiapDealer]

Originally Posted by KevinH

I did not think kindlegen even generated an ASIN field? Does it?

No it doesn't.

It's just another of those things that affects whether something shows up as a 'book' vs a 'document' on some devices. Usually, just the non-empty presence of the 113 EXTH record is enough for those devices that check for it. I'm not familiar with the 504 record myself, but maybe it's the same scenario?

Maybe it would be best to let the user choose what they want to put into the ASIN record(s)... if at all possible? I've never seen them contain a uuid-looking string like was previously mentioned. I've only ever seen them contain the usual 10 character Amazon ASIN string. *shrugs*

[KevinH]

Hi,
Would it be better to present a list of all meta values in each header with some sort of checkbox to delete specific ones plus text input fields to prompt for specific 501, 113, and 504 values to add.

We could probably pull something like that together using basic gui widgets, but then we would be bumping up against Mobi Perl which I do not want to duplicate the functionality of unless no one is currently supporting it.

Perhaps a better approach would be to patch Mobi Perl to handle dual mobis instead? It has been literally years since I have written any Perl but I once knew how to do that. Do you think the mobi perl author would be interested in adding dual mobi support?

If not, perhaps just hacking up a 501 EBOK adder will be enough to start and go from there.

KevnH

[DiapDealer]

tompe's still around from time to time. I wouldn't want to speak for him, so perhaps we should just ask in the mobiperl thread if he has any desire to see the functionality of mobiperl extended (or minds having another tool that overlaps a bit in functionality)?

[AcidWeb]

Fake ASIN allow self-made e-books to use HQ cover system.

Combo MOBI with no modifications -> Ugly Personal tag. Low quality build-in cover image.

EXTH 501 set to EBOK. -> No Personal tag. Low quality build-in cover image.

EXTH 501 set to EBOK. EXTH 113 and 504 - Fake ASIN. -> No Personal tag. No cover at all unless ASIN is real one. In that case HQ cover is downloaded from Amazon cloud.
But when we upload this file with Calibre a HQ cover will be sideloaded to Kindle cache.
HQ cover can be also uploaded manually: /system/thumbnails/thumbnail_{uuid}_{cdetype}_portrait.jpg

Format of fake ASIN is not important. @DiapDealer - previously mentioned format is used by Calibre. And it is output of python function: uuid.uuid4()

[DiapDealer]

Gotcha. Makes sense to use a uuid so you never have to worry about duplicates, I guess.

So if this tool would generate a uuid for those two asin records, you're then going to have to manually name and sideload the cover image you want to use to your device? Or is there a calibre feature/plugin that checks the asin record(s) and handles the naming/sideloading of the image automatically when using calibre to send books to your device?

[AcidWeb]

If cdetype and asin is set Calibre will automatically handle upload of HQ cover.
That is one of Calibre features.

[KevinH]

Hi,
When I get some time next week I will hack together a quick script from pieces of DumpMobiHeader.py and pieces of mobi_split.py to create a very simple command line tool that will take one parameter which is an ASIN string and the add (or change if it already exists) 501 to be EBOK and 113 and 504 to be set to the passed in ASIN string.

Once I get something hacked together, you can test it to see if it does what you want or not. If anything much more than that is needed, we ask the Mobi-Perl author (tompe) if they want to support dual mobis before doing anything else.

How does that sound?

KevinH

[AcidWeb]

IMHO perfect. That is enough for regular users. Power-users will have solid foundations to make own modifications - if they are will be needed.

Personally that is all I need. If that will work I will embed it to KCC ASAP.

[KevinH]

Here is a pre-pre-pre alpha code to test and play with.

Warnings!
- It has not really been properly tested
- It has been thrown together with bunches of little pieces
- It is not unicode safe (for paths and things)
- This is just really proof of concept
- I do almost no error checking
- it is just a quick and dirty mishmash

kbhend$ python dualmetafix.py thisisatest test.mobi out.mobi
DualMetaFixer v001
test.mobi .MOBI
adding enth so trimming 000000000000000000000000
adding enth so trimming 00000000000000000000000000000000000000
adding enth so trimming 00000000000000000000000000000000000000
adding enth so trimming 000000000000000000000000
adding enth so trimming 00000000000000000000000000000000000000
adding enth so trimming 00000000000000000000000000000000000000

When I examine the out.mobi using DumpMobiHeader_v012.py I see the following:

Mobi 7 Header:
EXTH MetaData

Key: "ASIN_(504)"
Value: "thisisatest"

Key: "ASIN_(113)"
Value: "thisisatest"

Key: "CDE_Type_(501)"
Value: "EBOK"

and in the Mobi 8 Header I see:

EXTH MetaData

Key: "ASIN_(504)"
Value: "thisisatest"

Key: "ASIN_(113)"
Value: "thisisatest"

Key: "CDE_Type_(501)"
Value: "EBOK"


So please test this to see if it actually shows the cover you want with one of your dual mobi ebooks. Please note, nothing is written to the input file but use this with file copies just to be super safe.

Please let me know what errors you get and if it helps at all. If so, I will spend the time to make it unicode safe and clean it up, etc. Otherwise, it may not be worth the effort.

See the attached zip.

KevinH

[KevinH]

I had some more spare time and adding in some basic error testing and attempted to make things unicode safe by grabbing all arguments as utf-8 encoded and converting paths to full unicode only when needed.

So I would say this is now beta quality. Still not pretty. Still a real mishmash.

The palm database access stuff would be better as its own class and and the header and exth should all probably be their own classes as well to encapsulate the access routines.

Oh well. Please give this a try and let me know what troubles you run into. I am sure there are ways to speed this up as well by memory mapping the file and replacing specific sections directly in memory as opposed to using python strings but let's see if this works and is fast enough for what is needed.

Attached is dualmetafix.py version v003

[AcidWeb]

I'm waiting for tester with Paperwhite to check it out but headers look OK.
Processing speed is quite high too - I tested it on 500mb MOBI files and it need only few seconds to complete the job. Current version is working fast enough.

EDIT:
It might be silly but please add credits and licence information to this code. I will be embedding it soon.

[KevinH]

Hi,

Since it is extracted from recent KindleUnpack code (and really does not use any of the original mobiunpack code at all) it is under the same license and basic credits:

print " This program is free software: you can redistribute it and/or modify"
print " it under the terms of the GNU General Public License as published by"
print " the Free Software Foundation, version 3."
print " Copyright © 2013 P. Durrant, K. Hendricks, S. Siebert, fandrieu, DiapDealer, nickredding."

So please just add that to the usage() function.

Kevin

[AcidWeb]

I will add it to file header and KCC README/Credits.
This script will not output any text when it will be part of KCC.

[KevinH]

BTW,

If this works, it should also work just as well for mobis that have only one header - either mobi 7's or mobi 8's as long as they have their own EXTH region already created. Some really old mobis will not have an EXTH and so the program will not work for those older ones.

KevinH