Generating mobi and KF8 parts of Kindle file from separate sources

[DiapDealer]

Also, I'd like to add to best8.mobi a small mobi7 part with a simple explanation that it doesn't support the old version, and the url to download a version that does - which should also let it work in Previewer, which would be nice.

I can get behind that scenario. It would be nice to know the ebook is simply not designed for your reader... rather than just receiving a random error message on your reader about; "not being able to open it; try deleting and re-downloading.." Even if it was only an option to have a one sentence "this ebook is not compatible with your device" old-mobi wrapper left behind by the splitter script.

On a related note... the old MOBI-only portion produced by the new splitter (now included with mobi_unpack) doesn't work with the Kindlepreviewer when emulating a Kindle Fire: "Open Error: error opening book.null" But the Kindle Fire Previewer handles other mobi-only files.

[KevinH]

Originally Posted by DiapDealer

I can get behind that scenario. It would be nice to know the ebook is simply not designed for your reader... rather than just receiving a random error message on your reader about; "not being able to open it; try deleting and re-downloading.." Even if it was only an option to have a one sentence "this ebook is not compatible with your device" old-mobi wrapper left behind by the splitter script.

On a related note... the old MOBI-only portion produced by the new splitter (now included with mobi_unpack) doesn't work with the Kindlepreviewer when emulating a Kindle Fire: "Open Error: error opening book.null" But the Kindle Fire Previewer handles other mobi-only files.

Hmm ... either the zero'd out sections are the culprit (but I don't really believe that) or some other information still exists that seems to convince the device that this is a mobi8 ebook when it is not.

There are other metadata elements than the element 121 BOUNDARY offset one that only seem to exist in the old mobi header of KF8s, so perhaps we should be removing them as well?

If I get a free moment over the weekend, I will run a few tests to see what if any info still exists that makes the device think it has a kf8 mobi when it does not.


KevinH

[KevinH]

Hi Peter and DiapDealer,

Okay, I think the easist way to deal with this is to take a kindlegen file optimized for the mobi8 part and the simply replace the mobi7 text with a single section simple raw text file. That way we won't have to worryabout image and font files. I am willing to take a shot at creating this program.

I will use the sample Jerome.mobi as an "optimized KF8 sample".

Would someone use kindlegen to create a simple mobi that only has the message you want to appear when viewed on an older Kindle (with the external url link in it).

I will unpack that mobi to get the mobi7 raw text of the message and the attempt to create a program that will replace the Jerome mobi7 side with this simple message while leaving the mobi8 side intact.

If that works, simple text editing of the raw mobi7 text should allow you to use this program with whatever books you want.

KevinH

Originally Posted by DiapDealer

I can get behind that scenario. It would be nice to know the ebook is simply not designed for your reader... rather than just receiving a random error message on your reader about; "not being able to open it; try deleting and re-downloading.." Even if it was only an option to have a one sentence "this ebook is not compatible with your device" old-mobi wrapper left behind by the splitter script.

On a related note... the old MOBI-only portion produced by the new splitter (now included with mobi_unpack) doesn't work with the Kindlepreviewer when emulating a Kindle Fire: "Open Error: error opening book.null" But the Kindle Fire Previewer handles other mobi-only files.

[PeterHatch]

Originally Posted by KevinH

Would someone use kindlegen to create a simple mobi that only has the message you want to appear when viewed on an older Kindle (with the external url link in it).

Okay, done. Thank you!

[KevinH]

Hi Peter and DiapDealer,

Attached is a very **ALPHA** version of mergeKF8Only.py script.

This is more proof of concept than anything else. It removes sections but leaves a zero length entry in the section table in the palm database to prevent errors with section offsets later. This cuts down on the size of the file but really we should be deleting those section entries as well and not just making them 0 size.

Your message is stored as test.rawml inside the zip archive.

The idea is to then do the following:

python ./mergeKF8Only.py Jerome.mobi test.rawml testoutput.mobi

I was able to open the testoutput.mobi in the KindlePreviewer.app when set to Kindle and see your message, and when set to KindleFire see the actual ebook.

I am sure this is missing lots of pieces that have to be updated but it will give you something to test with as I only have access to two sample KF8 documents.

Please give it a try and let me know how / where it fails. If this works okay, then we can try to make it actually remove the section entries from the table and try to update all other pieces that depend on it.

Edit: I removed the alpha version since the Beta version replaces it.

[KevinH]

Hi Peter and DiapDealer,

If you had a chance to try mergeKF8Only.py, did it work? If so, I will try to improve it to actually remove those sections and try to update the right pieces inside the mobi7 header to reflect the deleted sections of text. If not, what errors have you run into?

Thanks,

KevinH

[KevinH]

Hi,

Here is a Beta version of mergeKF8Only.zip that actually deletes what it needs to and hopefully properly does everything it needs. It will need a lot of testing to detect any omissions, bugs, and issues.

To use it:

grab mergeKF8Only.zip and unzip it

python ./mergeKF8Only.py Jerome.mobi test.rawml testoutput.mobi

If you load testoutput.mobi in KindlePreviewer with Device set to Kindle you should see your message that this is a KF8 Only ebook and that they should download the original mobi formatted ebook from your link.

If you load testoutput.mobi in KindlePreviewer with Device set to Fire you should see the full mobi8 version of the ebook.

Please test it and let me know what bugs you run into. Once I fix the bugs, I will rewrite the code to form proper classes and clean it up.

Hope this does the trick.

Edit: remove Beta version since release version is posted below

KevinH

[PeterHatch]

It seems to work perfectly - just tested it with Captain Blood in Previewer, where both versions look good, and on a Kindle 2 and 3 (as well as Kindle for PC), where the message shows up perfectly. I wasn't expecting the placeholder to include the cover image along with the message, but that makes sense.

Thank you! (And I apologize for taking so long to get to it.)

--
Peter Hatch

[KevinH]

Hi Peter,

Glad to hear it. The cover image comes from the metadata so we could try changing that if you do not like the cover to be present.

Also, it should be easy to edit the test.rawml with any text editor or create a short program to generate it on the fly from the book title and link info. Different size messages should work as well, as long as they are less than 4096 bytes in total which is the maximum size of any single section in the palm database.

Also the message file must be utf-8 encoded and not windows-1252.
So be sure to use utf-8 smart quotes and not windows 1252 smart quotes if you add any smart quotes to the message.

I will spend some time this weekend cleaning up the code, forming proper classes and methods (for speed I just threw code snippets in from various other files I had lying around) and then post a final version.

Take care,

Kevin

Originally Posted by PeterHatch

It seems to work perfectly - just tested it with Captain Blood in Previewer, where both versions look good, and on a Kindle 2 and 3 (as well as Kindle for PC), where the message shows up perfectly. I wasn't expecting the placeholder to include the cover image along with the message, but that makes sense.

Thank you! (And I apologize for taking so long to get to it.)

--
Peter Hatch

[KevinH]

Hi Peter and DiapDealer,

I cleaned up the code, removed redundant pieces, formed proper classes, added debug information, etc.

So here is the release version of mergeKF8Only_v050.zip

mergeKF8Only 0.50
based on mobi_unpack.py code written by: Charles M. Hannum,
P. Durrant, K. Hendricks, S. Siebert, fandrieu, DiapDealer, nickredding.

Description:
In a dual KF8 mobi file, replaces the original mobi portion
with a KF8Only message, leaving the KF8 portion untouched.

Usage:
mergeKF8Only_v050.py -h -d infile.mobi message outfile.mobi

Options:
-h print this help message
-d use verbose debugging

The message file must be:
- a mobi html file in raw markup language format
- less than 4096 bytes in total
- utf-8 encoded

One easy way to generate a message file is to use kindlegen to create a mobi which consists just of your message and then unpack that mobi with WRITE_RAW_DATA set to be true using mobi_unpack.py. A sample test.rawml file is included in the zip archive. It can be edited with any good text editor.

Have fun!

KevinH

[jttraverse]

If I create a Mobi file from an ePub using the new Previewer, and then use Kindlestrip to remove the epub portion of original content so I now have a much smaller Mobi file...

...can this smaller Mobi file be used successfully in both the Kindle and the Kindle Fire, or do I need to split it into separate files (mobi and KF8) and let people pick one or the other based on which Kindle the have?

-JT

[DiapDealer]

Originally Posted by jttraverse

If I create a Mobi file from an ePub using the new Previewer, and then use Kindlestrip to remove the epub portion of original content so I now have a much smaller Mobi file...

...can this smaller Mobi file be used successfully in both the Kindle and the Kindle Fire, or do I need to split it into separate files (mobi and KF8) and let people pick one or the other based on which Kindle the have?

-JT

If all you've done is use kindlestrip to remove the source of a file produced by the latest kindlegen/Previewer, then the resultant MOBI should work on all devices/apps. Those devices/apps that support KF8 will use that format... those that don't will use the MOBI-only portion.

If you're going to be distributing the book yourself, there's no reason you couldn't split it into separate files, but you certainly don't have to.

[jttraverse]

Thanks, I'd rather just leave one Mobi file and let users download that if I can be sure that they'll be able to read it on whatever Kindle they own.

Still, I think the Fire has 8mb internal memory and then 'the cloud'...but does anyone know if older Kindles ever come up against any internal memory/file size limitations. I've just been wondering why Amazon would bother stripping out and delivering a user just one or the other rather than just giving them the newer combined Mobi/KF8...especially on books that are pretty small size to begin with.
-JT

[DiapDealer]

Originally Posted by jttraverse

I've just been wondering why Amazon would bother stripping out and delivering a user just one or the other rather than just giving them the newer combined Mobi/KF8...especially on books that are pretty small size to begin with.
-JT

I think it's mostly because of potential 3G delivery charges... keep that delivery size as small as possible and they won't have angry authors complaining about half of the proceeds from their book being used up with wireless delivery charges. Charges for data that some devices can't even use. Plus eventually, I simply expect MOBI to go away. But in the meantime (and I believe the meantime will be a long time), they have all kinds of options open to them: deliver one, deliver the other, deliver both. The KF8 phase has only begun and they have processes in place to tailor the experience based on the device/app that's doing the downloading.

[Hitch]

Originally Posted by KevinH

Hi Peter and DiapDealer,

I cleaned up the code, removed redundant pieces, formed proper classes, added debug information, etc.

So here is the release version of mergeKF8Only_v050.zip

mergeKF8Only 0.50
based on mobi_unpack.py code written by: Charles M. Hannum,
P. Durrant, K. Hendricks, S. Siebert, fandrieu, DiapDealer, nickredding.

Description:
In a dual KF8 mobi file, replaces the original mobi portion
with a KF8Only message, leaving the KF8 portion untouched.

Usage:
mergeKF8Only_v050.py -h -d infile.mobi message outfile.mobi

Options:
-h print this help message
-d use verbose debugging

The message file must be:
- a mobi html file in raw markup language format
- less than 4096 bytes in total
- utf-8 encoded

One easy way to generate a message file is to use kindlegen to create a mobi which consists just of your message and then unpack that mobi with WRITE_RAW_DATA set to be true using mobi_unpack.py. A sample test.rawml file is included in the zip archive. It can be edited with any good text editor.

Have fun!

KevinH

Hi, Kevin:

Can you advise as to what happens to the output after the file is run through KDP? KDP currently looks for both the K7 and K8 files in order to create a "MasterMobi" that can display properly on both Fire and e-ink Kindles. I know that if, for example, I create an old prc file and the client uploads it, they get an azw-prc, not a Fire-optimized book, obviously. If I use a mobi that I've made with fallback CSS from an ePUB, created via Previewer, I get a MasterMobi that works on all devices. (Assuming, naturally, that my ePUB's CSS is correct).

What happens when KDP encounters your--what do you want to call them--"uncombined" K8 files? I've learned that a K8 mobi "merely" viewed in Previewer isn't an adequate test; a K8 mobi that's been run through the KDP is a different animal, so I'm curious to know what happens to your betas? Display-wise, what happens to the post-KDP files?

Thanks!

Hitch