KF8 conversion and support in Calibre?

Rhialto · 10-21-2011, 05:53 PM

As you will see from a long thread in News and a shorter thread in the Kindle forum, Amazon has announced a new ebook format called .KF8 to compete with the upcoming epub3. The Kindle fire will have KF8 support.

Will it take long for calibre to support this new format? Are there any implications for those of us like me who currently download .mobi and .epub files, interconvert and manage them in Calibre and read them on Kindles and on iPad?

JSWolf · 10-21-2011, 06:02 PM

There is no way to know how long it will take. It depends on how long it takes to get the DRM stripped and the format reverse engineered.

wallcraft · 10-21-2011, 07:10 PM

Quote:

Originally Posted by JSWolf

It depends on how long it takes to get the DRM stripped and the format reverse engineered.

The DRM stripping is important, but it is separate from reverse engineering the format which won't have to wait for DRM stripping. Samples from the Kindle store don't have DRM, and there will presumably be DRM-free KF8's just as there are DRM-free AZW's.

user_none · 10-21-2011, 07:23 PM

Quote:

Originally Posted by wallcraft

The DRM stripping is important, but it is separate from reverse engineering the format which won't have to wait for DRM stripping. Samples from the Kindle store don't have DRM, and there will presumably be DRM-free KF8's just as there are DRM-free AZW's.

I don't think I've ever seen a DRM free TPZ...

kovidgoyal · 10-21-2011, 09:32 PM

It's impossible to guess how long reverse engineering a format is likely to take based on the rather vague an incomplete information Amazon has released.

Since the use of KF8 is completely optional with Kindles, it shouldn't impact anything as far as calibre is concerned.

JeremyR · 10-22-2011, 01:21 AM

Obviously it would only be a one way conversion, but presumably someone could make a plugin that would invoke Kindlegen 2.0 (I believe someone did that with the first one to make azw files, not just mobi)

nickredding · 10-23-2011, 08:48 PM

Converting existing books/recipes to KF8 format won't be all that interesting since presumably the MOBI format will continue to be supported on the new generation devices as well as KF8, and the documents won't look any different. What is interesting is the potential to significantly enhance recipe formatting, taking advantage of the KF8 HTML5/CSS3 capabilities.

jimad · 01-13-2012, 09:25 PM

Quote:

Originally Posted by nickredding

Converting existing books/recipes to KF8 format won't be all that interesting since presumably the MOBI format will continue to be supported on the new generation devices as well as KF8, and the documents won't look any different. What is interesting is the potential to significantly enhance recipe formatting, taking advantage of the KF8 HTML5/CSS3 capabilities.

I don't understand this statement. I have a Kindle Fire, and I have previous versions of the Kindle, and I have the new kf8 kindlegen, and the documents I create certainly do look different, with kf8 and Kindle Fire generally rendering much more reasonably. Also converting an existing EPUB to KF8 is much more likely to be successful that when using the previous kindlegen output MOBI7. ?

theducks · 01-13-2012, 09:33 PM

Quote:

Originally Posted by jimad

I don't understand this statement. I have a Kindle Fire, and I have previous versions of the Kindle, and I have the new kf8 kindlegen, and the documents I create certainly do look different, with kf8 and Kindle Fire generally rendering much more reasonably. Also converting an existing EPUB to KF8 is much more likely to be successful that when using the previous kindlegen output MOBI7. ?

If the styling is not there (because Mobi did not support) in the original, you can't pull an 'improvement' out of thin air.

BTW Give the crew a break. KF8 has been in the wild only a few days.

Of course, you are free to write the new plugin and submit it soonest

KevinH · 01-14-2012, 10:22 AM

Hi,

If you check out the end of the New Mobi Decoder thread in the Mobi forum there already exists a python tool that will unpack KF8 mobis and convert them to epub. So it would be easy to make a stop gap plugin that that imports an epub when given a KF8 mobi as input.

That said, the format is not nearly reverse engineered enough to have it create KF8 mobis. For example, there are sometimes multiple DATP records that seem to be needed (but that we have not figured out yet) as well as FDST records that control how the rawml is split up, and three new index types, one for skeleton files, one for divs or fragments, and one for guide elements. These can not be skipped. Amazon has even added metadata elements to the old Mobi part of the file including a metadata 121 that that stores the section number of the BOUNDARY section, etc. And I am sure there is much more that needs to be looked at.

The problem is you would simply be better off grabbing the kindlegensrc.zip record SRCS section and using it instead with Calibre.

So people with nodrm mobis created by the new Kindlegen would be better to run the new unpacker and use the kindlegensrc.zip (typically an epub renamed) and working from there with Sigil and Calibre.

I would be happy to help with answering questions about the code in mobi_unpacker.py (and its support files) if any wants it.

KevinH

pdurrant · 01-14-2012, 10:32 AM

The next interesting thing that needs to be done is to find some more KF8 samples that have been downloaded from the Kindle store for the Kindle Fire, and see if Amazon has stripped out the mobi section, as they do for the SRCS section. I suspect that they will be doing this.

All this reverse engineering will become a lot easier once the Kindle for Mac and Kindle for PC are updated to handle KF8 file, both for ease of downloading samples from Amazon, and possible debugger use.

DiapDealer · 01-15-2012, 10:18 AM

Quote:

Originally Posted by pdurrant

The next interesting thing that needs to be done is to find some more KF8 samples that have been downloaded from the Kindle store for the Kindle Fire, and see if Amazon has stripped out the mobi section, as they do for the SRCS section. I suspect that they will be doing this.

I'm probably a bit off-topic here, but...
I'm hoping Amazon allows the average content creator to upload separate MOBI and KF8 files for their books at some point. There's just too many irreconcilable differences between the KF8 and MOBI content built from the exact same source, IMO. Too many situations where making the KF8 really shine, will really botch up the fall-back mobi.

If not, the creative crowd may have to come up with a tool to stitch together an optimized MOBI with an optimized KF8 for uploading.

I'm already experiencing situations where the compromise to "make it work" for both formats results in two inferior products.

pdurrant · 01-15-2012, 10:28 AM

Quote:

Originally Posted by DiapDealer

I'm probably a bit off-topic here, but...
I'm hoping Amazon allows the average content creator to upload separate MOBI and KF8 files for their books at some point. There's just too many irreconcilable differences between the KF8 and MOBI content built from the exact same source, IMO. Too many situations where making the KF8 really shine, will really botch up the fall-back mobi.

If not, the creative crowd may have to come up with a tool to stitch together an optimized MOBI with an optimized KF8 for uploading.

I'm already experiencing situations where the compromise to "make it work" for both formats results in two inferior products.

It is possible to specify different CSS for generating the Mobi and the KF8, which might be sufficient.

DiapDealer · 01-15-2012, 11:05 AM

Quote:

It is possible to specify different CSS for generating the Mobi and the KF8, which might be sufficient.

Well that's a huge start, at least... I obviously didn't read far enough.

KevinH · 01-15-2012, 11:26 AM

Hi Paul,

Quote:

Originally Posted by pdurrant

The next interesting thing that needs to be done is to find some more KF8 samples that have been downloaded from the Kindle store for the Kindle Fire, and see if Amazon has stripped out the mobi section, as they do for the SRCS section. I suspect that they will be doing this.

I agree with you. Since the DRM on each mobi ebook is tied specifically to the exact hardware and user, it would make no sense to give both when the DRM will only allow it to work on one specific device anyway.

Has no one been able to buy and download a KF8 file yet? (KF's are not available in Canada). It would be interesting to see what they are doing with the SRCS and older versus KF8 parts of the mobi.

KevinH

10-21-2011, 05:53 PM	#1
Rhialto Enthusiast Posts: 49 Karma: 505676 Join Date: Aug 2011 Device: iPad	KF8 conversion and support in Calibre? As you will see from a long thread in News and a shorter thread in the Kindle forum, Amazon has announced a new ebook format called .KF8 to compete with the upcoming epub3. The Kindle fire will have KF8 support. Will it take long for calibre to support this new format? Are there any implications for those of us like me who currently download .mobi and .epub files, interconvert and manage them in Calibre and read them on Kindles and on iPad?

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
New Kindle format: KF8	Hatgirl	News	146	10-25-2011 10:31 AM
Nook support in Calibre and conversion from Kindle to Nook Color?	doowopper	Devices	4	04-27-2011 04:21 PM
when will calibre support vector graphics in pdf to epub conversion	smith9	Calibre	5	11-13-2010 05:03 AM
Calibre V0.6.13 adds support for LRF conversion	Alexander Turcic	Calibre	3	09-20-2009 12:27 PM
Any Chance Calibre Will Support Direct DOC Conversion?	sirbruce	Calibre	1	03-22-2009 05:14 AM

10-21-2011, 06:02 PM	#2
JSWolf Resident Curmudgeon Posts: 78,944 Karma: 144284074 Join Date: Nov 2006 Location: Roslindale, Massachusetts Device: Kobo Libra 2, Kobo Aura H2O, PRS-650, PRS-T1, nook STR, PW3	There is no way to know how long it will take. It depends on how long it takes to get the DRM stripped and the format reverse engineered.

10-21-2011, 09:32 PM	#5
kovidgoyal creator of calibre Posts: 45,146 Karma: 27110894 Join Date: Oct 2006 Location: Mumbai, India Device: Various	It's impossible to guess how long reverse engineering a format is likely to take based on the rather vague an incomplete information Amazon has released. Since the use of KF8 is completely optional with Kindles, it shouldn't impact anything as far as calibre is concerned.

10-22-2011, 01:21 AM	#6
JeremyR Guru Posts: 973 Karma: 2458402 Join Date: Aug 2010 Location: St. Louis Device: Kindle Keyboard, Nook HD+	Obviously it would only be a one way conversion, but presumably someone could make a plugin that would invoke Kindlegen 2.0 (I believe someone did that with the first one to make azw files, not just mobi)

10-23-2011, 08:48 PM	#7
nickredding onlinenewsreader.net Posts: 324 Karma: 10143 Join Date: Dec 2009 Location: Phoenix, AZ & Victoria, BC Device: Kindle 3, Kindle Fire, IPad3, iPhone4, Playbook, HTC Inspire	Converting existing books/recipes to KF8 format won't be all that interesting since presumably the MOBI format will continue to be supported on the new generation devices as well as KF8, and the documents won't look any different. What is interesting is the potential to significantly enhance recipe formatting, taking advantage of the KF8 HTML5/CSS3 capabilities.

01-14-2012, 10:22 AM	#10
KevinH Sigil Developer Posts: 8,438 Karma: 5702578 Join Date: Nov 2009 Device: many	Hi, If you check out the end of the New Mobi Decoder thread in the Mobi forum there already exists a python tool that will unpack KF8 mobis and convert them to epub. So it would be easy to make a stop gap plugin that that imports an epub when given a KF8 mobi as input. That said, the format is not nearly reverse engineered enough to have it create KF8 mobis. For example, there are sometimes multiple DATP records that seem to be needed (but that we have not figured out yet) as well as FDST records that control how the rawml is split up, and three new index types, one for skeleton files, one for divs or fragments, and one for guide elements. These can not be skipped. Amazon has even added metadata elements to the old Mobi part of the file including a metadata 121 that that stores the section number of the BOUNDARY section, etc. And I am sure there is much more that needs to be looked at. The problem is you would simply be better off grabbing the kindlegensrc.zip record SRCS section and using it instead with Calibre. So people with nodrm mobis created by the new Kindlegen would be better to run the new unpacker and use the kindlegensrc.zip (typically an epub renamed) and working from there with Sigil and Calibre. I would be happy to help with answering questions about the code in mobi_unpacker.py (and its support files) if any wants it. KevinH

01-14-2012, 10:32 AM	#11
pdurrant The Grand Mouse 高貴的老鼠 Posts: 73,590 Karma: 315126578 Join Date: Jul 2007 Location: Norfolk, England Device: Kindle Oasis	The next interesting thing that needs to be done is to find some more KF8 samples that have been downloaded from the Kindle store for the Kindle Fire, and see if Amazon has stripped out the mobi section, as they do for the SRCS section. I suspect that they will be doing this. All this reverse engineering will become a lot easier once the Kindle for Mac and Kindle for PC are updated to handle KF8 file, both for ease of downloading samples from Amazon, and possible debugger use.

Advert

Advert