Does Calibre support the new kindle format? - Page 3

kovidgoyal · 03-11-2012, 10:18 PM

@KevinH: Look at writer2/indexer.py that contains code to generate CNCX and INDX records. And feel free to ask me if you have questions about that or anything else.

KevinH · 03-11-2012, 11:15 PM

Hi Kovid,

Thanks, will do.

KevinH

Quote:

Originally Posted by kovidgoyal

@KevinH: Look at writer2/indexer.py that contains code to generate CNCX and INDX records. And feel free to ask me if you have questions about that or anything else.

KevinH · 03-12-2012, 10:04 AM

Hi,

On the good news front, I used mobi_unpack.py's ability to split out a standalone kf8 ebook and while splitting, to remove the DATP section and its pointer in the header.

Using KindlePreviewer, I was able to open and read the standalone KF8 even through it was missing its DATP section. All links and things appear to work.

So we should not have to worry about reversing the DATP section at all for KF8's.

kovidgoyal · 03-12-2012, 11:26 AM

That's good. I would have been surprised if the DATP records were needed, as they exist in old style MOBI periodicals as well and didn't seem to be used for anything there either.

KevinH · 03-13-2012, 02:17 PM

Hi,

Okay, I have some test code that uses the latest version of lxml and html5lib and it can take an epub and process things to recreate the divtbl, skeltbl, and othtbl. I am sure it is full of bugs and missing features but all of that can be fixed.

I know lxml is already part of calibre, but what about html5lib? Is it okay to use that python module as well?

I seem to be stuck on how to take the divtbl, skeltbl, othtbl data and convert it to the correct set of index records to be stored in the mobi. The code in mobi/writer2/index.py seems to be hard coded to just the tags used for ncx's and not for general tag maps.

From the mobi_unpack code, I know the tag used (tag number, number of values, tag mask, etc) for each type of index (skelidx, dividx, othidx). Of course the meaning of what a 2 tag, or 3 tag is different for each type of index.

So it seems what I need to do is take the mobi/writer2/index.py and change it to have the structured data, list of ctoc strings, and the tag info (tag number, number of values, mask) all passed in and have it do the remainder. Basically create what is the reverse of the mobi_index.py MobiIndex code and then allow whatever needs to to call it.

The smarts for creating the input to this routine would come from the ncx code, skel code, div code, etc but the code for writing any index (given the data and tag info) would be generic. Effectively we are separating the index writing and reading code from the index interpretation code since that changes depending on what type of index is being used.

Does this sound like the right approach? If so, any hints on how best to approach this would be welcome.

Ps: If it helps, the skelidx, dividx, and othidx all use one control byte as follows:

skelidx 0x0a
dividx 0x0f
othidx 0x03

So we have the tag table, and control byte value for each (and know the number of control bytes is always 1), perhaps we should be passing in the control byte to the generic index writer as well.

KevinH · 03-21-2012, 07:03 PM

Hi,

Good news, I have a python based standalone KF8 mobi writer (not the joint type). It still needs a lot of work but it is working. It uses many pieces from calibre so incorporating it into calibre should not be that hard to do. So support for writing standalone Kindle KF8 files should be coming.

KevinH

bhoyt · 04-15-2012, 10:31 PM

I'm looking forward to producing KF8 output. Keep up the fantastic work KevinH.

JSWolf · 04-15-2012, 10:34 PM

At least we can convert KF8 to ePub.

apastuszak · 04-16-2012, 10:03 AM

Ok, dumb question time. Are KF8 files .mobi files also? Can I have a the ePub, mobi and KF8 files all sitting in Calibre linked to one record?

I do have to say that owning a nook and a kindle may have been a mistake. Converting files back and forth between devices is getting a tad tedious.

itimpi · 04-16-2012, 10:28 AM

Yes - you can have as many different formats linked to a single book record as you like. What you cannot have is multiple instances of the same format.

DiapDealer · 04-16-2012, 12:33 PM

Quote:

Originally Posted by itimpi

What you cannot have is multiple instances of the same format.

Which is why you can't really have a MOBI version AND a KF8 version of the same ebook at the same time. Calibre can handle MOBIs that contain a KF8 book, but for all intents and purposes, they are still both MOBIs to calibre... not separate, distinct formats like, say... .LIT and .EPUB.

apastuszak · 04-16-2012, 12:38 PM

But it's possible to have a KF8 and mobi7 inside the same .mobi files and the Kindle will use either or, depending on what it can handle?

DiapDealer · 04-16-2012, 01:06 PM

Quote:

Originally Posted by apastuszak

But it's possible to have a KF8 and mobi7 inside the same .mobi files and the Kindle will use either or, depending on what it can handle?

Technically, yes... if you or someone you know built the book using a recent version of Kindlegen/KindlePreviewer and chose to distribute the book independently of Amazon, then both the KF8 version and the fallback MOBI version will exist in the same file. Any of Amazon's devices/apps should be able to open and read that file (one version or the other). But the likelihood of Amazon delivering one of those dual-function MOBI/KF8 files to a customer is looking pretty slim. They seem to be delivering a single purpose file to customers based on what their device/app is capable of.

But regardless, calibre will still (currently) consider the ebook as being in the MOBI format. There is no KF8 selection when choosing the input format for a conversion—only MOBI.

kovidgoyal · 04-16-2012, 01:17 PM

Quote:

Originally Posted by DiapDealer

Which is why you can't really have a MOBI version AND a KF8 version of the same ebook at the same time. Calibre can handle MOBIs that contain a KF8 book, but for all intents and purposes, they are still both MOBIs to calibre... not separate, distinct formats like, say... .LIT and .EPUB.

Sure you can, name your kf8 file .azw3 (the next calibre release recognizes .awz3 as a valid ebook extension and can view and convert it). But if it were me, I'd just use .mobi.

DiapDealer · 04-16-2012, 01:33 PM

Quote:

Originally Posted by kovidgoyal

Sure you can, name your kf8 file .azw3 (the next calibre release recognizes .awz3 as a valid ebook extension and can view and convert it).

Only because calibre—and its development pace—is so spiffy!

03-13-2012, 02:17 PM	#35
KevinH Sigil Developer Posts: 8,764 Karma: 6000000 Join Date: Nov 2009 Device: many	Hi, Okay, I have some test code that uses the latest version of lxml and html5lib and it can take an epub and process things to recreate the divtbl, skeltbl, and othtbl. I am sure it is full of bugs and missing features but all of that can be fixed. I know lxml is already part of calibre, but what about html5lib? Is it okay to use that python module as well? I seem to be stuck on how to take the divtbl, skeltbl, othtbl data and convert it to the correct set of index records to be stored in the mobi. The code in mobi/writer2/index.py seems to be hard coded to just the tags used for ncx's and not for general tag maps. From the mobi_unpack code, I know the tag used (tag number, number of values, tag mask, etc) for each type of index (skelidx, dividx, othidx). Of course the meaning of what a 2 tag, or 3 tag is different for each type of index. So it seems what I need to do is take the mobi/writer2/index.py and change it to have the structured data, list of ctoc strings, and the tag info (tag number, number of values, mask) all passed in and have it do the remainder. Basically create what is the reverse of the mobi_index.py MobiIndex code and then allow whatever needs to to call it. The smarts for creating the input to this routine would come from the ncx code, skel code, div code, etc but the code for writing any index (given the data and tag info) would be generic. Effectively we are separating the index writing and reading code from the index interpretation code since that changes depending on what type of index is being used. Does this sound like the right approach? If so, any hints on how best to approach this would be welcome. Ps: If it helps, the skelidx, dividx, and othidx all use one control byte as follows: skelidx 0x0a dividx 0x0f othidx 0x03 So we have the tag table, and control byte value for each (and know the number of control bytes is always 1), perhaps we should be passing in the control byte to the generic index writer as well. Last edited by KevinH; 03-13-2012 at 03:43 PM. Reason: added more info

03-21-2012, 07:03 PM	#36
KevinH Sigil Developer Posts: 8,764 Karma: 6000000 Join Date: Nov 2009 Device: many	support coming Hi, Good news, I have a python based standalone KF8 mobi writer (not the joint type). It still needs a lot of work but it is working. It uses many pieces from calibre so incorporating it into calibre should not be that hard to do. So support for writing standalone Kindle KF8 files should be coming. KevinH

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Does Calibre support the Kindle FIRE?	TomTeaPack	Devices	3	11-15-2011 09:02 PM
Why People developed books in mobi or prc format, If Kindle support PDF book	tushar	Kindle Formats	4	08-11-2011 03:50 AM
Calibre support of new Kindle 3	chilady1	Calibre	7	01-15-2011 04:36 PM
Kindle support in calibre	user_none	Amazon Kindle	85	09-20-2009 05:26 PM
Does Calibre Support Kindle DX?	Sonist	Calibre	16	06-20-2009 07:59 PM

03-11-2012, 10:18 PM	#31
kovidgoyal creator of calibre Posts: 45,355 Karma: 27182818 Join Date: Oct 2006 Location: Mumbai, India Device: Various	@KevinH: Look at writer2/indexer.py that contains code to generate CNCX and INDX records. And feel free to ask me if you have questions about that or anything else.

03-12-2012, 10:04 AM	#33
KevinH Sigil Developer Posts: 8,764 Karma: 6000000 Join Date: Nov 2009 Device: many	Hi, On the good news front, I used mobi_unpack.py's ability to split out a standalone kf8 ebook and while splitting, to remove the DATP section and its pointer in the header. Using KindlePreviewer, I was able to open and read the standalone KF8 even through it was missing its DATP section. All links and things appear to work. So we should not have to worry about reversing the DATP section at all for KF8's.

03-12-2012, 11:26 AM	#34
kovidgoyal creator of calibre Posts: 45,355 Karma: 27182818 Join Date: Oct 2006 Location: Mumbai, India Device: Various	That's good. I would have been surprised if the DATP records were needed, as they exist in old style MOBI periodicals as well and didn't seem to be used for anything there either.

04-15-2012, 10:31 PM	#37
bhoyt Enthusiast Posts: 39 Karma: 10 Join Date: Jun 2011 Location: Oamaru, New Zealand Device: KT, Android Tablet and Phone	I'm looking forward to producing KF8 output. Keep up the fantastic work KevinH.

04-15-2012, 10:34 PM	#38
JSWolf Resident Curmudgeon Posts: 79,757 Karma: 145864619 Join Date: Nov 2006 Location: Roslindale, Massachusetts Device: Kobo Libra 2, Kobo Aura H2O, PRS-650, PRS-T1, nook STR, PW3	At least we can convert KF8 to ePub.

04-16-2012, 10:03 AM	#39
apastuszak Fanatic Posts: 509 Karma: 125599 Join Date: Feb 2011 Location: Bensalem, PA Device: Kindle Scribe, Nook Glowlight 4, Apple iPad Pro 12.9"	Ok, dumb question time. Are KF8 files .mobi files also? Can I have a the ePub, mobi and KF8 files all sitting in Calibre linked to one record? I do have to say that owning a nook and a kindle may have been a mistake. Converting files back and forth between devices is getting a tad tedious.

04-16-2012, 10:28 AM	#40
itimpi Wizard Posts: 4,553 Karma: 950151 Join Date: Nov 2008 Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)	Yes - you can have as many different formats linked to a single book record as you like. What you cannot have is multiple instances of the same format.

04-16-2012, 12:38 PM	#42
apastuszak Fanatic Posts: 509 Karma: 125599 Join Date: Feb 2011 Location: Bensalem, PA Device: Kindle Scribe, Nook Glowlight 4, Apple iPad Pro 12.9"	But it's possible to have a KF8 and mobi7 inside the same .mobi files and the Kindle will use either or, depending on what it can handle?

Advert

Advert