Embedding fonts in AZW3/KF8 - Page 3

kovidgoyal · 07-13-2012, 12:11 AM

Just adding fontsignature to the EXTH header has no effect on my Touch with both joint and standalone KF8 output.

kovidgoyal · 07-13-2012, 12:18 AM

Incidentally, to make everybodies life easier the new version 2.5 of KindlePreviewer from Amazon has a Kindle Touch device mode that can be used for testing this. No need to copy to the touch everytime.

NiLuJe · 07-13-2012, 04:07 PM

@kovidgoyal: Thanks for the KP 2.5 hint, that's indeed way more practical to check this kind of stuff

.

Anyway, got it to pop up in KF8 (standalone) with this version of the patch.

Not sure what did it, since I simply updated the reported KindleGen version, and tweaked the EXTH flags to match KindleGen (and keeping the fontsignature). I'm guessing the EXTH flags did the trick... Since I didn't look at the joint output at all, there may be similar tweaks needed there to get it working on joint files. (I used the flags from the mobi7 part, there's another bit set on the mobi6 part, I put a comment about it in there).

On a sidenote, KG 2.5 adds new stuff in the exth headers:

Code:

524 (524): 'fr'
525 (525): 'horizontal-lr'
535 (535): '0626-3a91e28'

524 seems to be the language of the book (again).
525 looks related to the fixed-layout/orientation stuff... The firmware on the touch is a bit more verbose about it, I've seen a bunch of stuff related to this in my logs:

Code:

120627:192615 cvm[1484]: E HTMLBookMetadata:Error::Unknown string metadata for HXDATA constant: 518
120627:192615 cvm[1484]: I HTMLBookMetadata:Information::Failed to get orientation lock value
120627:192615 cvm[1484]: I HTMLBookMetadata:Information::Failed to get book type value
120627:192615 cvm[1484]: I HTMLBookMetadata:Information::Unknown fixed layout value
120627:192615 cvm[1484]: I HTMLBookMetadata:Information::Unknown original resolution value
120627:192615 cvm[1484]: I HTMLBookMetadata:Information::Unknown zero gutter value
120627:192615 cvm[1484]: I HTMLBookMetadata:Information::Unknown zero margin value
120627:192615 cvm[1484]: I HTMLBookMetadata:Information::Unknown region magnification value
120627:192616 cvm[1484]: E HTMLBookMetadata:Error::Unknown string metadata for HXDATA constant: 525

535 is the build/rev number of KG

EDIT: Killed deprecated attachments. Fix merged upstream.

DiapDealer · 07-13-2012, 04:13 PM

Thanks for tip about KP 2.5. KP 2.4 obliviously informs me I have the latest version.

119MB ?!?! What the heck?

NiLuJe · 07-13-2012, 04:58 PM

Okay, just checked with another input file *without* embedded fonts, and, yep, that seems to be it.

No fonts:
mobi6 EXTH Flags: 100001010000
mobi7 EXTH Flags: 1010000

Fonts:
mobi6 EXTH Flags: 1100001010000
mobi7 EXTH Flags: 1000001010000

So bit 12 seems to be the key here

.

Haven't checked if everything implodes if it's set and there's no fonts embedded, so, err, don't use the patch expecting everything will be rainbows and unicorns

...

(fontsignature is there in every case, so it doesn't seem to hurt or be of use for this).

JSWolf · 07-13-2012, 04:59 PM

So can someone put together a Windows version and I will happily test it with KF8/AZW3 files. Thanks.

NiLuJe · 07-13-2012, 07:48 PM

Yeah, bit 12 in the EXTH flags seems to be the sole trigger

.

So here's an updated patch that doesn't inject a fake fontsignature anymore, and with a crappy hackish conditional that *should* set the correct bit in the bitflag only when we actually *have* embedded fonts. I *think* I patched joint KF8 output too, but I've only checked standalone KF8 files. (And I only patched the mobi7 header, not both, unlike Kindlegen, since I don't see how it would be useful on a mobi6...)

EDIT: Updated patch with a completely unrelated fix, but it's in the same file, and I don't feel like setting up quilt right now... Anyway, the 'rights' metadata entry was injected twice in the exth headers.

EDIT²: Killed deprecated attachments. Fix merged upstream.

kovidgoyal · 07-13-2012, 11:44 PM

Quote:

Originally Posted by DiapDealer

119MB ?!?! What the heck?

Probably because they have to include the toolkits of all the devices they emulate.

@Niluje: Thanks you saved me a good deal of hair pulling

@JSWolf: The fix will be in the next release. You can test it to your hearts content at that time.

NiLuJe · 07-14-2012, 02:14 AM

@kovidgoyal: Happy to help

.

Since EXTH 129 (KF8 Cover URI) was mentioned earlier, I took a closer look, and it seems to always point to the thumbnail... That might explain why the automatic custom thumbnail generation on the Touch was broken... (or not, it looks like a bit of a mess, I've actually seen it try to get it directly from Amazon via HTTP in the system logs...)
Tiny patch to test this theory

(I haven't checked further than making sure it was applied, and didn't make the conversion process implode, I'll test my Touch theory tomorrow...).

On a sidenote: KindleGen seems to use KF8 resources (images/fonts) count (125) / KF8 unknown count (131) the other way around than Calibre... (say, 0 image/fonts & 10 unknown with Kindlegen vs. 10 image/fonts & 0 unkown with Calibre when converting the exact same thing). :?

EDIT: On the other hand, I have a book built with KG where thumboffset is larger than both those fields... I'm starting to get what you meant by the hair pulling comment... ;D

EDIT²: Fixed patch, refs are base32, not base16

.

EDIT^3: Killed deprecated attachments. Fix merged upstream.

kovidgoyal · 07-14-2012, 02:18 AM

Since the touch has problems generating cover thumbnail for mobi 6 files as well, I doubt the cover uri field in the exth will make any difference. However, there's no harm in adding it, provided it does not mess with the go to cover function.

NiLuJe · 07-14-2012, 02:45 AM

@kovidgoyal: So far, it doesn't seem to help either on the few KF8 files that I tried, but then it's not terribly reliable in the first place... I'll check again after a restart, in case there's some weird caching involved. (Go To > Cover seems to still point at the right place, at least).

I did get to check the KF8 fonts stuff on the actual device, though, and, yep, it works

.

EDIT: Well, after a reboot, it seems to do something: it *deletes* the thumbnail that Calibre built. -_-" (On the plus side, there's no broken .part file, but you still end up with the default crappy thumbnail). So, yeah, still no idea how it's really supposed to work, but that was definitely a bad idea ;D

kovidgoyal · 07-14-2012, 02:57 AM

I've committed code to add the thumbnail URI and swap the resources/unknown counts.

NiLuJe · 07-14-2012, 03:30 AM

Yeah, even with the swap, it still *deletes* the thumbnail sent by calibre, and we end up with the default thumb ;/. Sorry about that...

One thing I missed, though: KindleGen seems to systematically put the thumbnail over/after the resources count, and as the last resource:

KG:
KF8 unknown count (125): 0
KF8 thumbnail URI (129): 'kindle:embed:000B'
KF8 resources (images/fonts) count (131): 10
coveroffset (201): 9
thumboffset (202): 11

Calibre:
KF8 unknown count (125): 0
KF8 thumbnail URI (129): 'kindle:embed:0001'
KF8 resources (images/fonts) count (131): 11
coveroffset (201): 0
thumboffset (202): 1

----

KG:
KF8 unknown count (125): 0
KF8 thumbnail URI (129): 'kindle:embed:0003'
KF8 resources (images/fonts) count (131): 2
coveroffset (201): 1
thumboffset (202): 3

Calibre:
KF8 unknown count (125): 0
KF8 thumbnail URI (129): 'kindle:embed:0002'
KF8 resources (images/fonts) count (131): 3
coveroffset (201): 1
thumboffset (202): 2

----

KG:
KF8 unknown count (125): 0
KF8 thumbnail URI (129): 'kindle:embed:0009'
KF8 resources (images/fonts) count (131): 8
coveroffset (201): 5
thumboffset (202): 9

Calibre:
KF8 unknown count (125): 0
KF8 thumbnail URI (129): 'kindle:embed:0004'
KF8 resources (images/fonts) count (131): 7
coveroffset (201): 3
thumboffset (202): 4

I'm getting a bit cross eyed, so I'll look into that tomorrow

.

kovidgoyal · 07-14-2012, 03:36 AM

Different versions of kindlegen have historically done different things with regard to placement of the thumbnail. I highly doubt that is going to influence anything.

JSWolf · 07-14-2012, 10:51 AM

Quote:

Originally Posted by kovidgoyal

Probably because they have to include the toolkits of all the devices they emulate.

@Niluje: Thanks you saved me a good deal of hair pulling

@JSWolf: The fix will be in the next release. You can test it to your hearts content at that time.

Thanks. Be happy to test it out.

07-13-2012, 04:07 PM	#33
NiLuJe BLAM! Posts: 13,477 Karma: 26012494 Join Date: Jun 2010 Location: Paris, France Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E	@kovidgoyal: Thanks for the KP 2.5 hint, that's indeed way more practical to check this kind of stuff . Anyway, got it to pop up in KF8 (standalone) with this version of the patch. Not sure what did it, since I simply updated the reported KindleGen version, and tweaked the EXTH flags to match KindleGen (and keeping the fontsignature). I'm guessing the EXTH flags did the trick... Since I didn't look at the joint output at all, there may be similar tweaks needed there to get it working on joint files. (I used the flags from the mobi7 part, there's another bit set on the mobi6 part, I put a comment about it in there). On a sidenote, KG 2.5 adds new stuff in the exth headers: Code: 524 (524): 'fr' 525 (525): 'horizontal-lr' 535 (535): '0626-3a91e28' 524 seems to be the language of the book (again). 525 looks related to the fixed-layout/orientation stuff... The firmware on the touch is a bit more verbose about it, I've seen a bunch of stuff related to this in my logs: Code: 120627:192615 cvm[1484]: E HTMLBookMetadata:Error::Unknown string metadata for HXDATA constant: 518 120627:192615 cvm[1484]: I HTMLBookMetadata:Information::Failed to get orientation lock value 120627:192615 cvm[1484]: I HTMLBookMetadata:Information::Failed to get book type value 120627:192615 cvm[1484]: I HTMLBookMetadata:Information::Unknown fixed layout value 120627:192615 cvm[1484]: I HTMLBookMetadata:Information::Unknown original resolution value 120627:192615 cvm[1484]: I HTMLBookMetadata:Information::Unknown zero gutter value 120627:192615 cvm[1484]: I HTMLBookMetadata:Information::Unknown zero margin value 120627:192615 cvm[1484]: I HTMLBookMetadata:Information::Unknown region magnification value 120627:192616 cvm[1484]: E HTMLBookMetadata:Error::Unknown string metadata for HXDATA constant: 525 535 is the build/rev number of KG EDIT: Killed deprecated attachments. Fix merged upstream. Last edited by NiLuJe; 09-30-2012 at 05:17 PM.

07-13-2012, 04:58 PM	#35
NiLuJe BLAM! Posts: 13,477 Karma: 26012494 Join Date: Jun 2010 Location: Paris, France Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E	Okay, just checked with another input file without embedded fonts, and, yep, that seems to be it. No fonts: mobi6 EXTH Flags: 100001010000 mobi7 EXTH Flags: 1010000 Fonts: mobi6 EXTH Flags: 1100001010000 mobi7 EXTH Flags: 1000001010000 So bit 12 seems to be the key here . Haven't checked if everything implodes if it's set and there's no fonts embedded, so, err, don't use the patch expecting everything will be rainbows and unicorns ... (fontsignature is there in every case, so it doesn't seem to hurt or be of use for this). Last edited by NiLuJe; 07-13-2012 at 05:07 PM.

07-13-2012, 07:48 PM	#37
NiLuJe BLAM! Posts: 13,477 Karma: 26012494 Join Date: Jun 2010 Location: Paris, France Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E	Yeah, bit 12 in the EXTH flags seems to be the sole trigger . So here's an updated patch that doesn't inject a fake fontsignature anymore, and with a crappy hackish conditional that should set the correct bit in the bitflag only when we actually have embedded fonts. I think I patched joint KF8 output too, but I've only checked standalone KF8 files. (And I only patched the mobi7 header, not both, unlike Kindlegen, since I don't see how it would be useful on a mobi6...) EDIT: Updated patch with a completely unrelated fix, but it's in the same file, and I don't feel like setting up quilt right now... Anyway, the 'rights' metadata entry was injected twice in the exth headers. EDIT²: Killed deprecated attachments. Fix merged upstream. Last edited by NiLuJe; 09-30-2012 at 08:15 PM.

07-14-2012, 02:14 AM	#39
NiLuJe BLAM! Posts: 13,477 Karma: 26012494 Join Date: Jun 2010 Location: Paris, France Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E	@kovidgoyal: Happy to help . Since EXTH 129 (KF8 Cover URI) was mentioned earlier, I took a closer look, and it seems to always point to the thumbnail... That might explain why the automatic custom thumbnail generation on the Touch was broken... (or not, it looks like a bit of a mess, I've actually seen it try to get it directly from Amazon via HTTP in the system logs...) Tiny patch to test this theory (I haven't checked further than making sure it was applied, and didn't make the conversion process implode, I'll test my Touch theory tomorrow...). On a sidenote: KindleGen seems to use KF8 resources (images/fonts) count (125) / KF8 unknown count (131) the other way around than Calibre... (say, 0 image/fonts & 10 unknown with Kindlegen vs. 10 image/fonts & 0 unkown with Calibre when converting the exact same thing). :? EDIT: On the other hand, I have a book built with KG where thumboffset is larger than both those fields... I'm starting to get what you meant by the hair pulling comment... ;D EDIT²: Fixed patch, refs are base32, not base16 . EDIT^3: Killed deprecated attachments. Fix merged upstream. Last edited by NiLuJe; 09-30-2012 at 03:58 PM.

07-14-2012, 02:45 AM	#41
NiLuJe BLAM! Posts: 13,477 Karma: 26012494 Join Date: Jun 2010 Location: Paris, France Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E	@kovidgoyal: So far, it doesn't seem to help either on the few KF8 files that I tried, but then it's not terribly reliable in the first place... I'll check again after a restart, in case there's some weird caching involved. (Go To > Cover seems to still point at the right place, at least). I did get to check the KF8 fonts stuff on the actual device, though, and, yep, it works . EDIT: Well, after a reboot, it seems to do something: it deletes the thumbnail that Calibre built. -_-" (On the plus side, there's no broken .part file, but you still end up with the default crappy thumbnail). So, yeah, still no idea how it's really supposed to work, but that was definitely a bad idea ;D Last edited by NiLuJe; 07-14-2012 at 03:01 AM.

07-13-2012, 12:11 AM	#31
kovidgoyal creator of calibre Posts: 43,858 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Just adding fontsignature to the EXTH header has no effect on my Touch with both joint and standalone KF8 output.

07-13-2012, 12:18 AM	#32
kovidgoyal creator of calibre Posts: 43,858 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Incidentally, to make everybodies life easier the new version 2.5 of KindlePreviewer from Amazon has a Kindle Touch device mode that can be used for testing this. No need to copy to the touch everytime.

07-13-2012, 04:13 PM	#34
DiapDealer Grand Sorcerer Posts: 27,549 Karma: 193191846 Join Date: Jan 2010 Device: Nexus 7, Kindle Fire HD	Thanks for tip about KP 2.5. KP 2.4 obliviously informs me I have the latest version. 119MB ?!?! What the heck?

07-13-2012, 04:59 PM	#36
JSWolf Resident Curmudgeon Posts: 73,976 Karma: 128903378 Join Date: Nov 2006 Location: Roslindale, Massachusetts Device: Kobo Libra 2, Kobo Aura H2O, PRS-650, PRS-T1, nook STR, PW3	So can someone put together a Windows version and I will happily test it with KF8/AZW3 files. Thanks.

07-14-2012, 02:18 AM	#40
kovidgoyal creator of calibre Posts: 43,858 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Since the touch has problems generating cover thumbnail for mobi 6 files as well, I doubt the cover uri field in the exth will make any difference. However, there's no harm in adding it, provided it does not mess with the go to cover function.

07-14-2012, 02:57 AM	#42
kovidgoyal creator of calibre Posts: 43,858 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	I've committed code to add the thumbnail URI and swap the resources/unknown counts.

07-14-2012, 03:30 AM	#43
NiLuJe BLAM! Posts: 13,477 Karma: 26012494 Join Date: Jun 2010 Location: Paris, France Device: Kindle 2i, 3g, 4, 5w, PW, PW2, PW5; Kobo H2O, Forma, Elipsa, Sage, C2E	Yeah, even with the swap, it still deletes the thumbnail sent by calibre, and we end up with the default thumb ;/. Sorry about that... One thing I missed, though: KindleGen seems to systematically put the thumbnail over/after the resources count, and as the last resource: KG: KF8 unknown count (125): 0 KF8 thumbnail URI (129): 'kindle:embed:000B' KF8 resources (images/fonts) count (131): 10 coveroffset (201): 9 thumboffset (202): 11 Calibre: KF8 unknown count (125): 0 KF8 thumbnail URI (129): 'kindle:embed:0001' KF8 resources (images/fonts) count (131): 11 coveroffset (201): 0 thumboffset (202): 1 ---- KG: KF8 unknown count (125): 0 KF8 thumbnail URI (129): 'kindle:embed:0003' KF8 resources (images/fonts) count (131): 2 coveroffset (201): 1 thumboffset (202): 3 Calibre: KF8 unknown count (125): 0 KF8 thumbnail URI (129): 'kindle:embed:0002' KF8 resources (images/fonts) count (131): 3 coveroffset (201): 1 thumboffset (202): 2 ---- KG: KF8 unknown count (125): 0 KF8 thumbnail URI (129): 'kindle:embed:0009' KF8 resources (images/fonts) count (131): 8 coveroffset (201): 5 thumboffset (202): 9 Calibre: KF8 unknown count (125): 0 KF8 thumbnail URI (129): 'kindle:embed:0004' KF8 resources (images/fonts) count (131): 7 coveroffset (201): 3 thumboffset (202): 4 I'm getting a bit cross eyed, so I'll look into that tomorrow . Last edited by NiLuJe; 07-14-2012 at 03:32 AM.

07-14-2012, 03:36 AM	#44
kovidgoyal creator of calibre Posts: 43,858 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Different versions of kindlegen have historically done different things with regard to placement of the thumbnail. I highly doubt that is going to influence anything.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Questions about KF8/AZW3/Mobi8 from/to conversion	pirl8	Calibre	12	09-14-2015 11:01 AM
Science Fiction Wells, H.G.: The War of the Worlds. KF8/AZW3. v1.0 2012-05-21	JSWolf	Other Books	5	09-13-2012 06:43 AM
Embedding fonts	JaneFancher	Calibre	14	04-10-2010 09:28 PM
Embedding fonts	AprilHare	ePub	6	01-02-2010 11:42 AM

Advert

Advert