KFX Format - Page 16

darryl · 11-11-2015, 10:01 PM

Quote:

Originally Posted by elborak

You're not the only one scratching their head over that.

I'm half convinced he has points, but I've been struggling to parse them out of the rambling writing style he utilizes.

From what I can see he does raise points. Though he implies that they are new and have not been debated on mobileread, the contrary seems to be true.

With great difficulty, what I have been able to discern from his posts is:

He seems to subscribe to the books as special snowflakes argument, but makes no mention of the Authors United letter and surrounding events.

His other points are associated with the dangers of the .kfx format, but he seems to ignore what happened when .kf8 was released and treats the worst case scenario as if it is not only inevitable but has already happened,

He is rabidly anti-Amazon to the exclusion of any discussion of the practices of competitors.

EDIT: I should make it clear that these topics are always worthwhile to discuss, though the special snowflake argument really has been done to death. KFX as a format, formats in general and drm are also constant topics of discussion. But these topics should be raised plainly without the anti-Amazon rant and without the misrepresentation not only that they are new and haven't previously been debated but also that such discussion is shut down and frowned upon because Amazon supposedly cannot be criticised. Amazon cops plenty of criticism in these forums even from its supporters, of which I count myself one. At present there is a thread where Paul is attempting to hold Amazon to account for its past statements about access to purchases after an account is closed by them. Also, I and others have made it very plain that we have fears about kfx. I have raised a worst case scenario in the longer term where Amazon goes exclusively .kfx and Adobe's new drm becomes mandatory. However, it seems that our friend believes the sky is falling now and that anyone who does not believe this is burying their heads in the sand.

Rizla · 11-12-2015, 08:49 AM

Quote:

Originally Posted by darryl

But these topics should be raised plainly without the anti-Amazon rant

Thou shalt not criticize Amazon.

elborak · 11-12-2015, 01:30 PM

Criticizing Amazon is fine, in the appropriate venue. In the context of a technical discussion of file formats, it's misplaced.

elborak · 11-12-2015, 03:45 PM

EDIT: (in response to a since deleted post claiming that Amazon criticisms were frowned upon but that defending them was permitted)

Nope, equally unwelcome (as are meta comments like this one, so mea culpa).

Hitch · 11-12-2015, 04:20 PM

Agreed, can we all argue about whether or not Amazon is EVIL in a thread unrelated to the ins-and-outs of the format itself? Or, at LEAST, if/when we KNOW what the hell we're all discussing? As nobody here really knows JACK about KFX, other than a few educated guesses, and some grossly uneducated guesses, it seems fruitless to discuss whether or not Amazon is doing "more evil" with it, AT THIS POINT IN TIME and KNOWLEDGE. I don't mind a good healthy debate on the topic, but leaping to death-and-destruction Chicken Little guesses...just not my bag.

Hitch

JSWolf · 11-12-2015, 04:28 PM

Given what we currently know about KFX and what's being down with KFX, what can KFX do that KF8 cannot do or could not do with a new renderer to do the enhanced typesetting with KF8?

Dr. Drib · 11-12-2015, 05:03 PM

Hitch has it right: Rather than hi-jacking a thread on formats, one can create an anti-Amazon thread.

Any further anti-Amazon posts will be deleted from this thread.

Don
(Moderator)

eschwartz · 11-12-2015, 11:04 PM

Quote:

Originally Posted by notimp

@HarryT: Maybe you have a point there. (Allthough not in regards to this being openly discussed as one of the few options to still get the legacy file format, and that people should do that.)

Still - how many quoteunquote normal Kindle users are actively using it? (Get out your PC, log into Amazon, go to manage your Kindle, search for the Book you bought, look for the submenu item to download the book to your device, select your Kindle, downlod the file, get your USB cable out, connect to the Kindle --

and if you dont do it now - Amazon could, at any time, flip the switch that makes those files still turn up as azw3's, at which point all books you havent done this with - are "lost " in that sense.

As supposed to the way it worked before - where you could entice people to do the switch to calibre without loosing anything, at any point in time - because, there was no race against time against the distribution of a format designed to...

Fill out the rest in your head.

Also, please dont ignore the rest of the points I brought forward.

To go with what HarryT said, in the unusual circumstance where someone is responsible and decides to back up their books in calibre, calibre warns you that KFX is not able to be understood, and helpfully suggests you download your books from Manage Your Kindle.

And KFX is no more likely to become the only available choice than KF8 is likely to receive new, unbreakable DRM.
(Either way, you're screwed.)

Which means... that you should backup your books immediately!
There has NEVER been a guarantee that you will be able to download them later. Many MR members can tell you about the books they lost when their bookstore went out of business!!!

And... as soon as Amazon stops allowing that, IF they stop allowing that, we will all drop Amazon like a hot potato and find some other source of books.

Or, as per the purpose of this thread, someone will find it worthwhile to invest the time to reverse-engineer KFX.
In which case we won't drop Amazon, because we are still able to come to a mutually beneficial business agreement.

...But until KFX is needed, I'm not sure I really expect that it will be reverse-engineered.
There is lower-hanging fruit.

AaronShep · 11-13-2015, 02:41 AM

Quote:

Originally Posted by JSWolf

Given what we currently know about KFX and what's being down with KFX, what can KFX do that KF8 cannot do or could not do with a new renderer to do the enhanced typesetting with KF8?

If I may propose a tentative answer:

KF8 is basically Webkit, with some features enabled and others not. KFX's rendering is a combination of newly-enabled features of Webkit, plus an entirely Amazon-created module that sits on top of Webkit and overrides it in some ways.

Newly-enabled Webkit features include hyphenation and kerning. The new Amazon module includes drop caps, JPEG XR, and for now at least, a bunch of incorrect behaviors.

This Amazon module is where we can expect to see the most future development. I suspect that just hooking it up to Webkit has taken the most effort so far, and has generated the most errors.

I imagine Amazon does NOT want to simply add new features to the open-source Webkit, because then its competitors would have them. So, it has come up with this composite rendering engine. Maybe someone more familiar with the Webkit licensing terms can tell us whether this makes sense.

That's the new layout engine. As to the file formats, I assume they're designed to feed that composite layout engine most efficiently. But the formats we're seeing may also be transitional.

poxalew · 11-15-2015, 04:50 PM

Instead of speculating blindly, let's speculate in a more informed manner

WebKit is governed by the LGPL license, and Amazon does respect the letter (if not necessarily the spirit) of its license by making available the source code to the open source parts of their binaries, including WebKit. You can find the (complete, they claim) open source code used by them at http://www.amazon.com/gp/help/custom...deId=200203720 .

From there we can see that the 5.6.5 firmware for the Kindle e-Readers (specifically the PW2, but the Voyage source code dump is virtually identical) uses the following open source component versions:

Spoiler:

Note WebKit 1.4.2, which is a 4 year old release (see https://trac.webkit.org/browser/rele...K/webkit-1.4.2 for the original code)--but sticking to old-but-known-good code is by no means unusual, and they don't seem to be too concerned about the renderer security. And, since HTML5 has so far been less of a standard and more of "whatever it is that the WhatWG comes up today" while Amazon must offer a stable platform for content generation, it makes sense to stick to a specific release for years.

This, of course, doesn't directly tell us that KFX is being rendered by WebKit--after all, it could be a legacy component for rendering MOBI7 and KF8 content. But we can infer with a strong likelihood that KFX is being rendered by WebKit. Nobody in their right mind would run a widely distributed HTML5 rendering engine written from scratch due to the staggering costs involved in writing a conformant yet stable implementation, and the market has settled on the open source rendering engines (the exception being Microsoft's renderer, which I'm fairly certain we can exclude here

). All of them are licensed under the LGPL. Therefore, if Amazon were to use one, we'd find its code in the list above. The only one is WebKit, so I'm calling WebKit...

kovidgoyal · 11-15-2015, 11:11 PM

@poxalew: As far as I know 4 years ago, WebKit had no support for hyphenation. (calibre's viewer -- which also uses an old version of webkit -- gets hyphenation support via a javascript library). So that would imply that either Amazon is

1) Modifying webkit in which case they would have to open source their modifications

2) Using a javascript library -- which would be very slow, especially on a mobile device

3) Pre-rendering the soft hyphens in KFX which AaronShep's evidence seems to make unlikely

poxalew · 11-16-2015, 03:11 AM

Correct, it didn't. Amazon modified it (or, at least, merged stuff from elsewhere) and has open sourced it; I found some spare time and looked into it.

We can compare upstream https://trac.webkit.org/browser/rele.../Hyphenation.h (which is mostly a stub) to Source/WebCore/platform/text/Hyphenation.h from Amazon's tarball. The latter contains a reasonable hyphenation engine. It uses hunspell's dictionaries (telltale "hyph_en_US.dic" and "hyph_en_GB.dic" references in Hyphenation.cpp.) The code is reasonably well documented and quite readable. It's guarded by #ifdef LAB126 / #endif //LAB126 stanzas, so grepping for LAB126 through Amazon's source should yield interesting data.

So it looks like #1 is the case. #2 is right out (the source code dump doesn't even contain a JS interpreter and I doubt Amazon would have rolled their own for the same economic reasons for which they didn't reimplement rendering), and #3 indeed looks not to be the case.

Having the source is interesting in a number of ways, perhaps even for calibre if at some point in the future the JS hyphenation becomes a bottleneck and needs to be replaced with a lower runtime overhead implementation. Amazon's hyphenation implementation is BSD-licensed according to the copyright headers in the source files.

(By the way, my blanket statement about WebKit being LGPL was wrong; it's mostly BSD. LGPL governs just small parts of JavaScriptCore and a part of WebCore, but fortunately that was enough to force the entire WebCore into the open.)

kovidgoyal · 11-16-2015, 06:35 AM

Interesting, thanks. As for hyphenation in the calibre viewer, at some point, I will be moving it to a modern webkit engine (migrate from QtWebKit to QtWebEngine) so that should be automatic. Though I doubt I will bundle hyphenation dictionaries for too many languages out of the box, calibre is large enough as it is

Yeah WebKit is mostly BSD, thank the stars for the few GPLed bits.

Doitsu · 11-16-2015, 07:03 AM

@poxalew: Amazon introduced automatic hyphenation support for Russian language AZW3 books with the 5.6.1 Kindle PW2 firmware update. (First reported by EbokJunkie in this post.)

I.e., current eInk Kindle models already have built-in hyphenation support, which has been disabled for all languages, except for Russian.

You can easily see this in my test file, which doesn't contain soft hyphens.

DuckieTigger · 11-16-2015, 08:16 AM

Quote:

Originally Posted by kovidgoyal

Though I doubt I will bundle hyphenation dictionaries for too many languages out of the box, calibre is large enough as it is

How about you don't bundle any and make them plugins instead? As needed it will have to get the dictionary as plugin, but it never needs to get it again unless dictionary changes (unlikely).

11-12-2015, 03:45 PM	#229
elborak Enthusiast Posts: 46 Karma: 727310 Join Date: Feb 2011 Device: Kindle Voyage	EDIT: (in response to a since deleted post claiming that Amazon criticisms were frowned upon but that defending them was permitted) Nope, equally unwelcome (as are meta comments like this one, so mea culpa). Last edited by elborak; 11-12-2015 at 05:37 PM.

11-15-2015, 04:50 PM	#235
poxalew Junior Member Posts: 9 Karma: 166666 Join Date: Aug 2014 Device: Kindle PW2	Instead of speculating blindly, let's speculate in a more informed manner WebKit is governed by the LGPL license, and Amazon does respect the letter (if not necessarily the spirit) of its license by making available the source code to the open source parts of their binaries, including WebKit. You can find the (complete, they claim) open source code used by them at http://www.amazon.com/gp/help/custom...deId=200203720 . From there we can see that the 5.6.5 firmware for the Kindle e-Readers (specifically the PW2, but the Voyage source code dump is virtually identical) uses the following open source component versions: Spoiler: Code: atk_1.30.0.tar.gz awesome_3.4.3.tar.gz build_linaro-gcc_4.8.3.tar.gz busybox_1.17.1.tar.gz cairo_1.10.0.tar.gz cramfs_1.1.tar.gz dosfstools_2.11.tar.gz fuse_2.9.2.tar.gz gdb_6.6.tar.gz glib-2.0_2.29.18.tar.gz glib-networking_2.31.0.tar.gz gnutls_2.8.4.tar.gz gtk+-2.0_2.20.1.tar.gz hyphen_2.8.6.tar.gz iptables_1.4.15.tar.gz iw_3.8.tar.gz klibc_1.5.25.tar.gz libgcrypt_1.4.4.tar.gz libgpg-error_1.4.tar.gz libnih_1.0.2.tar.gz libnl_3.2.22.tar.gz libsoup-2.4_2.35.90.tar.gz libstartup-notification-1.0_0.10.tar.gz libusb_0.1.12.tar.gz linux-3.0.35.tar.gz module-init-tools_3.2.2.tar.gz multitouch_7380af2c93dc83f4f09e293717d46eadf7799e89.tar.gz ncurses_5.6.tar.gz pango_1.26.2.tar.gz procps_3.2.7.tar.gz readline_5.2.tar.gz schedutils_1.5.0.tar.gz squashfs_4.2.tar.gz syslog-ng_1.6.12.tar.gz taglib_1.5.tar.gz uboot_2009.08.tar.gz udev_146.tar.gz upstart_0.6.6.tar.gz util-linux_2.12r.tar.gz uuid_1.41.10.tar.gz webkit-1.0_1.4.2.tar.gz wireless_tools_29.tar.gz x11_1.3.2.tar.gz xau_1.0.5.tar.gz xaw7_1.0.7.tar.gz xcb_1.5.tar.gz xcb-proto_1.6.tar.gz xcb-util_0.3.6.tar.gz xclock_1.0.4.tar.gz xdamage_1.1.2.tar.gz xdmcp_1.0.3.tar.gz xev_1.0.4.tar.gz xext_1.1.1.tar.gz xf86-input-evdev_2.4.0.tar.gz xf86-video-imx_2011-04-11.tar.gz xfd_1.0.1.tar.gz xfixes_4.0.4.tar.gz xfont_1.4.1.tar.gz xft_2.1.14.tar.gz xi_1.3.tar.gz xinerama_1.1.tar.gz xkbcomp_1.1.1.tar.gz xkbfile_1.0.6.tar.gz xkeyboard-config_1.4.tar.gz xload_1.0.2.tar.gz xlogo_1.0.2.tar.gz xmlsec1_1.2.16.tar.gz xmu_1.0.5.tar.gz xorg-macros_1.10.0.tar.gz xpm_3.5.8.tar.gz xrandr_1.3.0.tar.gz xrandr-cmd_1.3.2.tar.gz xrender_0.9.5.tar.gz xserver_1.8.2.tar.gz xset_1.2.1.tar.gz xt_1.0.7.tar.gz xtst_1.1.0.tar.gz xv_1.0.5.tar.gz xwd_1.0.3.tar.gz xwininfo_1.0.5.tar.gz Note WebKit 1.4.2, which is a 4 year old release (see https://trac.webkit.org/browser/rele...K/webkit-1.4.2 for the original code)--but sticking to old-but-known-good code is by no means unusual, and they don't seem to be too concerned about the renderer security. And, since HTML5 has so far been less of a standard and more of "whatever it is that the WhatWG comes up today" while Amazon must offer a stable platform for content generation, it makes sense to stick to a specific release for years. This, of course, doesn't directly tell us that KFX is being rendered by WebKit--after all, it could be a legacy component for rendering MOBI7 and KF8 content. But we can infer with a strong likelihood that KFX is being rendered by WebKit. Nobody in their right mind would run a widely distributed HTML5 rendering engine written from scratch due to the staggering costs involved in writing a conformant yet stable implementation, and the market has settled on the open source rendering engines (the exception being Microsoft's renderer, which I'm fairly certain we can exclude here ). All of them are licensed under the LGPL. Therefore, if Amazon were to use one, we'd find its code in the list above. The only one is WebKit, so I'm calling WebKit... Last edited by poxalew; 11-15-2015 at 04:52 PM. Reason: Bah, spelling is hard.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Using Kindle format vs ePub format is like using a compiler vs winzip?	Julius Caesar	Workshop	1	09-01-2013 07:34 PM
iPhone Convert epub format to kindle for iPhone format. Is it possible?	thecyberphotog	Apple Devices	16	03-14-2013 01:04 AM
No 'epub' format shown in Plugboards Format dropdown list	kakkalla	Library Management	3	06-16-2012 04:23 AM
Ebook in PRC format will not convert to any other format	Katelyn	Calibre	0	10-01-2010 07:02 PM
Master Format for multi-format eBook Generation?	cerement	Workshop	43	04-01-2009 12:00 PM

11-12-2015, 01:30 PM	#228
elborak Enthusiast Posts: 46 Karma: 727310 Join Date: Feb 2011 Device: Kindle Voyage	Criticizing Amazon is fine, in the appropriate venue. In the context of a technical discussion of file formats, it's misplaced.

11-12-2015, 04:20 PM	#230
Hitch Bookmaker & Cat Slave Posts: 11,503 Karma: 158448243 Join Date: Apr 2010 Location: Phoenix, AZ Device: K2, iPad, KFire, PPW, Voyage, NookColor. 2 Droid, Oasis, Boox Note2	Agreed, can we all argue about whether or not Amazon is EVIL in a thread unrelated to the ins-and-outs of the format itself? Or, at LEAST, if/when we KNOW what the hell we're all discussing? As nobody here really knows JACK about KFX, other than a few educated guesses, and some grossly uneducated guesses, it seems fruitless to discuss whether or not Amazon is doing "more evil" with it, AT THIS POINT IN TIME and KNOWLEDGE. I don't mind a good healthy debate on the topic, but leaping to death-and-destruction Chicken Little guesses...just not my bag. Hitch

11-12-2015, 04:28 PM	#231
JSWolf Resident Curmudgeon Posts: 79,792 Karma: 146391129 Join Date: Nov 2006 Location: Roslindale, Massachusetts Device: Kobo Libra 2, Kobo Aura H2O, PRS-650, PRS-T1, nook STR, PW3	Given what we currently know about KFX and what's being down with KFX, what can KFX do that KF8 cannot do or could not do with a new renderer to do the enhanced typesetting with KF8?

11-12-2015, 05:03 PM	#232
Dr. Drib Grand Sorcerer Posts: 45,449 Karma: 59592133 Join Date: Jan 2007 Location: Peru Device: KINDLE: Oasis 3, Scribe (1st), Matcha; KOBO: Libra 2, Libra Colour	Hitch has it right: Rather than hi-jacking a thread on formats, one can create an anti-Amazon thread. Any further anti-Amazon posts will be deleted from this thread. Don (Moderator)

11-15-2015, 11:11 PM	#236
kovidgoyal creator of calibre Posts: 45,374 Karma: 27230406 Join Date: Oct 2006 Location: Mumbai, India Device: Various	@poxalew: As far as I know 4 years ago, WebKit had no support for hyphenation. (calibre's viewer -- which also uses an old version of webkit -- gets hyphenation support via a javascript library). So that would imply that either Amazon is 1) Modifying webkit in which case they would have to open source their modifications 2) Using a javascript library -- which would be very slow, especially on a mobile device 3) Pre-rendering the soft hyphens in KFX which AaronShep's evidence seems to make unlikely

11-16-2015, 03:11 AM	#237
poxalew Junior Member Posts: 9 Karma: 166666 Join Date: Aug 2014 Device: Kindle PW2	Correct, it didn't. Amazon modified it (or, at least, merged stuff from elsewhere) and has open sourced it; I found some spare time and looked into it. We can compare upstream https://trac.webkit.org/browser/rele.../Hyphenation.h (which is mostly a stub) to Source/WebCore/platform/text/Hyphenation.h from Amazon's tarball. The latter contains a reasonable hyphenation engine. It uses hunspell's dictionaries (telltale "hyph_en_US.dic" and "hyph_en_GB.dic" references in Hyphenation.cpp.) The code is reasonably well documented and quite readable. It's guarded by #ifdef LAB126 / #endif //LAB126 stanzas, so grepping for LAB126 through Amazon's source should yield interesting data. So it looks like #1 is the case. #2 is right out (the source code dump doesn't even contain a JS interpreter and I doubt Amazon would have rolled their own for the same economic reasons for which they didn't reimplement rendering), and #3 indeed looks not to be the case. Having the source is interesting in a number of ways, perhaps even for calibre if at some point in the future the JS hyphenation becomes a bottleneck and needs to be replaced with a lower runtime overhead implementation. Amazon's hyphenation implementation is BSD-licensed according to the copyright headers in the source files. (By the way, my blanket statement about WebKit being LGPL was wrong; it's mostly BSD. LGPL governs just small parts of JavaScriptCore and a part of WebCore, but fortunately that was enough to force the entire WebCore into the open.)

11-16-2015, 06:35 AM	#238
kovidgoyal creator of calibre Posts: 45,374 Karma: 27230406 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Interesting, thanks. As for hyphenation in the calibre viewer, at some point, I will be moving it to a modern webkit engine (migrate from QtWebKit to QtWebEngine) so that should be automatic. Though I doubt I will bundle hyphenation dictionaries for too many languages out of the box, calibre is large enough as it is Yeah WebKit is mostly BSD, thank the stars for the few GPLed bits.