KFX Format

[Freehunter]

wrong thread

[mattmc]

Originally Posted by kovidgoyal

@mattmc: Inserting soft hyphens before hand means the text rendering engine does not have to figure out the positions at which it is suitable to insert hyphens in a word by itself. This makes its performance somewhat better and means the rendering engine does not have to ship with large collections of language specific hyphenation rules.

Originally Posted by jhowell

See the Wikipedia page for more information on soft hyphens: ...

Thanks, this makes sense, guys.

@Hitch, A lot to say, as usual! And always enlightening. I'll break your response into pieces, here...

Originally Posted by Hitch

I have NOT yet discussed KFX with Amazon. I don't know enough about it to discuss it with them intelligently, and as youse guys know, I'm not a programmer.

Not to jinx it, but I get the feeling that you will be talking to them soon--programming wisdom or not. If Amazon is converting new books into KFX, I'm guessing it's a matter of time before you have a customer whose book is doing something funky after passing through kfxgen, and you will have no way of fixing and re-checking it without giving it back to Amazon to process.

That means unhappy customers, which means unhappy Hitch, which means Amazon is going to get an earful--and I won't envy them for being on the receipt end of your wrath.

Anyway, since I remember from some earlier thread that you have an ECS/ECR account, you could possibly short-circuit all that by emailing them and asking what their plan is for creators proofing books in this format. (Keep in mind that the KFX conversion doesn't happen as part of the normal KDP machinery, it's like that weird SRL thing, but it happens days (or weeks or whatever) after the book has been up for sale.)

You can tell me if I'm being too presumptuous, but I had to suggest it. (If I had an ECS/ECR account, I'd email them--heck, maybe I'll try their generic customer support. Probably at least as likely to work as cutting steel with a butter knife.)

EDIT: Actually, this is rather interesting: https://watchdognation.com/what-is-j...email-address/

A method of last resort...

Originally Posted by Hitch

To wit: here's what we all don't know: we all don't yet know if every bloody book out there, that has escaped into the KDP, over the last 7-ish years, will be reprocessed.

Yeah, hopefully not; it would be chaos if they tried. But I do worry about new ones I'll be putting up.

Originally Posted by Hitch

...the damned ePUB won't intake at iBooks or wherever, because the bloody thing won't validate, naturally.

This rather innocuous comment in the middle of your post alerted me to the fact that iBooks uses ePubCheck as part of their ingest machinery...I have to say, thanks for that! I was using some ePub 3 features in my ePub 2 book, and it definitely doesn't pass ePubCheck.

Back to the workshop...

P.S. Conscious of @Kovid's request earlier, but I think this is still relevant to the format...

[AaronShep]

Originally Posted by mattmc

@kovid, On the soft hyphens, maybe you can clear something up for me. What benefit would there be to adding those to the file itself? What if the user sets a font size that doesn't give a word a reason to break on a particular line? Then you just have a hyphen in the middle of a word...unless the rendering engine is hiding them when they're not necessary, or something, and then you haven't gained anything. I suppose they could be inserting invisible markers in the middle of words that the rendering engine uses as clues to do soft-hyphenation.

Matt, soft hyphens, by definition, are SUPPOSED to be invisible unless they fall at the end of a line. But in early Kindles, they showed up as HARD hyphens, always visible, so you couldn't safely use them anywhere -- and the Publishing Guidelines warned against them. If Amazon is now adding them to KFX files, that would only go to Kindles that could handle them properly.

Aaron

[tomsem]

Originally Posted by AaronShep

Matt, soft hyphens, by definition, are SUPPOSED to be invisible unless they fall at the end of a line. But in early Kindles, they showed up as HARD hyphens, always visible, so you couldn't safely use them anywhere -- and the Publishing Guidelines warned against them. If Amazon is now adding them to KFX files, that would only go to Kindles that could handle them properly.

Aaron

kf8 is webkit based, and that handles soft hyphens. There's a hyphenate plugin for calibre, and I've used that to add hyphenation to things I convert to kf8. So it is only Kindle DX and prior that cannot handle them. If kindlegen is good for anything, it should be stripping out soft hyphens when generating the mobi part. Amazon probably even does that for personal documents that get sent to old Kindles. But I would not bet on it until trying it.

I actually have a K1 and could try the above. I bought it on eBay a couple of years ago for $25 and the seller left a number of books on it. One of them had hyphenation! Unfortunately I deleted the evidence, and when I downloaded a sample of the same book, hyphens were gone. I have to think that soft hyphens work on the thing, but it may be a little tricky to verify this.

[Hitch]

A quickie response, as it's getting lateish here, and I've sworn that I won't be up until the wees again tonight:

Quote mattmc:

Yeah, hopefully not; it would be chaos if they tried. But I do worry about new ones I'll be putting up.

Yabbut, I think you've/this thread has already demonstrated that books that were pubbed QUITE a while back are now able to be downloaded as KFX, no? And if that's the case, doesn't that argue the concept that re-processing is, indeed, occurring?

Non? Oui?

We don't think that this is happening on the fly, do we? Like POD, but for KFX? Someone with an enhanced-typography device orders Book X, and voila!, it's created? No, that sounds daft. Ergo....if you're able to get existing and even PD books, in KFX, my sweeties, I'd say that this does not bode well for the "no re-processing" idea. Hate to be the wearer of the Spock ears, here, but I fear, unless I've misunderstood, when I tried to catch up on the thread, that's the takeaway from the experiments reported here. Right? Or...not?

P.S.: yes, good memory. I do have an ECR account, (rather than a seller's account) for some inexplicable reason. I think they just decided that the poor regular KDP guys didn't need the aggro of my questions, etc. And, yes: I suspect that discussion is going to be sooner rather than later. URGH.

Hitch

[pdurrant]

Originally Posted by mattmc

I admit I'm not familiar with DMCA and how it applies to DRM and so on. I thought the reason why de-DRMing software can legally exist is because it is "pitched" as being for users who want to convert their purchased book into a different format, to read on another device or something.

Unfrotuantely, you are completely wrong.

Providing DRM removal software or using DRM removal software in the US* is a civil offence, so the copyright holder would have to sue you.

If the DRM removal software is provided or used "willfully and for purposes of commercial advantage or private financial gain" then it's a criminal offence, and convicted offenders "shall be fined not more than $500,000 or imprisoned for not more than 5 years, or both, for the first offense".




*Except for certain limited exceptions granted by the Librarian of Congress.

[mattmc]

Originally Posted by AaronShep

Matt, soft hyphens, by definition, are SUPPOSED to be invisible unless they fall at the end of a line. But in early Kindles, they showed up as HARD hyphens, always visible, so you couldn't safely use them anywhere -- and the Publishing Guidelines warned against them. If Amazon is now adding them to KFX files, that would only go to Kindles that could handle them properly.

Tracking, thanks.

Originally Posted by pdurrant

Unfrotuantely, you are completely wrong.

Providing DRM removal software or using DRM removal software in the US* is a civil offence, so the copyright holder would have to sue you.

If the DRM removal software is provided or used "willfully and for purposes of commercial advantage or private financial gain" then it's a criminal offence, and convicted offenders "shall be fined not more than $500,000 or imprisoned for not more than 5 years, or both, for the first offense".

Yeah, I'd kinda gotten that from @jhowell's post, but thanks for clarifying the penalties

Originally Posted by Hitch

Yabbut, I think you've/this thread has already demonstrated that books that were pubbed QUITE a while back are now able to be downloaded as KFX, no? And if that's the case, doesn't that argue the concept that re-processing is, indeed, occurring?

Hm, good point. So, the apocalypse comes

Originally Posted by Hitch

P.S.: yes, good memory. I do have an ECR account, (rather than a seller's account) for some inexplicable reason. I think they just decided that the poor regular KDP guys didn't need the aggro of my questions, etc. And, yes: I suspect that discussion is going to be sooner rather than later. URGH.

Make sure to update us with any official "party lines" you get out of Amazon!

[jhowell]

I have done some more research on how different quality images are sent to different device types for the same book. For this test I used Wafu Cool Japan Guide by Japan Publicity. It is composed of large color images and is free. I chose a representative page, titled "Where to Go?". It is the fifth page (counting the cover) and contains a map of Japan. The results shown below are: image type, dimensions (WxH), file size, and quality of the smallest text on the map.


Test results:

AZW3 download targeted for a Paperwhite II: color JPEG, 925 x 1181, 119 KB, just barely readable.

KFX on Paperwhite II: gray scale JPEG-XR, 654 x 834, 76 KB, slightly less readable than the AZW3 version.

KFX on iPad Air 2: color JPEG, 1504 x 1919, 776 KB, very readable.

KFX on iPhone 6: Identical to the iPad.



That is as far as I can go using the devices I have on hand. I am curious what the results would be for other devices.

If anyone with a higher resolution e-ink Kindle (PW3 or Voyage) wants to contribute they should get this book both as an AZW3 download from Amazon.com and as a KFX delivered directly to device and see what image formats are present.

I am also curious about the Fire tables. I believe that they also support KFX, but I don't know how accessible the document storage is on those devices.

[JSWolf]

Originally Posted by pdurrant

Unfrotuantely, you are completely wrong.

Providing DRM removal software or using DRM removal software in the US* is a civil offence, so the copyright holder would have to sue you.

If the DRM removal software is provided or used "willfully and for purposes of commercial advantage or private financial gain" then it's a criminal offence, and convicted offenders "shall be fined not more than $500,000 or imprisoned for not more than 5 years, or both, for the first offense".




*Except for certain limited exceptions granted by the Librarian of Congress.

We have many grey areas in the US. One of the biggest is when BOTH Tor and John Scalzi posted a link tot he DRM removal tools and advocated their use when you had eBook sites not properly selling Redshirts without DRM. This was authorized by all the rights holders of the eBook.

[JSWolf]

Originally Posted by jhowell

I have done some more research on how different quality images are sent to different device types for the same book. For this test I used Wafu Cool Japan Guide by Japan Publicity. It is composed of large color images and is free. I chose a representative page, titled "Where to Go?". It is the fifth page (counting the cover) and contains a map of Japan. The results shown below are: image type, dimensions (WxH), file size, and quality of the smallest text on the map.


Test results:

AZW3 download targeted for a Paperwhite II: color JPEG, 925 x 1181, 119 KB, just barely readable.

KFX on Paperwhite II: gray scale JPEG-XR, 654 x 834, 76 KB, slightly less readable than the AZW3 version.

KFX on iPad Air 2: color JPEG, 1504 x 1919, 776 KB, very readable.

KFX on iPhone 6: Identical to the iPad.



That is as far as I can go using the devices I have on hand. I am curious what the results would be for other devices.

If anyone with a higher resolution e-ink Kindle (PW3 or Voyage) wants to contribute they should get this book both as an AZW3 download from Amazon.com and as a KFX delivered directly to device and see what image formats are present.

I am also curious about the Fire tables. I believe that they also support KFX, but I don't know how accessible the document storage is on those devices.

So the solution is to download to an iPad or iPhone 6 (or better) and grab the images and replace the ones in the KF8 version with the better images from the iPad.

How does one grab images from an iPad/iPhone KFX eBook?

[Hitch]

Originally Posted by JSWolf

So the solution is to download to an iPad or iPhone 6 (or better) and screen grab the images and replace the ones in the KF8 version with the better images from the iPad.

Jon:

Methinks you're not paying attention, sweetie.

Hitch

[Purple Lady]

Originally Posted by jhowell

I have done some more research on how different quality images are sent to different device types for the same book. For this test I used Wafu Cool Japan Guide by Japan Publicity. It is composed of large color images and is free. I chose a representative page, titled "Where to Go?". It is the fifth page (counting the cover) and contains a map of Japan. The results shown below are: image type, dimensions (WxH), file size, and quality of the smallest text on the map.


Test results:

AZW3 download targeted for a Paperwhite II: color JPEG, 925 x 1181, 119 KB, just barely readable.

KFX on Paperwhite II: gray scale JPEG-XR, 654 x 834, 76 KB, slightly less readable than the AZW3 version.

KFX on iPad Air 2: color JPEG, 1504 x 1919, 776 KB, very readable.

KFX on iPhone 6: Identical to the iPad.



That is as far as I can go using the devices I have on hand. I am curious what the results would be for other devices.

If anyone with a higher resolution e-ink Kindle (PW3 or Voyage) wants to contribute they should get this book both as an AZW3 download from Amazon.com and as a KFX delivered directly to device and see what image formats are present.

I am also curious about the Fire tables. I believe that they also support KFX, but I don't know how accessible the document storage is on those devices.

Were the Paperwhite images barely readable because of font size too small or resolution too low?

[jhowell]

Originally Posted by JSWolf

So the solution is to download to an iPad or iPhone 6 (or better) and grab the images and replace the ones in the KF8 version with the better images from the iPad.

Solution? I sure wouldn't go to that trouble.

Originally Posted by JSWolf

How does one grab images from an iPad/iPhone KFX eBook?

Images in KFX books are unencrypted in the "attachables" files as has been mentioned in this thread previously. Manual extraction is tedious.

If you are asking about iOS hacking then I'm afraid I can't help you.


ETA:One of the oddest things about KFX is that while the text is very heavily protected, the images are totally in the clear.

[jhowell]

Originally Posted by Purple Lady

Were the Paperwhite images barely readable because of font size too small or resolution too low?

The book is composed entirely of images, many of which contain text. The resolution of the image in the KFX PW2 version is too low for the text within the image to be readable, even at maximum magnification. (This book is an extreme, unusual case.)



ETA: The images on the iOS devices were very clear. I suspect that they are at (or very near) the original resolution used when the publisher submitted the book to Amazon.

Edit 2: This may be a sign that Amazon is processing books for KFX from the original source. (As opposed to doing a conversion from the already processed KF8 format.)

[AaronShep]

Originally Posted by Hitch

Yabbut, I think you've/this thread has already demonstrated that books that were pubbed QUITE a while back are now able to be downloaded as KFX, no? And if that's the case, doesn't that argue the concept that re-processing is, indeed, occurring?

Non? Oui?

I think Amazon's statements have made it very clear that they're gradually working their way through at the least the majority of all Kindle books, and probably all. That's the only reason it's taking so long. They've also stated that books may be reprocessed more than once and whenever necessary, as features change. I imagine that slower sellers get low priority, but that would only be temporary protection.

Aaron