Handling KFX files in Calibre

[jhowell]

Originally Posted by Amazondoc

And the more I think of it, the more silly and annoying it seems to me to have a "book" broken up into multiple files as KFX does it. Get off my lawn, you young whippersnappers!

There is some justification for that approach. In KFX the cover and text of the book is delivered first and then other images are delivered as separate files of about 3 MB each. That allows the book to be opened and read while it is still downloading. For some books it can make a noticable difference in the time you need to wait to start reading if you have a slow internet connection.

Also, by splitting out the part of the DRM that is individual to each customer into a separate, small voucher file thay are able to make the rest of the files identical for all users. That is not true of the other Kindle formats.

Originally Posted by Amazondoc

TL;DR -- I went ahead and used the chmod technique (Method 2), and all is well for now. Much simpler to deal with the files this way.

Thanks to Terisa, jhowell, and pdurrant for your input.

Glad to hear that you got it sorted out.

[Amazondoc]

Originally Posted by davidfor

But, they can measure how long a page is open. They probably need a minimum time on a page before it gets counted. That way just quickly turning a lot of pages without pause wouldn't be counted.

Is this actually possible? This is the hypothetical timer I was talking about.

Amazon may measure the time I spend looking at a page on a website, but this isn't the same thing as looking at a page in a book on my laptop.

Originally Posted by jhowell

There is some justification for that approach. In KFX the cover and text of the book is delivered first and then other images are delivered as separate files of about 3 MB each. That allows the book to be opened and read while it is still downloading. For some books it can make a noticable difference in the time you need to wait to start reading if you have a slow internet connection.

That's interesting. I can see how some impatient souls may value this difference, but it isn't something I would care about -- not because I have a fast connection, which I don't, but because I don't often download a book to read the instant it's downloaded.

Also, by splitting out the part of the DRM that is individual to each customer into a separate, small voucher file thay are able to make the rest of the files identical for all users. That is not true of the other Kindle formats.

I can see the rationale for a separate "voucher" or "key" file. But that doesn't require the multiple files I was seeing with the few KFX books I downloaded.

Let me stipulate very clearly, though, that I am NOT a techy type, so I have zero legs to stand on when I disparage KFX's multi-file format. All I can justifiably say is that I'd rather deal with one file in my library than five or ten, and things like SD images instead of HD images aren't gonna change my mind on that.

[theducks]

Who knows how much / what information is passed while syncing Amazon books

[davidfor]

Originally Posted by Amazondoc

Is this actually possible? This is the hypothetical timer I was talking about.

Amazon may measure the time I spend looking at a page on a website, but this isn't the same thing as looking at a page in a book on my laptop.

It is absolutely possible and they almost definitely do it. For their e-ink devices, they are recording the time since the last tap to decide if the device should go to sleep. Extending that and recording the time between taps to turn the page would be easy. It is likely they aren't actually reporting the time spent on each page, but an average time, or a number of page turns and duration for a reading session. Or maybe a simple count of the number of pages displayed for long enough to be counted as "read". But, as theducks said, we don't really know what information Amazon is syncing. Or for that matter what the other ereader companies or apps sync.

[Amazondoc]

Originally Posted by davidfor

It is absolutely possible and they almost definitely do it. For their e-ink devices, they are recording the time since the last tap to decide if the device should go to sleep.

Hmmm. That's an interesting point -- but I see some problems with leaping from "timing taps" to "timing book pages".

Remember, ebooks are not necessarily read on Kindle devices -- and even when they are read on Kindles, they are not necessarily read on devices that are dedicated to reading. For example, I read ebooks either on my laptop or on one of my Fires. On the laptop, the Kindle app never "sleeps" as such, and I never have to "tap" at all to advance through a book -- I can scroll continuously. And when I read on a Fire (which I can also scroll instead of tap), my "tap" may be either a tap to advance a page, or it may be a tap to switch to another app, or any of a multitude of other things.

Or maybe a simple count of the number of pages displayed for long enough to be counted as "read".

But "page" count doesn't mean much with ebooks, because that will vary greatly with customized features such as font size, margin size, and so on. You'd have to calculate on something like word count or percentage instead.

An additional problem: Would you report total time to advance to the end of the book, or average time per page, or what? For instance, if I opened a book and left it open on the first page, then went off to do something else for a few hours, then came back and scrolled quickly through the rest of the book, how would that be tallied?

And another problem: If you are trying to count time spent on each and every page (or each and every given percentage point), wouldn't that require a lot of complicated record-keeping? Those files that accompany, for instance, azw-format books (mbp, phl, apnx) are small files. I'm not educated enough to know what's in them, but is there really room there for such granular timekeeping records?

It's an interesting puzzle -- and yes, a bit pointless to ponder, given that Amazon is unlikely to tell us what method it actually uses. OTOH, IMHO their very reluctance tells us that their methods are not all that impressive; there would be no need to hide them if they were.

[jhowell]

Originally Posted by Amazondoc

But "page" count doesn't mean much with ebooks, because that will vary greatly with customized features such as font size, margin size, and so on. You'd have to calculate on something like word count or percentage instead.

Amazon uses Kindle Edition Normalized Page Count (KENPC v3.0) to determine what counts as a "page" for Kindle Unlimited.

To determine a book's page count in a way that works across genres, devices, and display settings, we developed the Kindle Edition Normalized Page Count (KENPC). KENPC is calculated using standard formatting settings (font, line height, line spacing, etc.). We use KENPC to measure the number of pages customers read in your book, starting with the Start Reading Location (SRL) to the end of your book. Amazon typically sets SRL at chapter 1 so readers can start reading the core content of your book as soon as they open it. Non-text elements within books including images, charts and graphs will count toward a book’s KENPC.

Originally Posted by Amazondoc

And another problem: If you are trying to count time spent on each and every page (or each and every given percentage point), wouldn't that require a lot of complicated record-keeping? Those files that accompany, for instance, azw-format books (mbp, phl, apnx) are small files. I'm not educated enough to know what's in them, but is there really room there for such granular timekeeping records?

There are a lot of files on your Kindle that you don't see. And I disagree that it would be all that much data to keep. The current page number, the start time that the current page was flipped to and one bit for each page in the book showing whether or not it has been read would be enough.

[DoctorOhh]

Originally Posted by Amazondoc

Hmmm. That's an interesting point -- but I see some problems with leaping from "timing taps" to "timing book pages".

...~~~...

But "page" count doesn't mean much with ebooks, because that will vary greatly with customized features such as font size, margin size, and so on. You'd have to calculate on something like word count or percentage instead.

An additional problem: ...

And another problem: ...

There are no "problems." Given that Authors get paid per pages read, it goes without saying that Kindle both monitors what they consider pages read and monitors, in some manner, the time it takes to read such a page. They do this to minimize any person or group from scamming the system. How they do it is immaterial. The fact is they are a business and obviously take safeguards to protect their interests. If you don't read the book on a Kindle or Kindle app then it is reasonably safe to say that the author does not get paid full value.

[davidfor]

Originally Posted by Amazondoc

Hmmm. That's an interesting point -- but I see some problems with leaping from "timing taps" to "timing book pages".

Remember, ebooks are not necessarily read on Kindle devices -- and even when they are read on Kindles, they are not necessarily read on devices that are dedicated to reading. For example, I read ebooks either on my laptop or on one of my Fires. On the laptop, the Kindle app never "sleeps" as such, and I never have to "tap" at all to advance through a book -- I can scroll continuously. And when I read on a Fire (which I can also scroll instead of tap), my "tap" may be either a tap to advance a page, or it may be a tap to switch to another app, or any of a multitude of other things.

Firstly, my mention of the e-ink ereaders was just an example of the interaction and how it can be done. But, the same does go for every application. Each application knows everything about your interaction with it. It knows when you scroll whether by tapping, using arrow or page keys or the scroll bar. It also knows things like if it has focus or is covered by other applications. So it is just a matter of deciding what interaction is significant, and what a "page view" is. Compared to the latter, the counting is easy.

But "page" count doesn't mean much with ebooks, because that will vary greatly with customized features such as font size, margin size, and so on. You'd have to calculate on something like word count or percentage instead.

An additional problem: Would you report total time to advance to the end of the book, or average time per page, or what? For instance, if I opened a book and left it open on the first page, then went off to do something else for a few hours, then came back and scrolled quickly through the rest of the book, how would that be tallied?

And another problem: If you are trying to count time spent on each and every page (or each and every given percentage point), wouldn't that require a lot of complicated record-keeping? Those files that accompany, for instance, azw-format books (mbp, phl, apnx) are small files. I'm not educated enough to know what's in them, but is there really room there for such granular timekeeping records?

And that's why I wondered if it was a simple average. Just record the number of "pages" and the time spent. Very simple and not much data needed.

But, I don't know much about the Kindles, I do know a lot about Kobos. They do keep a record of these things. They have a table in their database that includes a page turn count for each book that has been opened. And it keeps records of the reading sessions. It counts each time a book is opened or closed and when. And separate counts for starting and finishing books. Luckily for my free time, I don't have any real details on how this data is stored, so I'm not to tempted to try and read it.

It's an interesting puzzle -- and yes, a bit pointless to ponder, given that Amazon is unlikely to tell us what method it actually uses. OTOH, IMHO their very reluctance tells us that their methods are not all that impressive; there would be no need to hide them if they were.

Well, @jhowell has found some details, but whether they publish the full algorithm is another matter. A more detailed version might be available to authors under the contracts they sign. And it is likely they have an non-disclosure clause in the contract.

[Amazondoc]

Originally Posted by jhowell

There are a lot of files on your Kindle that you don't see.

But as I mentioned previously, lots of Kindle books get read on non-Kindle devices.

The current page number, the start time that the current page was flipped to and one bit for each page in the book showing whether or not it has been read would be enough.

But as I mentioned previously, Kindle books can be read without "flipping" anything.

But let's go with your interpretation -- you know a helluva lot more on the computer front than I do, so you can help educate me. How many bits to record a start time for each page? How many bits to record a stop time for the same page? How many bits to calculate overall time spent on each page? Then multiply that by however many hundred pages a book has. How many bits total to contain that record?

As you quoted from Amazon: "We use KENPC to measure the number of pages customers read in your book, starting with the Start Reading Location (SRL) to the end of your book. "

I think this is the real answer on how Amazon pays authors -- based solely on the "number of pages customers read in your book". It's Occam's Razor.

Originally Posted by DoctorOhhh

Given that Authors get paid per pages read

This is what I've been saying all along. It's based on the number of pages, not whether a reader has spent some minimum amount of time on each page.

it goes without saying that Kindle both monitors what they consider pages read and monitors, in some manner, the time it takes to read such a page.

No, it doesn't go without saying at all. There is no implication of time at all.

Originally Posted by davidfor

Firstly, my mention of the e-ink ereaders was just an example of the interaction and how it can be done.

But whatever method Amazon uses would have to work across all devices.

But, the same does go for every application. Each application knows everything about your interaction with it.

Errrr.... I think we may be talking about multiple definitions of "know". There's a difference between noticing an action and remembering an action -- keeping a permanent record of it.

And that's why I wondered if it was a simple average. Just record the number of "pages" and the time spent. Very simple and not much data needed.

So, according to this theory, you could leave one page of a book open for a few hours while you were off doing something else, then quickly scroll through the rest of the pages, and Amazon would count the book as being read.

But, I don't know much about the Kindles, I do know a lot about Kobos. They do keep a record of these things. They have a table in their database that includes a page turn count for each book that has been opened. And it keeps records of the reading sessions. It counts each time a book is opened or closed and when. And separate counts for starting and finishing books.

This is interesting, and would support your average-page-time thesis -- but only on a dedicated reading device, not on something like a laptop where a book can be left "open" indefinitely. And it doesn't support the minimum-time-per-page thesis in the granular sense (if a page is flipped over too quickly, it doesn't get counted in the total pages read).

[jhowell]

Originally Posted by Amazondoc

But as I mentioned previously, lots of Kindle books get read on non-Kindle devices.

I have no problem with that except for cases that result in the author not being paid.

Originally Posted by Amazondoc

But as I mentioned previously, Kindle books can be read without "flipping" anything.

Flip, scroll, however the user does it the information displayed changes and the software has a chance to update the statistics it collects.

Originally Posted by Amazondoc

But let's go with your interpretation -- you know a helluva lot more on the computer front than I do, so you can help educate me. How many bits to record a start time for each page? How many bits to record a stop time for the same page? How many bits to calculate overall time spent on each page? Then multiply that by however many hundred pages a book has. How many bits total to contain that record?

There is no need to record the start and stop time of each page. It could work by recording the start time when the user flips or scrolls the screen and then comparing it to the current time when a new page was displayed to see if it was long enough. If so then set the one bit that represents that page.

In any case I doubt that Amazon is doing anything this complicated. More likely they just use the highest page reached, as they did before, but also apply a sanity check based on the total time the book was open for reading to make sure that the user did not skip over to many pages to get there.

Or they could be doing something else entirely. I don’t know.

[BetterRed]

Hey folks, how does the current wordy and protracted discussion about how Amazon counts the number 'pages' read relate to converting KFX files with Calibre?

If it's to be continued, perhaps it should be in the Kindle forum. If that's true, one of the other moderators who has participated in the discussion could move the relevant posts to a new thread in that forum. Not all Kindle users use calibre, some of them may have insights into the matter being discussed.

If it does directly relate to converting KFX files with calibre, could someone enlighten me

BR

[darryl]

Whilst we do not know exactly what Amazon is doing we do know how KU works and we do know what Amazon claims to be able to do, albeit without details. We cannot be at all certain that an author will be paid if the book is read on a non-Amazon device or app. I'm not even certain an author will be paid if DRM is removed and the version without DRM is read on a Kindle device or App. I think it is far more probable than not that Amazon has measures in place to detect page-flipping, whether it is accomplished by flipping, scrolling or black magic.

The only way at the moment to be sure a KU author is paid is to read the author's book on a Kindle device or app without removing the DRM infection.

[darryl]

Originally Posted by BetterRed

Hey folks, how does the current wordy and protracted discussion about how Amazon counts the number 'pages' read relate to converting KFX files with Calibre?

If it's to be continued, perhaps it should be in the Kindle forum. If that's true, one of the other moderators who has participated in the discussion could move the relevant posts to a new thread in that forum. Not all Kindle users use calibre, some of them may have insights into the matter being discussed.

If it does directly relate to converting KFX files with calibre, could someone enlighten me

BR

Sorry. Our posts crossed. Like many discussions this has veered off-topic to an extent. The relationship between the topic and the posts in question has become a little more remote. It is certainly not relevant to the technical issues. It's relevance otherwise is that it is discussing whether conversion should take place at all with KU books given the way KU pays authors.

[BetterRed]

@darryl - no worries, and thanks for the explanation, but what's KU? Google says it's Kansas University - does Bezos own that too

It's up to td and DrO to do what they think is best.

BR

[jhowell]

Originally Posted by BetterRed

@darryl - no worries, and thanks for the explanation, but what's KU? Google says it's Kansas University - does Bezos own that too

KU is Kindle Unlimited, an Amazon subscription service that lets you borrow an unlimited number of books (ten at a time) from a large selection of mostly self-published books. Authors are paid based on actual number of pages read by customers.