[Device Plugin] Kobo Touch Extended Driver

[chaley]

Originally Posted by Terisa de morgan

Just out of curiosity, which values will be used for updating these fields? Custom columns? So, a nice relationship with CountPages?

Originally Posted by ownedbycats

Interestingly, I was considering dropping the word count column as I found it less useful than page and chapter counts. But now I have a reason to keep it.

@davidfor is implementing all of these with templates not column keys, giving you the choice of returning the value of a column or calculating something. If so then you can produce a word count "pages * something" where "something" is some average words per page. Isn't as accurate but it lets you get rid of the column.

[OPINION]I think his choice to move to templates for things is inspired. It doesn't take away any functionality, but adds many possibilities. Of course some basic "recipes" will need to be provided, specifically how to return a column value ("{column_key}"), but that isn't hard.[/OPINION]

[JSWolf]

Originally Posted by chaley

@davidfor is implementing all of these with templates not column keys, giving you the choice of returning the value of a column or calculating something. If so then you can produce a word count "pages * something" where "something" is some average words per page. Isn't as accurate but it lets you get rid of the column.

[OPINION]I think his choice to move to templates for things is inspired. It doesn't take away any functionality, but adds many possibilities. Of course some basic "recipes" will need to be provided, specifically how to return a column value ("{column_key}"), but that isn't hard.[/OPINION]

Can we get the formula that Kobo uses for time to read added in as a default?

[chaley]

Originally Posted by JSWolf

Can we get the formula that Kobo uses for time to read added in as a default?

Perhaps, if you figure out what the formula is and where it gets the input numbers.

[ownedbycats]

If Kobo adjusts the speed from the user's activity—I don't quite remember if this is done—it might not even be a 'default.'

[JSWolf]

Originally Posted by chaley

Perhaps, if you figure out what the formula is and where it gets the input numbers.

Here is the post where I saw the formula.

Originally Posted by geek1011

BTW, the algorithm kobo.com uses for calculating the read time range is:

HoursMax = round(WordCount / 200 / 60)
HoursMin = HoursMax - 1
if (HoursMax >= 10) {
        HoursMax += round((HoursMax + .5) / 10) - 1)
}

The word count and page number are from https://kobostats.azurewebsites.net/bookstats/{ISBN}. The word count seems to exclude front/back matter (but it includes tables and code blocks) (I'll try to figure out the details of this later), and the page count seems to have a linear relationship with the word count. Based on the data from a few hundred books, it seems to be the floor of the word count divided by 272 (this is perfect for nearly all of the books, and ~20-40 pages too large for some older ones, which makes me think they were either calculated differently at that time or manually overridden).

___

The device (i.e., storeapi) seems to use the same algorithm, except that the max hour adjustment is floored instead of rounded (is this a bug?):

HoursMax = round(WordCount / 200 / 60)
HoursMin = HoursMax - 1
if (HoursMax >= 10) {
        HoursMax += floor((HoursMax + .5) / 10) - 1)
}

Also note that the on-device KEPUB word counter (i.e., HtmlUtility::countWords) counts differently than the Kobo Store.

[ownedbycats]

Ah, i missed (or forgot—brain's not good today) that post. That looks easy enough to turn into template.

[fogice]

I'm testing the new beta, and I've crashed it twice. The first two transfers were fine (1 book, a handful of books), then crashed at (100 books, 20 books). I have attached the logs from Calibre, as well as the crash reports generated by MacOS. I'm back to only transferring 8 books at a time, which is working so far.

If there's anything more you would like me to test, let me know.

[davidfor]

Originally Posted by chaley

@davidfor is implementing all of these with templates not column keys, giving you the choice of returning the value of a column or calculating something. If so then you can produce a word count "pages * something" where "something" is some average words per page. Isn't as accurate but it lets you get rid of the column.

[OPINION]I think his choice to move to templates for things is inspired. It doesn't take away any functionality, but adds many possibilities. Of course some basic "recipes" will need to be provided, specifically how to return a column value ("{column_key}"), but that isn't hard.[/OPINION]

I don't know about "inspired", but, the choice was a template or some way to specify a column name. Using a template just seemed to be the best for flexibility. And any amusement I get from @ownedbycats attempts to break the template system are just a bonus.

When I announced this in the firmware thread, I put what I was using. Though they were program mode rather than the simpler that you have used.

@JSWolf: No, I will not be adding anything as defaults. Anything is dependent on the column names people use. Not everyone just uses "#pages". And the calculations for the reading times that Kobo supplies are far to simple to bother with, especially as the range is just 1 hour. And I am surprised that you are suggesting using these as you have been critical of the reading estimates that Kobo uses on the device.

[chaley]

Originally Posted by fogice

I'm testing the new beta, and I've crashed it twice. The first two transfers were fine (1 book, a handful of books), then crashed at (100 books, 20 books). I have attached the logs from Calibre, as well as the crash reports generated by MacOS. I'm back to only transferring 8 books at a time, which is working so far.

If there's anything more you would like me to test, let me know.

I looked at the crash reports. In both cases it crashed in libxml2 with a SEGV. That raises the question: are lxml and libxml2 thread safe? This document says libxml2 is but with caveats. I don't know what versions are being used nor can I determine if xmlInitParser() is called as directed.

I also noticed that both crashes happened when serializing the tree for output. Some notes:

• This happens when container.flush_cache() is called. flush_cache() is called from within a thread pool worker. That means that one thread pool is creating a second thread pool. I can easily imagine that this isn't a good thing.
• Because multiple threads can call flush_cache(), more than one thread can call commit_item() on the same tree. Does this write the file twice?
• Because multiple threads can call flush_cache on the same dirtied set, trees can be committed at the same time. I don't know what happens if the two threads attempt to write the same file.

Two suggestions:

• Change flush_cache() to do the work directly without creating threads. This avoids nesting thread pools.
• Put generating the set of dirtied trees and calling commit_item() into a locked with. This avoids possibly committing the same item more than once at the same time.

Something like this:

# This goes in __init__
        super(KEPubContainer, self).__init__(epub_path, log, *args, **kwargs)
        self.flush_lock = threading.Lock() # New code
        self.log = log
        self.log.debug(f"Creating KePub Container for ePub at {epub_path}")

# This replaces flush_cache()
    def flush_cache(self) -> None:
        with self.flush_lock.acquire():
            for name in list(self.dirtied):
                self.commit_item(name, keep_parsed=True)

#    def __flush_cache_impl(self, name: str) -> None:
#       """Flush the cache, writing all cached values to disk."""
#       self.commit_item(name, keep_parsed=True)

EDIT: Looking further in the code in calibre, it seems to me that committing all the dirty items while threads are running is very dangerous. The calibre ebook container frequently sets an item as dirty before doing its work. Some other thread could then attempt to commit changes that are concurrently being made.

Personally, I would choose either

• to remove the threading, or
• ensure that:
  • no threaded item calls flush_cache()
  • the thread controller (__run_async()) runs all the threads, then after the "with" exits calls flush_cache().

The goal is to guarantee that no thread can affect the processing of another thread via the dirty settings.

My first choice would be to remove the threading to see if the problems go away. If they do then look at how threading can be put back safely. If they don't then back to the drawing board.

[EDIT 2]@davidfor: see the PM I sent you.

[fogice]

Chaley, thanks so much for researching my issue!

[BiliBug]

I installed the new beta version (3.5.7), have the Count Pages plugin working correctly with custom columns, but I don't see the options in the Kobo Extended Driver configuration to sync these fields, as per post 2717. Any ideas? Calibre version is 5.40, running on macOS 12.4.

[chaley]

Originally Posted by BiliBug

I installed the new beta version (3.5.7), have the Count Pages plugin working correctly with custom columns, but I don't see the options in the Kobo Extended Driver configuration to sync these fields, as per post 2717. Any ideas? Calibre version is 5.40, running on macOS 12.4.

Those changes will be in the next calibre release. I see them because I always run from up to date source.

[BiliBug]

Originally Posted by chaley

Those changes will be in the next calibre release. I see them because I always fun from up to date source.

Ah. Got it. Thanks!

[fogice]

On the positive side, I've checked about 30 books, and I don't see any missing chapters in them. Previously, about 1/3 of my books were showing missing chapters.

[giddie]

Originally Posted by davidfor

I don't know about "inspired", but, the choice was a template or some way to specify a column name. Using a template just seemed to be the best for flexibility. And any amusement I get from @ownedbycats attempts to break the template system are just a bonus.

I'm excited about the added flexibility here. Any chance you could apply the same approach to collection management to address the issues raised the post linked below?

https://www.mobileread.com/forums/sh...postcount=2678