[Device Plugin] Kobo Touch Extended Driver

[davidfor]

Originally Posted by chaley

I looked at the crash reports. In both cases it crashed in libxml2 with a SEGV. That raises the question: are lxml and libxml2 thread safe? This document says libxml2 is but with caveats. I don't know what versions are being used nor can I determine if xmlInitParser() is called as directed.

I also noticed that both crashes happened when serializing the tree for output. Some notes:

• This happens when container.flush_cache() is called. flush_cache() is called from within a thread pool worker. That means that one thread pool is creating a second thread pool. I can easily imagine that this isn't a good thing.
• Because multiple threads can call flush_cache(), more than one thread can call commit_item() on the same tree. Does this write the file twice?
• Because multiple threads can call flush_cache on the same dirtied set, trees can be committed at the same time. I don't know what happens if the two threads attempt to write the same file.

Two suggestions:

• Change flush_cache() to do the work directly without creating threads. This avoids nesting thread pools.
• Put generating the set of dirtied trees and calling commit_item() into a locked with. This avoids possibly committing the same item more than once at the same time.

Something like this:

# This goes in __init__
        super(KEPubContainer, self).__init__(epub_path, log, *args, **kwargs)
        self.flush_lock = threading.Lock() # New code
        self.log = log
        self.log.debug(f"Creating KePub Container for ePub at {epub_path}")

# This replaces flush_cache()
    def flush_cache(self) -> None:
        with self.flush_lock.acquire():
            for name in list(self.dirtied):
                self.commit_item(name, keep_parsed=True)

#    def __flush_cache_impl(self, name: str) -> None:
#       """Flush the cache, writing all cached values to disk."""
#       self.commit_item(name, keep_parsed=True)

EDIT: Looking further in the code in calibre, it seems to me that committing all the dirty items while threads are running is very dangerous. The calibre ebook container frequently sets an item as dirty before doing its work. Some other thread could then attempt to commit changes that are concurrently being made.

Personally, I would choose either

• to remove the threading, or
• ensure that:
  • no threaded item calls flush_cache()
  • the thread controller (__run_async()) runs all the threads, then after the "with" exits calls flush_cache().

The goal is to guarantee that no thread can affect the processing of another thread via the dirty settings.

My first choice would be to remove the threading to see if the problems go away. If they do then look at how threading can be put back safely. If they don't then back to the drawing board.

[EDIT 2]@davidfor: see the PM I sent you.

Firstly, I'm not ignoring anyone, between work and being away, I haven't had the time or energy to look at this. Or the question of how to use templates for collections.

Having made my excuses...

I do agree with your analysis. The way the flush_cache was implemented and the commits were called did look to be prone to timing issues with the threading. I have attached a version of the driver with the your updated "container.py" in it. It is working with no issues here under Windows. Hopefully it solve the problems with Mac.

[davidfor]

Originally Posted by giddie

I'm excited about the added flexibility here. Any chance you could apply the same approach to collection management to address the issues raised the post linked below?

https://www.mobileread.com/forums/sh...postcount=2678

It is my intention to add template based collections to the base KoboTouch driver. @chaley has made some suggestions in https://www.mobileread.com/forums/sh...d.php?t=346321. And there was more discussion in another thread. I'm hoping to look at it over the weekend.

[Sirtel]

I updated Calibre to 5.41 and filled in the page and word count templates in the Extended Driver options as per your example in the new firmware thread (I use the same names for my page and word columns). But no page or word count is sent to my Kobo, either with new books or as part of the updated metadata for books already on the device.

These are the templates I entered:
Words: program:$$#words
Pages: program:$$#pages

What could be wrong? My knowledge about template language is close to zero.

[chaley]

I too updated the firmware and tried to use the stats stuff. Nothing happens. Turns out the problem is that the driver wants a db_version >= 169. My db_version is 168.

Do I need to do something to upgrade the database, or is 169 the wrong number?

From the log:

Firmware version: 4.32.19501
DEBUG:    0.7 Database Version:  168

[Terisa de morgan]

@davidfor, would it be posible to include in the driver the option for removing series with less than a configurable number in them? Currently, I execute the script every time I connect the ereader, but it's a PIA.

[ownedbycats]

Originally Posted by Terisa de morgan

@davidfor, would it be posible to include in the driver the option for removing collections with less than a configurable number in them? Currently, I execute the script every time I connect the ereader, but it's a PIA.

I'm not sure what you're doing, but you can prevent specific colls from appearing on device by putting them in [Square Brackets.] I did this with '[Send to Device]' and '[Kobo Store]' before switching them over to a different column.

[Terisa de morgan]

Originally Posted by ownedbycats

I'm not sure what you're doing, but you can prevent specific colls from appearing on device by putting them in [Square Brackets.] I did this with '[Send to Device]' and '[Kobo Store]' before switching them over to a different column.

What I'm doing is writing the wrong entity . I want to clean series, not collections. Series with a book in them only are clutter in the list.

[ownedbycats]

Oops. Well, I hope that tip will be useful for someone else.

[Terisa de morgan]

Originally Posted by ownedbycats

Oops. Well, I hope that tip will be useful for someone else.

The oops is mine , your answer has shown that I had written the wrong entity, so thank you

[chaley]

Originally Posted by davidfor

It is my intention to add template based collections to the base KoboTouch driver. @chaley has made some suggestions in https://www.mobileread.com/forums/sh...d.php?t=346321. And there was more discussion in another thread. I'm hoping to look at it over the weekend.

Unfortunately I don't think that will solve the OP's problem without very serious surgery to the collections stuff. My understanding is that they want auto management for one device and manual management for the other. That can't be accomplished by playing with collection columns or items. It could be done by adding an override for the management style, set using a template that looks at which device is connected.

[fogice]

Originally Posted by davidfor

I have attached a version of the driver with the your updated "container.py" in it. It is working with no issues here under Windows. Hopefully it solve the problems with Mac.

I have just tested it on Mac (12.3.1), and was able to move about 1,000 books without a crash! The last batch was 542 in one go. Thanks so much for the new version!

[chaley]

Originally Posted by fogice

I have just tested it on Mac (12.3.1), and was able to move about 1,000 books without a crash! The last batch was 542 in one go. Thanks so much for the new version!

And we hope no missing chapters.

Did you see a difference in performance?

[fogice]

Originally Posted by chaley

And we hope no missing chapters.

Did you see a difference in performance?

I couldn't say, I have nothing to compare it to. But here's the results from what I did today.

According to the Jobs window:
Books: Time
524: 16m 31s
300: 10m 9s
102: 3m 10s
40: 1m 16s
20: 35s
7: 12s

And even if it took twice as long, I don't care. I usually upload while I'm doing something else, and let it sit in the background.

[davidfor]

Originally Posted by chaley

I too updated the firmware and tried to use the stats stuff. Nothing happens. Turns out the problem is that the driver wants a db_version >= 169. My db_version is 168.

Do I need to do something to upgrade the database, or is 169 the wrong number?

From the log:

Firmware version: 4.32.19501
DEBUG:    0.7 Database Version:  168

Bugger. It looks like Kobo removed a change in one of the last betas. That means that my database ended up on DB version 169. I check the database for changes as the beta's come out, but, haven't had something like this happen before.

I'll check in the change for the DB version in the driver, but, that won't be out until the next release. You can change the DB version in the database to make it work until then. That will be work, but, will need to be undone before the next firmware release.

[compurandom]

Originally Posted by davidfor

Bugger.

Since it doesn't work anyway, I can't test it without mangling things.
But I would think something like this would be more sensible? comments?

Words: program:floor($$#words)
Pages: program:floor($$#pages)
Lower: program:floor($$#words/15000)
Upper: program:floor($$#words/12000)

Was the range value suppose to be minutes or hours?