[Device Plugin] Kobo Touch Extended Driver

[sheldon_es]

Thanks. I'll try to observe the tenmp directory to see what's happening to the files. It could be the antivirus, but I don't have a temp cleaner at a machine level, and I already tried to change the directory to a normal one and the error was still there.

[Boilerplate4U]

Does anyone know what is causing the errors below? I get them every time I transfer books to my Kobo Clara HD. However, everything seems to work fine afterwards except that some metadata like the book series is not updated on the device.

caliber, version 5.33.2
ERROR: Error: Error communicating with device

CorruptError: database disk image is malformed

Traceback (most recent call last):
   File "caliber / gui2 / device.py", line 87, in run
   File "caliber / gui2 / device.py", line 543, in _sync_booklists
   File "calibre_plugins.kobotouch_extended.device.driver", line 459, in sync_booklists
     all_nulls = __rows_needing_imageid ()
   File "caliber_plugins.kobotouch_extended.device.driver", line 454, in __rows_needing_imageid
     c.execute (select_query, (self.content_types ["main"],))
apsw.CorruptError: CorruptError: database disk image is malformed

[DNSB]

Originally Posted by Boilerplate4U

Does anyone know what is causing the errors below? I get them every time I transfer books to my Kobo Clara HD. However, everything seems to work fine afterwards except that some metadata like the book series is not updated on the device.

Spoiler:

caliber, version 5.33.2
ERROR: Error: Error communicating with device
CorruptError: database disk image is malformed

Traceback (most recent call last):
   File "caliber / gui2 / device.py", line 87, in run
   File "caliber / gui2 / device.py", line 543, in _sync_booklists
   File "calibre_plugins.kobotouch_extended.device.driver", line 459, in sync_booklists
     all_nulls = __rows_needing_imageid ()
   File "caliber_plugins.kobotouch_extended.device.driver", line 454, in __rows_needing_imageid
     c.execute (select_query, (self.content_types ["main"],))
apsw.CorruptError: CorruptError: database disk image is malformed

Basically, it's telling you that your database on the devices has errors. If you have @davidfor's Kobo Utilities plugin installed, the check database will likely tell you the same information that your database is corrupted. Sometime running the compress database item will correct some minor errors. A quick search on Mobileread ( I used Google with kobo database corrupt site:mobileread.com as the search string and limited to past year) will give you quite a few bits of advice.

Though generally, I find the signing out and signing back in as the easiest way to generate a new database. This does lose your annotations, statistics, etc.

[ownedbycats]

I got my Kobo Libra H2O. The first few times connecting & sending books went fine. Then I plugged it in again and got this:

'utf-8' codec can't decode byte 0x83 in position 637: invalid start byte

Traceback (most recent call last):
  File "calibre\gui2\device.py", line 87, in run
  File "calibre\gui2\device.py", line 512, in _books
  File "calibre\devices\kobo\driver.py", line 1988, in books
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x83 in position 637: invalid start byte

Switching to the regular driver works, but I prefer kepubs. I checked the database and it's good. Any idea what I'm doing wrong?

EDIT: Attached debug log.

ANOTHER EDIT: I noticed the KoboTouchExtended settings was a lot of null characters in the debug log. So just to see what happened i moved the device_drivers_KOBOTOUCHEXTENDED.py.json file out and re-created all the settings. Now it's working. I've attached the old json (renamed to txt because of file upload limitations) for investigation.

[dhampny]

I had posted this in the Kobo forum but was advised it might be more appropriately posted here.

Please excuse me if this has been asked and answered already. I've searched but haven't been able to find an answer to what I see happening when I use the kobo touch extended driver.

My understanding is that the driver should create a kepub file only on my Libra 2 or Forma when I send an epub to it. However, in my case, the driver also creates a kepub format file in my Calibre library. So I end up with both epub and kepub format files of the book. It does correctly convert the epub format of my ebook and send it to the Kobo as a kepub (kobo epub) ebook.

What am I doing wrong?

Thanks.

[theducks]

I suspect you set Preference:Behavior: Preferred output to KEPUB (and the normal device driver converts the file FIRST, since that does not exist)
Set it to EPUB and let the Utilities do the on the fly work

[dhampny]

Originally Posted by theducks

I suspect you set Preference:Behavior: Preferred output to KEPUB (and the normal device driver converts the file FIRST, since that does not exist)
Set it to EPUB and let the Utilities do the on the fly work

Thanks. I already have that set to EPUB. Also, I do have the Kobo Touch plugin disabled and I have Enable Extended Kobo Features selected in the KoboTouchExtended driver plugin.

[theducks]

Originally Posted by dhampny

Thanks. I already have that set to EPUB. Also, I do have the Kobo Touch plugin disabled and I have Enable Extended Kobo Features selected in the KoboTouchExtended driver plugin.

I use the Extended driver all the time and it has never put a KEPUB file into the Library, that is why I thought the Preference was the cause

[dhampny]

I'll explain what is happening when I send a kepub to my Kobo to make sure my expectations of what should be happening is correct.

With my Kobo connected to Calibre I choose an epub to 'Send to device'. In a popup window Calibre asks me to choose the format to send and shows I have '0 of 1 kepub'. I choose that format, Calibre does the conversion and sends to my Kobo. The kepub format then appears under Formats in the Book Details and it also appears as a file when I look under the Author folder/Book subfolder in my Calibre Library.

My expectation from everything I've read about the kobo touch extended driver is that I would not see the kepub format appear in Book Details nor within the Author/Book subfolder. Am I correct in this regard?

[jackie_w]

Originally Posted by dhampny

With my Kobo connected to Calibre I choose an epub to 'Send to device'. In a popup window Calibre asks me to choose the format to send and shows I have '0 of 1 kepub'. I choose that format...

I think this is your problem. Because you're choosing KEPUB (which doesn't currently exist) the calibre driver decides it had better create a KEPUB, in your library, so that it can be sent. Pick the EPUB format (which does currently exist) instead. The KoboTouchExtended driver will handle the on-the-fly creation of the kepub, without storing it in the library.

[dhampny]

Originally Posted by jackie_w

I think this is your problem. Because you're choosing KEPUB (which doesn't currently exist) the calibre driver decides it had better create a KEPUB, in your library, so that it can be sent. Pick the EPUB format (which does currently exist) instead. The KoboTouchExtended driver will handle the on-the-fly creation of the kepub, without storing it in the library.

That was it! Thank you. And thank you again to theducks. Karma to you both. (I've already added some to theducks.)

[davidfor]

Originally Posted by dhampny

That was it! Thank you. And thank you again to theducks. Karma to you both. (I've already added some to theducks.)

There is still something wrong with what you have said you are doing and how it is configured.

Firstly, if you click the "Send to device" button, or use the menu and choose "Send to main memory" from the menu, it doesn't prompt for the format. When doing this, the calibre finds the format to send based on the order of the formats in the driver configuration. The format for the book that is the first in the enabled formats will be sent. If you do not have a suitable format, it will do a conversion. And it will convert to the top format in that list. That means, that if you have the default option, which is "kepup, epub, PDF, txt, cbz, cbr", and you have an epub, it will be sent. But, if your only format is a MOBI, it will be converted to kepub, if you have the Kepub Output plugin installed. Otherwise it will be converted to epub. After the conversion, the book will be sent.

The dialog you describe sounds like what you get if you choose the "Send specific format to" option. In this case, the dialog shows all formats that are configured for the device (the list above). And it shows which formats actually exist in the books you have selected. For kepubs, you were seeing "0 of 1 kepub" which means that you had selected 1 book and it didn't have a kepub format. When you selected this and pressed OK, you were telling calibre to convert the book to kepub and then send that to the device. Calibre will prompt to tell you it is doing this, but, that can be turned off.

If you are using the KoboTouchExtended driver, my recommendation is to use the simple "Send to device" and leave the driver configuration for formats to the default. This will mean the driver will transform epubs to kepubs when sending them to the device. But, if you do have a kepub in the library, it will be sent. There is generally no reason to have kepubs in the library, but, if you are resending books frequently, or to multiple devices, is faster to convert once and then send the kepub.

[davidfor]

Originally Posted by ownedbycats

I got my Kobo Libra H2O. The first few times connecting & sending books went fine. Then I plugged it in again and got this:

'utf-8' codec can't decode byte 0x83 in position 637: invalid start byte

Traceback (most recent call last):
  File "calibre\gui2\device.py", line 87, in run
  File "calibre\gui2\device.py", line 512, in _books
  File "calibre\devices\kobo\driver.py", line 1988, in books
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x83 in position 637: invalid start byte

Switching to the regular driver works, but I prefer kepubs. I checked the database and it's good. Any idea what I'm doing wrong?

EDIT: Attached debug log.

ANOTHER EDIT: I noticed the KoboTouchExtended settings was a lot of null characters in the debug log. So just to see what happened i moved the device_drivers_KOBOTOUCHEXTENDED.py.json file out and re-created all the settings. Now it's working. I've attached the old json (renamed to txt because of file upload limitations) for investigation.

All I can say to that is that it is weird. I can see the strangeness in the logs, but, I can't see anything in configuration file that would be causing this. The other times I have seen this problem, it seemed to be an unusual character in one of the fields pulled from the database. But, I've never worked out how it got there or exactly what it was.

[ownedbycats]

Maybe the encoding of the json went weird? I did some searching and found this:

The character frequently appears in Japanese mojibake. The lead byte 0x83 appears before a Katakana character in Shift JIS, but if interpreted as Windows-1252 encoding, it becomes the ƒ character.

[compurandom]

I'm still having problems with my kobo refusing to sync.
I think when calibre side loads news articles to the kobo, subsequent attempts to sync shelf info for those books fails. (At least, I think it is news articles.)

I've found running the following sqlite3 queries before ejecting fixes and/or prevents the issue.

update shelf set _issynced='true' where _issynced='false';
update ShelfContent set _issynced='true' where _issynced='false';

I don't create new shelves that often, so typically the first line doesn't do anything.

Would it be possible to add an option to this plugin to do something similar so that side loaded books have _issynced set to true when they are newly added?

Maybe a filter so that only certain shelves are marked to sync? (I do add kobo and overdrive books to specific shelves that it would be OK to sync, but probably fine if they are not synced, so I don't care either way on that.)

Since I've been doing this by hand, my sync issues have gone away, at least until I forget and have to reconnect to do it.

Let me know if you'd like me to experiment with this and have suggestions on how to narrow down the problem's exact trigger.

Note also, since this fixes sync, my kobo isn't getting stuck in a sync loop anymore and battery life is better!