Libra 2 and Calibre - all sorts of strange behaviour

[w00dent0p]

I'm a bit stuck at the moment.

I have a 3-month old Libra 2 that has been syncing with Calibre without any problems. Lately it's started misbehaving. I'm getting reports of upload failures and sometimes database corruption from Calibre, even though the Libra seems to be happy. I recovered by doing a factory reset (and on a later occasion a hard reset) but am still having problems. Couple of logs below, in case it helps ...

calibre, version 5.38.0
ERROR: Error: Error communicating with device

Set changed size during iteration

Traceback (most recent call last):
File "calibre\gui2\device.py", line 87, in run
File "calibre\gui2\device.py", line 592, in _upload_books
File "calibre_plugins.kobotouch_extended.device.driver" , line 416, in upload_books
File "calibre\devices\kobo\driver.py", line 2171, in upload_books
File "calibre_plugins.kobotouch_extended.device.driver" , line 211, in _modify_epub
File "calibre_plugins.kobotouch_extended.container" , line 119, in __init__
File "calibre_plugins.kobotouch_extended.container" , line 375, in __run_async_over_content
File "calibre_plugins.kobotouch_extended.container" , line 368, in __run_async
File "concurrent\futures\_base.py", line 432, in result
File "concurrent\futures\_base.py", line 388, in __get_result
File "concurrent\futures\thread.py", line 57, in run
File "calibre_plugins.kobotouch_extended.container" , line 310, in forced_cleanup
File "calibre_plugins.kobotouch_extended.container" , line 169, in flush_cache
File "calibre_plugins.kobotouch_extended.container" , line 169, in <listcomp>
RuntimeError: Set changed size during iteration


calibre, version 5.38.0
ERROR: Error: Error communicating with device

CorruptError: database disk image is malformed

Traceback (most recent call last):
File "calibre\gui2\device.py", line 87, in run
File "calibre\gui2\device.py", line 512, in _books
File "calibre\devices\kobo\driver.py", line 1987, in books
apsw.CorruptError: CorruptError: database disk image is malformed


Any thoughts? Thanks

[JSWolf]

Are you running firmware version 4.31? If so, then that's the reason for these errors and the database corruption. There are two solutions.

1. Install firmware 4.30 and install the patch to not update the firmware via WiFi

2. Install firmware 4.30 and do not sync. Keep WiFi off.

The problem is that 4.31 is rather buggy and is known to corrupt the database. I suggest you read the thread on the 4.31 firmware.

Firmware 4.31.19086

[w00dent0p]

Hi, yes, 4.31. Many thanks.

Good to know that it's the FW at fault, not the HW.

Does that thread explain how to revert to 4.30 and install the patch? Would the patch prevent updates to future releases?

[w00dent0p]

Second thoughts, since I don't need to sync anything regularly, perhaps I can just do another factory reset, then upload a small number of books that I'm actually reading, and wait for 4.32.

Thanks for your help !!

[theducks]

Originally Posted by w00dent0p

Hi, yes, 4.31. Many thanks.

Good to know that it's the FW at fault, not the HW.

Does that thread explain how to revert to 4.30 and install the patch? Would the patch prevent updates to future releases?

This is a bit of a juggle if your DB is damaged. You need to fix that first. THEN disable WiFi on the device.

The FW is updated/backdated via USB

Reverting is as simple as installing the down rev FW from a PC download.
simply select the other versions next to your model name: select the version and download that.
per the instructions on the FW page, put the contents of the ZIP into the proper place.

[DNSB]

If you are interested in reverting to the 4.30 firmware, the best source is geek1011's Kobo Firmware Downloads page.

[JSWolf]

Originally Posted by w00dent0p

Second thoughts, since I don't need to sync anything regularly, perhaps I can just do another factory reset, then upload a small number of books that I'm actually reading, and wait for 4.32.

Thanks for your help !!

Just install 4.30. You can do so manually. It's very easy.

1. Download the 4.30 firmware using this link. It's direct from Kobo (this firmware file is just for the Libra 2).

2. Plug your Kobo into a USB port of your computer. UnZIP the contents of the firmware file to the .kobo directory on your Kobo.

3. Eject your Kobo from the computer

4. Wait for the firmware to be installed

5. If the WiFi is on, turn it off.

6. Done!

[w00dent0p]

Jon, David, thanks for the advice.

I was going to try and revert to 4.30, but this morning I did a factory reset and just uploaded the handful of books I'm reading, so I don't think the DB is corrupted any more. Or at least the Libra 2 is now behaving as an e-reader, not a confused brick. So I'm thinking this will keep me going for a few weeks. And by keeping Wifi on, I should get to hear if/when there's a 4.32 available.

Ever the optimist ...

[JSWolf]

Originally Posted by w00dent0p

Jon, David, thanks for the advice.

I was going to try and revert to 4.30, but this morning I did a factory reset and just uploaded the handful of books I'm reading, so I don't think the DB is corrupted any more. Or at least the Libra 2 is now behaving as an e-reader, not a confused brick. So I'm thinking this will keep me going for a few weeks. And by keeping Wifi on, I should get to hear if/when there's a 4.32 available.

Ever the optimist ...

Go to 4.30 using the directions I posted. Also, do not keep WiFi on and do not sync and no Pocket. Otherwise you will get 4.31 back and all the hassles.

[davidfor]

Originally Posted by JSWolf

Are you running firmware version 4.31? If so, then that's the reason for these errors and the database corruption. There are two solutions.

1. Install firmware 4.30 and install the patch to not update the firmware via WiFi

2. Install firmware 4.30 and do not sync. Keep WiFi off.

The problem is that 4.31 is rather buggy and is known to corrupt the database. I suggest you read the thread on the 4.31 firmware.

Firmware 4.31.19086

Jon: Please stop answering any question about anything. @w00dent0p is having a problem with sending a book to the device using the KoboTouchExtended driver. It is clear from the error message where this is happening. It is COMPLETELY and UTTERLY unrelated to the firmware running on the device. And if you would bother to actually read the post, you would see this. Or at least you would see that you have absolutely no idea about what is going on and would hence not make a completely and utterly useless reply to the post. I don't know about others here, but you useless responses are pissing me off.

Edit:
And yes, I should read the full message as I was only concerned with the failure in the transformation of the book to the kepub. The database problem is separate and I stand by the idea that your basic type of response. And honestly, not even commenting on the first error demonstrates thos.


@w00dent0p: Did you get the first error when you tried sending the book again? That is related to the book and possibly how the extended driver is coded.

[w00dent0p]

Hi David, unfortunately I don't know which book (from ~600) caused the first error, or even if it was caused by a particular book. After doing a factory reset (but still running 4.31) I uploaded just 5 books (which I've uploaded before and so I think they're fine) and didn't get any errors. Libra 2 is behaving ok with these 5 books.

I'm not planning to mess around with reverting firmware or installing patches, but if you're sure the kobotouch_extended error relates to a book (and could not be caused by a corrupted database), I can try uploading some recent additions to my library, to try and narrow it down.

[davidfor]

Originally Posted by w00dent0p

Hi David, unfortunately I don't know which book (from ~600) caused the first error, or even if it was caused by a particular book. After doing a factory reset (but still running 4.31) I uploaded just 5 books (which I've uploaded before and so I think they're fine) and didn't get any errors. Libra 2 is behaving ok with these 5 books.

I'm not planning to mess around with reverting firmware or installing patches, but if you're sure the kobotouch_extended error relates to a book (and could not be caused by a corrupted database), I can try uploading some recent additions to my library, to try and narrow it down.

The error is some code doing cleanup of the book code. It happens before attempting to send a particular book to the device. It could be related to sending a lot of books in one go, it could be something unusual in a particular book.

Are you using a 32bit or 64bit build of calibre? It could be a memory issue if you are using a 32bit version and very big book.

[w00dent0p]

I'm using 64-bit calibre.

I might have been uploading a lot of books at the time, to restore everything from the first time I did a factory reset; 600 books maybe. I've since learned that 50 or so is more reasonable.

Ok, I suggest we park this. Then once 4.31 is fixed, I'll have another go at uploading everything and will then identify any culprit books. (Although I think it's unlikely - as a rule of thumb I do edit all books, repair any issues that can be auto-repaired, then run polish.)

Thanks again.
Nigel

[w00dent0p]

Just a quick note to say that I'm now back to normal behaviour. Updated to vanilla 4.32 over wifi, then factory reset my Libra 2 to make sure the database is clean, then uploaded my entire library (about 40 books at a time). No issues seen.

Jon, David, thanks very much for the great support.

[chaley]

@davidfor: I think it worth noting that this error:

Originally Posted by w00dent0p

calibre, version 5.38.0
ERROR: Error: Error communicating with device

Set changed size during iteration

Traceback (most recent call last):
File "calibre\gui2\device.py", line 87, in run
File "calibre\gui2\device.py", line 592, in _upload_books
File "calibre_plugins.kobotouch_extended.device.driver" , line 416, in upload_books
File "calibre\devices\kobo\driver.py", line 2171, in upload_books
File "calibre_plugins.kobotouch_extended.device.driver" , line 211, in _modify_epub
File "calibre_plugins.kobotouch_extended.container" , line 119, in __init__
File "calibre_plugins.kobotouch_extended.container" , line 375, in __run_async_over_content
File "calibre_plugins.kobotouch_extended.container" , line 368, in __run_async
File "concurrent\futures\_base.py", line 432, in result
File "concurrent\futures\_base.py", line 388, in __get_result
File "concurrent\futures\thread.py", line 57, in run
File "calibre_plugins.kobotouch_extended.container" , line 310, in forced_cleanup
File "calibre_plugins.kobotouch_extended.container" , line 169, in flush_cache
File "calibre_plugins.kobotouch_extended.container" , line 169, in <listcomp>
RuntimeError: Set changed size during iteration

is almost certainly caused by the problem we fixed this week, where multiple threads call flush_task(). It would happen if thread A starts and adds to the dirty set while thread B is flush_cache iterating over the dirty set with a comprehension. The set it is complaining about is self.dirty.