Glo Phantom disconnect - solved ?

[BobC]

Some time since I experienced problems with my Glo disconnecting from my PC -

https://www.mobileread.com/forums/sho...&postcount=214

I have now done a lot more delving into what is happening and it is clear that the problem was triggered when Calibre was sending Metadata to the device, either after getting the list of books on connection or after books had been uploaded to the Glo.

Further investigation made it appear that this happened at the end of the Metadata upload following an attempt to delete empty shelves. This interested me as I didn't think I had any shelves eligible for deletion.

I realised that some Kobo books I had downloaded might be sat on "invisible" shelves (I had problems with duplicated shelves that couldn't be deleted in the past) such as the Glo's "Im_Reading".

I solved the problem (fingers crossed) by deleting all Kobo books from the device (they were redundant anyway) and synching the Glo. After this the phantom disconnects have not re-occurred.

It looks like it's possible to get the Glo's database into a condition where some Metadata update by Calibre will cause it to trigger the disconnect.

I know a few other folk (not many) have had a similar problem and they might want to see if any of the above applies to them.

BobC

[jackastor]

Originally Posted by BobC

Some time since I experienced problems with my Glo disconnecting from my PC -

https://www.mobileread.com/forums/sho...&postcount=214

I have now done a lot more delving into what is happening and it is clear that the problem was triggered when Calibre was sending Metadata to the device, either after getting the list of books on connection or after books had been uploaded to the Glo.

Further investigation made it appear that this happened at the end of the Metadata upload following an attempt to delete empty shelves. This interested me as I didn't think I had any shelves eligible for deletion.

I realised that some Kobo books I had downloaded might be sat on "invisible" shelves (I had problems with duplicated shelves that couldn't be deleted in the past) such as the Glo's "Im_Reading".

I solved the problem (fingers crossed) by deleting all Kobo books from the device (they were redundant anyway) and synching the Glo. After this the phantom disconnects have not re-occurred.

It looks like it's possible to get the Glo's database into a condition where some Metadata update by Calibre will cause it to trigger the disconnect.

I know a few other folk (not many) have had a similar problem and they might want to see if any of the above applies to them.

BobC

I had this happen quite a bit in last week. The problem was a faulty Micro usb cable. If it happens again for you, you may want to consider replacing the the usb cable with a brand new one.

regards

JAck

[BobC]

Originally Posted by jackastor

I had this happen quite a bit in last week. The problem was a faulty Micro usb cable. If it happens again for you, you may want to consider replacing the the usb cable with a brand new one.

regards

JAck

Been down that route already, thanks

BobC

[davidfor]

Originally Posted by BobC

Some time since I experienced problems with my Glo disconnecting from my PC -

https://www.mobileread.com/forums/sho...&postcount=214

I have now done a lot more delving into what is happening and it is clear that the problem was triggered when Calibre was sending Metadata to the device, either after getting the list of books on connection or after books had been uploaded to the Glo.

Further investigation made it appear that this happened at the end of the Metadata upload following an attempt to delete empty shelves. This interested me as I didn't think I had any shelves eligible for deletion.

The code to delete empty shelves is three SQL statements that will either delete the shelves or mark them for deletion at the next sync to Kobo. There are no checks beforehand to see if anything will be deleted. If you uncheck the "delete empty shelves" option in the driver configuration, they don't get run.

I realised that some Kobo books I had downloaded might be sat on "invisible" shelves (I had problems with duplicated shelves that couldn't be deleted in the past) such as the Glo's "Im_Reading".

Where do you see an "Im_Reading" shelf? Are you referring to what is in the calibre device list? If so, that isn't a real shelf. That is simply a name given by the driver to the collection of books that are open on the device. I think the name came from on of the older devices which had it as a way to view books. With the Glo, I think the only way to get this as a shelf on the device is to manually create it.

I solved the problem (fingers crossed) by deleting all Kobo books from the device (they were redundant anyway) and synching the Glo. After this the phantom disconnects have not re-occurred.

It looks like it's possible to get the Glo's database into a condition where some Metadata update by Calibre will cause it to trigger the disconnect.

I know a few other folk (not many) have had a similar problem and they might want to see if any of the above applies to them.

I'm glad it fixed it, but I can't think of why. The only difference I can think of is the total size of the database due to not having these books.

[BobC]

Originally Posted by davidfor

The code to delete empty shelves is three SQL statements that will either delete the shelves or mark them for deletion at the next sync to Kobo. There are no checks beforehand to see if anything will be deleted. If you uncheck the "delete empty shelves" option in the driver configuration, they don't get run.

This is the tail end of the debug log I got when the problem was occurring :

Spoiler:

DEBUG: 128.1 KoboTouch:update_device_database_collections - end for category='Historic'
DEBUG: 128.1 KoboTouch:update_device_database_collections - category='Espionage' books=9
DEBUG: 128.4 KoboTouch:update_device_database_collections - end for category='Espionage'
DEBUG: 128.4 KoboTouch:update_device_database_collections - category='Nordic' books=4
DEBUG: 128.5 KoboTouch:update_device_database_collections - end for category='Nordic'
DEBUG: 128.5 KoboTouch:update_device_database_collections - managing bookshelves and series.
DEBUG: 153.9 KoboTouch:update_device_database_collections - about to clear empty bookshelves
DEBUG: 153.9 KoboTouch:delete_empty_bookshelves - start
DeviceJob: 3 Send metadata to device done, calling callback
unable to open database file

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 85, in run
File "site-packages\calibre\gui2\device.py", line 506, in _sync_booklists
File "calibre_plugins.kobotouch_extended.device.driver" , line 282, in sync_booklists
File "site-packages\calibre\devices\kobo\driver.py", line 888, in sync_booklists
File "site-packages\calibre\devices\kobo\driver.py", line 2242, in update_device_database_collections
File "site-packages\calibre\devices\kobo\driver.py", line 2498, in delete_empty_bookshelves
OperationalError: unable to open database file

DeviceJob: 3 Send metadata to device callback returned
Job: 3 Send metadata to device finished
unable to open database file

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 85, in run
File "site-packages\calibre\gui2\device.py", line 506, in _sync_booklists
File "calibre_plugins.kobotouch_extended.device.driver" , line 282, in sync_booklists
File "site-packages\calibre\devices\kobo\driver.py", line 888, in sync_booklists
File "site-packages\calibre\devices\kobo\driver.py", line 2242, in update_device_database_collections
File "site-packages\calibre\devices\kobo\driver.py", line 2498, in delete_empty_bookshelves
OperationalError: unable to open database file

It appeared from this as though the problem occurred after the "Delete" instructions were sent.

Originally Posted by davidfor

Where do you see an "Im_Reading" shelf? Are you referring to what is in the calibre device list? If so, that isn't a real shelf. ...

Yes, I have only seen "Im_Reading" in the Device View "Collections" column along with all the Shelves. I though I had seen a reference to it in the SQL database but can no longer locate it so perhaps that was just an organic memory glitch.

Originally Posted by davidfor

I'm glad it fixed it, but I can't think of why. The only difference I can think of is the total size of the database due to not having these books.

Overall it appeared that Calibre was doing "something" that, with the Glo's database in perhaps a conflicting state, caused the Glo to disconnect and that the solution was to build a new database without the conflicts. All my books should be sideloaded kepubs but there have been a few downloaded from Kobo (sometimes with the same name as the sideloaded ones) and it seems like purging the Kobo ones and synching is what resolved the issue. Originally I thought the problem was in transferring the books themselves (possibly the size of the books) and that it was the USB Mass Storage Gadget on the Glo that was at fault; I'm now coming round to the way of thinking that it is some form of database conflict that is at the root of the problem. As the problem has survived a number of factory resets it clearly wasn't a "corrupt" database in the usual sense. It may even be to do with some of the Name/Titles in my sideloaded books as I have had issues with multiple authors and punctuation in titles which seems to make it difficult for Calibre to match the books it has uploaded to ones shown as "On Device".


BobC

[14nc310t]

It's always happening to me since 2.6.x

I've tried to use several usb cables but without any change. I thought it was the USB connector but I'm glad I was wrong.

Is there a way to solve it without resetting the db? I would not miss the reading history and the statistics.

[BobC]

Originally Posted by 14nc310t

It's always happening to me since 2.6.x

I've tried to use several usb cables but without any change. I thought it was the USB connector but I'm glad I was wrong.

Is there a way to solve it without resetting the db? I would not miss the reading history and the statistics.

I can only recount my experience - you may of course have a different problem. It took a lot of trial and error for me to solve my problem, the intention of the post was to offer a hint of one possible solution - database conflict -as I was pretty sure I had ruled out the hardware problem that most folk were blaming it on.

As I found out though, doing a factory reset then reloading all the same books via Calibre and Synching from Kobo brought the problem back - it's only after some severe surgery on the books on the device that it seems to have settled down.

BobC

[14nc310t]

Yes,
I know how to interact with the sqlite tables but, with more than 800 books, I don't know where to look and what...

[davidfor]

@14nc310t: I don't think BobC did anything in the database directly. He has removed the books from the device using it's interface. But, if you want to fiddle in the database, ask away. I can probably tell you what to look for.

@BobC: The error in your post is happening while calibre is syncing data at the first connect. Did calibre display an error dialog when this happened? The reason for the question is that the error in the post you linked to is later. It happened on the next interaction between calibre and the device. That either means the error started happening earlier, or an error was not being reported.

[BobC]

Originally Posted by davidfor

@14nc310t: I don't think BobC did anything in the database directly. He has removed the books from the device using it's interface. But, if you want to fiddle in the database, ask away. I can probably tell you what to look for.

Your assessment is correct - I haven't manipulated the database directly.

Originally Posted by davidfor

@BobC: The error in your post is happening while calibre is syncing data at the first connect. Did calibre display an error dialog when this happened? The reason for the question is that the error in the post you linked to is later. It happened on the next interaction between calibre and the device. That either means the error started happening earlier, or an error was not being reported.

I'm pretty sure that on the occasion the log refers to the error displayed was something like "Error Communicating with Device". The error has occurred a different stages from time to time, both on initial connect and on later interaction. It is only with 2.8.1 that I started looking more closely at the logs and noticed that the error seems to occur during/after the "Update Metadata" phase. I can still get it to happen, although not every time, if I delete a book on the device from Device View. To date though an eject and re-connect has always got things going again.

Do you want the full debug log ? - I trimmed it down rather than posting the whole 10Kb.

I haven't been able to interest Kobo in the problem because it only seems to happen when the database is being manipulated and that isn't a supported activity (perhaps the desktop app does it but I don't use that as it's no good for loading the books I have). Simple book/file copying using File Manager doesn't trigger the issue.

BobC

[davidfor]

Originally Posted by BobC

I'm pretty sure that on the occasion the log refers to the error displayed was something like "Error Communicating with Device". The error has occurred a different stages from time to time, both on initial connect and on later interaction. It is only with 2.8.1 that I started looking more closely at the logs and noticed that the error seems to occur during/after the "Update Metadata" phase. I can still get it to happen, although not every time, if I delete a book on the device from Device View. To date though an eject and re-connect has always got things going again.

The fact the error happens during the "Update Metadata" phase makes sense. That is when updates are made to the database. The earlier phase is read only. Sending a book writes to the drive and updates the database.

Do you want the full debug log ? - I trimmed it down rather than posting the whole 10Kb.

Yes please. Post it or send it in a PM.

I haven't been able to interest Kobo in the problem because it only seems to happen when the database is being manipulated and that isn't a supported activity (perhaps the desktop app does it but I don't use that as it's no good for loading the books I have). Simple book/file copying using File Manager doesn't trigger the issue.

I'm not surprised. Kobo accepts that calibre exists. But, they don't officially support using it. If we could reproduce the problem with the desktop app, that would be a different matter. The desktop app does update the database, but probably not as many as the calibre driver can. That might be the issue.

[BobC]

Originally Posted by davidfor

I'm not surprised. Kobo accepts that calibre exists. But, they officially support using it.

ITYM they DON'T officially support using it.

(Just in case anyone else reading this takes this as gospel)

BobC

[davidfor]

Originally Posted by BobC

ITYM they DON'T officially support using it.

(Just in case anyone else reading this takes this as gospel)

Thanks for picking that up. I fixed it in the post.

[taming]

Interestingly, one version of an earlier device's user guide had many mentions of calibre and how to use it in conjunction with Kobo devices. It wasn't up for very long.