calibre "On Device" column goes blank for kobo

[sallyg]

I've been working on loading books onto a new kobo aura from calibre. After I re-mounted the kobo, calibre's "On Device" column changed to empty for every book, sent or not.

In this state, I sent a new book to the kobo, and it arrived, but calibre did not set the "On Device" field for the book. Eventually calibre gave the "database disk image is malformed" popup. I didn't see any other indications of trouble while I was sending the books. I sent about 15 books at a time to the device, but did not disconnect the kobo between every set.

On the kobo side, it lists the books, but the collections are goofed up. For example there are empty collections for some series even though the books are present.

This happened once before. I deleted the book files on the kobo, reset, and started over. Thankfully, calibre held onto its list of added books.

My setup:
The calibre and kobo fw versions are less than a month old: mac 10.10.5; calibre 2.50.1; kobo aura fw 3.19.5761
All books are epubs. I have about 170 books in calibre. Added kobo Utilities plugin 2.1.0. I don't know what kobo device driver is being utilized, but I customized koboTouch and something produces kobo collections based on series. I believe I am always disconnecting the kobo properly.

I started calibre in debug mode after getting the "database malformed" message. Output is shown below. I ran the kobo db query shown at the end of the debug output from the command line. It worked fine. Is the malformed db Kobo's or calibre's?

I would very much like to have calibre coordinate with the kobo. That is, have the "on device" column work reliably, and secondarily have the kobo collections work. I don't mind coddling the loading process in order to have success or to help pin-point the problem. I'm already thrilled my books are organized with so much more info using calibre and the kobo. With the options I could access on my old color Nook with a bad screen I could only look up epubs by file name.

Any advice on a good way to start over and to transfer books to the kobo properly? Should I step back in kobo fw? I'm glad to provide any other information, and happy to accept any hints about things to try.

Thank you.

Spoiler:

calibre debug output:

Job: 6 Get device information started
DeviceJob: 6 Get device information done, calling callback
DeviceJob: 6 Get device information callback returned
Job: 7 Set library information started
DeviceJob: 7 Set library information done, calling callback
DeviceJob: 7 Set library information callback returned
Job: 8 Get list of books on device started
DEBUG: 4552.9 KoboTouch:books - oncard='None'
DEBUG: 4552.9 KoboTouch:books - oncard='None', prefix='/Volumes/KOBOeReader/'
DEBUG: 4552.9 Kobo device: Kobo Aura
DEBUG: 4552.9 Version of driver: (2, 1, 11) Has kepubs: True
DEBUG: 4552.9 Version of firmware: (3, 19, 5761) Has kepubs: True
DEBUG: 4552.9 Firmware supports cover image tree: True
Job: 6 Get device information finished
No details available.
Job: 7 Set library information finished
No details available.
DEBUG: 4553.0 KoboTouch:books - opts.extra_customization= [u'#kobocollection,series', True, True, True, True, True, False, False, False, True, False, False, u'']
DEBUG: 4553.0 KoboTouch:books - prefs['manage_device_metadata']= manual
DEBUG: 4553.0 KoboTouch:books - set_debugging_title to ''
DEBUG: 4553.0 KoboTouch:books - length bl=0
2016-02-11 11:55:50.168 Calibre[27856:1136914] deliverNotificationWithTitle()
* Notification delivered.
DEBUG: 4553.3 KoboTouch:books - length bl after sync=157
DEBUG: 4553.3 KoboTouch:books - reading device database
DEBUG: 4553.3 Database Version=125
DEBUG: 4553.3 KoboTouch:books - shelf list: [u'SlowCooker', u'Ian Rutledge', u'Science', u'NonFiction', u'Mystery', u'SciFi', u'Humor', u'Food', u'Lord John Grey', u'Fantasy', u'Outlander', u'Fiction', u'Biography', u'Health', u'Hobby', u'Skink', u'Mick Stranahan', u'Inspector Brant', u'Jack Taylor', u'Charles Paris', u'Harry Bosch', u'In Death', u'No1 Ladies', u'Dr. Fu', u'Amelia Peabody', u'Mickey Haller', u'Shandy', u'Rumpole', u'Jack McEvoy', u'Inspector Montalbano', u'zen', u'Isabel Dalhousie', u'44 Scotland Street', u"No 1 Ladies' Detective agency", u'Flavia de Luce Mystery', u'Black Lizard', u'Melanie Travis', u'Rama', u'Bill Smith, Lydia Chin']
DEBUG: 4553.3 KoboTouch:books - query= SELECT Title, Attribution, DateCreated, ContentID, MimeType, ContentType, ImageID, ReadStatus, ___ExpirationStatus, FavouritesIndex, Accessibility, IsDownloaded, Series, SeriesNumber, ___UserID, ExternalId FROM content where BookID is Null and ((Accessibility = -1 and IsDownloaded in ('true', 1 )) or (Accessibility in (1,2) and IsDownloaded in ('true', 1)) ) and not ((___ExpirationStatus=3 or ___ExpirationStatus is Null) and ContentType = 6) AND (externalId IS NULL OR externalId = '') AND contentId NOT LIKE 'file:///mnt/sd/%'
DeviceJob: 8 Get list of books on device done, calling callback
database disk image is malformed

Traceback (most recent call last):
File "site-packages/calibre/gui2/device.py", line 87, in run
File "site-packages/calibre/gui2/device.py", line 503, in _books
File "site-packages/calibre/devices/kobo/driver.py", line 1798, in books
File "site-packages/calibre/devices/kobo/driver.py", line 1674, in get_bookshelvesforbook
DatabaseError: database disk image is malformed
DeviceJob: 8 Get list of books on device callback returned
Job: 8 Get list of books on device finished
database disk image is malformed

Traceback (most recent call last):
File "site-packages/calibre/gui2/device.py", line 87, in run
File "site-packages/calibre/gui2/device.py", line 503, in _books
File "site-packages/calibre/devices/kobo/driver.py", line 1798, in books
File "site-packages/calibre/devices/kobo/driver.py", line 1674, in get_bookshelvesforbook
DatabaseError: database disk image is malformed

[PeterT]

The issue seems to be that the database on the Kobo is corrupted; that's what the malformed Disk image refers too.

Standard advice here would be to do a log out on the device itself and then redo the device setup .

[sallyg]

Hi PeterT,

Thank you for your quick reply. I ran "pragma integrity_check" on KoboReader.sqlite and it reports inconsistencies as you predicted.

Could I save the kobo sqlite file after loading a few books successfully, to use to get restor to back where I was if there is a failure after a group? Is there any other file calibre needs for "On Device" information?

Do you think the December 2015 Kobo firmware could be at fault?

[PeterT]

One of the functions of the Kobo Utilities plugin is to take a backup of the database if it's not corrupted.

Take a look at the Kobo Utilities Customize Plugin menu, especially the Devices tab. On it you should see options for Device Database Backup, where you can select either Backup the device database daily which will take a backup once a day, or Backup the device database on each connection which will take a backup each time the device is connected.

Also set a larger number for the Copies to keep option.

Be aware that the database gets updated when you disconnect the device from your computer, so if you want to load books in small quantities you need to send to device, then disconnect the device, wait for the Kobo to complete the Processing function and then reconnect the device and send more books.

There are no issues with the f/w; trust me when I say that if there were issues, either davidfor or myself would have ensured that the issue was prominently displayed in these forums, and that we would have worked with Kobo to get the issues resolved ASAP.

[sallyg]

The automated kobo database backups by the Kobo Utilities plugin you explained are great. If the kobo database gives trouble again and if by taking advantage of the backups I can pinpoint the reason, I'll post the details here. Threads about that plugin have lots of interesting information. Thanks for your clear help.