Quick Guide to Shelf Management with calibre 0.9.1 & above (Touch / Glo / Mini /Aura)

[Bald Eagle]

Ah, I misunderstood the plugin part, then. Thank you for your answer.

[Shadowjack]

I need some further help, please. I am trying to use the metadata plugboards to rename the title and authors fields on upload. I want to add the series to the beginning ahead of the title to sort on series collections; and append the pubdate to the end of the authors field so that I can sort within author collections by publication date. The collections are being created but the new title and authors fields are not being updated. I am using:
{series:re(([^\s])[^\s]+(\s|$),\1)}{series_index:0>2s| - | - }{title}
{authors}{pubdate}

Thoughts?

thanks.

[Lilarcor]

i got H2O several days ago and tried to use Calibre to manage device's collection and always got following warning.Tried to reset several times to fix the database in device , then sent books with Calibre and got warning again and again.

calibre, version 2.13.0
ERROR: Error: Error communicating with device

database disk image is malformed

Traceback (most recent call last):
  File "site-packages\calibre\gui2\device.py", line 87, in run
  File "site-packages\calibre\gui2\device.py", line 608, in _delete_books
  File "site-packages\calibre\devices\kobo\driver.py", line 496, in delete_books
  File "site-packages\calibre\devices\kobo\driver.py", line 2009, in delete_via_sql
  File "site-packages\calibre\devices\kobo\driver.py", line 428, in delete_via_sql
DatabaseError: database disk image is malformed

following is the Debug log

calibre Debug log
calibre 2.13 Portable isfrozen: True is64bit: False
Windows-8-6.2.9200 Windows ('32bit', 'WindowsPE')
32bit process running on 64bit windows
('Windows', '8', '6.2.9200')
Python 2.7.8
Windows: ('8', '6.2.9200', '', 'Multiprocessor Free')
Successfully initialized third party plugins: Kindle Collections && KoboTouchExtended && Kobo Utilities
Starting up...
DEBUG:    0.0 KoboUtilites::action.py - loading translations
DEBUG:    0.0 KoboUtilites::dialogs.py - loading translations
DEBUG:    0.0 KoboUtilites::action.py - loading translations
DEBUG:    0.6 No Kobo Touch, Glo or Mini appears to be connected
DEBUG:    0.6 rebuild_menus - have device.
DEBUG:    0.6 rebuild_menus - self.supports_ratings=False
DEBUG:   10.6 KoboUtilities:_on_device_connection_changed - self.plugin_device_connection_changed.__class__:  <type 'PyQt5.QtCore.pyqtBoundSignal'>
DEBUG:   10.6 Methods for self.plugin_device_connection_changed:  ['__call__', '__class__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__getitem__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'connect', 'disconnect', 'emit', 'signal']
Job: 1 Get device information started
DeviceJob: 1 Get device information done, calling callback
DeviceJob: 1 Get device information callback returned
Job: 2 Set library information started
DeviceJob: 2 Set library information done, calling callback
DeviceJob: 2 Set library information callback returned
Job: 3 Get list of books on device started
DEBUG:    0.0 KoboTouch:books - oncard='None'
DEBUG:    0.0 KoboTouch:books - oncard='None', prefix='G:\'
DEBUG:    0.0 Kobo device: Kobo Aura H2O
DEBUG:    0.0 Version of driver: (2, 1, 7) Has kepubs: True
DEBUG:    0.0 Version of firmware: (3, 12, 0) Has kepubs: True
DEBUG:    0.0 Firmware supports cover image tree: True
DEBUG:    0.0 KoboTouch:books - opts.extra_customization= [u'tags,seriel', True, True, False, False, False, False, False, False, False, False, False, u'']
DEBUG:    0.0 KoboTouch:books - prefs['manage_device_metadata']= on_connect
DEBUG:    0.0 KoboTouch:books - set_debugging_title to ''
DEBUG:    0.0 KoboTouch:books - length bl=0
DEBUG:    0.1 KoboTouch:books - length bl after sync=16
DEBUG:    0.1 KoboTouch:books - reading device database
DEBUG:    0.1 Database Version=111
DEBUG:    0.1 KoboTouch:books - shelf list: [u'China', u'Fiction', u'History', u'Philosophy']
DEBUG:    0.1 KoboTouch:books - query= SELECT Title, Attribution, DateCreated, ContentID, MimeType, ContentType, ImageID, ReadStatus, ___ExpirationStatus, FavouritesIndex, Accessibility, IsDownloaded, Series, SeriesNumber, ___UserID, ExternalId FROM content  where BookID is Null  and ((Accessibility = -1 and IsDownloaded in ('true', 1 )) or (Accessibility in (1,2) and IsDownloaded in ('true', 1))       ) and not ((___ExpirationStatus=3 or ___ExpirationStatus is Null) and ContentType = 6) AND (externalId IS NULL OR externalId = '') AND contentId NOT LIKE 'file:///mnt/sd/%'
DEBUG:    0.3 KoboTouch:books - automatically managing metadata
DEBUG:    0.3 KoboTouch:books - about to sync_booklists
DEBUG:    0.3 USBMS: starting sync_booklists
DEBUG:    0.4 USBMS: finished sync_booklists
DEBUG:    0.4 KoboTouch:books - have done sync_booklists
DEBUG:    0.4 KoboTouch:books - end - oncard='None'
DEBUG:    0.4 KoboTouch:books - oncard='carda'
DEBUG:    0.4 KoboTouch:books - oncard='carda', prefix='H:\'
DEBUG:    0.4 Kobo device: Kobo Aura H2O
DEBUG:    0.4 Version of driver: (2, 1, 7) Has kepubs: True
DEBUG:    0.4 Version of firmware: (3, 12, 0) Has kepubs: True
DEBUG:    0.4 Firmware supports cover image tree: True
DEBUG:    0.4 KoboTouch:books - opts.extra_customization= [u'tags,seriel', True, True, False, False, False, False, False, False, False, False, False, u'']
DEBUG:    0.4 KoboTouch:books - prefs['manage_device_metadata']= on_connect
DEBUG:    0.4 KoboTouch:books - set_debugging_title to ''
DEBUG:    0.4 KoboTouch:books - length bl=0
DEBUG:    0.4 KoboTouch:books - length bl after sync=11
DEBUG:    0.4 KoboTouch:books - reading device database
DEBUG:    0.4 Database Version=111
DEBUG:    0.4 KoboTouch:books - shelf list: [u'China', u'Fiction', u'History', u'Philosophy']
DEBUG:    0.4 KoboTouch:books - query= SELECT Title, Attribution, DateCreated, ContentID, MimeType, ContentType, ImageID, ReadStatus, ___ExpirationStatus, FavouritesIndex, Accessibility, IsDownloaded, Series, SeriesNumber, ___UserID, ExternalId FROM content  where BookID is Null  and ((Accessibility = -1 and IsDownloaded in ('true', 1 )) or (Accessibility in (1,2) and IsDownloaded in ('true', 1))       ) and not ((___ExpirationStatus=3 or ___ExpirationStatus is Null) and ContentType = 6) AND (externalId IS NOT NULL AND externalId <> '' OR contentId LIKE 'file:///mnt/sd/%')
DEBUG:    0.5 KoboTouch:books - automatically managing metadata
DEBUG:    0.5 KoboTouch:books - about to sync_booklists
DEBUG:    0.5 USBMS: starting sync_booklists
DEBUG:    0.5 USBMS: finished sync_booklists
DEBUG:    0.5 KoboTouch:books - have done sync_booklists
DEBUG:    0.5 KoboTouch:books - end - oncard='carda'
DEBUG:    0.5 KoboTouch:books - oncard='cardb'
DEBUG:    0.5 KoboTouch:books - Asked to process 'cardb', but do not have one!
DeviceJob: 3 Get list of books on device done, calling callback
DeviceJob: metadata_downloaded: Starting set_books_in_library
DeviceJob: set_books_in_library: books to process= 25
DeviceJob: set_books_in_library finished: time= 0.00300002098083
DeviceJob: metadata_downloaded: updating views
DeviceJob: metadata_downloaded: syncing
DeviceJob: metadata_downloaded: refreshing ondevice
DeviceJob: metadata_downloaded: sending metadata_available signal
DEBUG:   11.2 KoboUtilities:_on_device_metadata_available - Metadata available: {'A': {u'prefix': 'H:/', u'device_name': u'Kobo Aura H2O', u'device_store_uuid': u'd098b1d5-8fc0-49f8-bfcf-2af3a72d3cf8', u'calibre_version': u'2.13.0', u'date_last_connected': u'2014-12-26T01:18:26.525000+00:00', u'location_code': 'A', u'last_library_uuid': None}, 'main': {u'prefix': 'G:/', u'device_name': u'Kobo Aura H2O', u'device_store_uuid': u'57c9f227-776c-4906-a579-929e620dd0e9', u'calibre_version': u'2.13.0', u'date_last_connected': u'2014-12-26T01:18:26.499000+00:00', u'location_code': 'main', u'last_library_uuid': None}}
DEBUG:   11.2 Have a Kobo device connected connected
DEBUG:   11.3 Have a Kobo device connected connected
DEBUG:   11.3 rebuild_menus - have device.
DEBUG:   11.3 rebuild_menus - self.supports_ratings=True
DeviceJob: 3 Get list of books on device callback returned
Job: 4 Send metadata to device started
DEBUG:    0.6 KOBO:sync_booklists - start
DEBUG:    0.6 KoboTouch:update_device_database_collections - oncard='main'
DEBUG:    0.6 KoboTouch:update_device_database_collections - set_debugging_title to ''
Job: 2 Set library information finishedDEBUG:    0.6
 KTCollectionsBookList:get_collections - start - collection_attributes= [u'tags', u'seriel']
No details available.DEBUG:    0.6
 KTCollectionsBookList:get_collections - collection_attributes= [u'tags', u'seriel']
Job: 1 Get device information finished
No details available.DEBUG:    0.6
 KTCollectionsBookList:get_collections - end
Job: 3 Get list of books on device finished
No details available.
DEBUG:    0.6 No Collections - reseting ReadStatus
DEBUG:    0.6 KoboTouch:update_device_database_collections - managing bookshelves and series.
DEBUG:    0.6 KoboTouch:update_device_database_collections - about to clear empty bookshelves
DEBUG:    0.6 KoboTouch:delete_empty_bookshelves - start
DEBUG:    0.6 KoboTouch:delete_empty_bookshelves - end
DEBUG:    0.6 KoboTouch:update_device_database_collections - Number of series set=0 Number of books=0
DEBUG:    0.6 KoboTouch:update_device_database_collections - Finished 
DEBUG:    0.6 KoboTouch:update_device_database_collections - oncard='carda'
DEBUG:    0.6 KoboTouch:update_device_database_collections - set_debugging_title to ''
DEBUG:    0.6 KTCollectionsBookList:get_collections - start - collection_attributes= [u'tags', u'seriel']
DEBUG:    0.6 KTCollectionsBookList:get_collections - collection_attributes= [u'tags', u'seriel']
DEBUG:    0.6 KTCollectionsBookList:get_collections - end
DEBUG:    0.7 No Collections - reseting ReadStatus
DEBUG:    0.7 KoboTouch:update_device_database_collections - managing bookshelves and series.
DEBUG:    0.7 KoboTouch:update_device_database_collections - about to clear empty bookshelves
DEBUG:    0.7 KoboTouch:delete_empty_bookshelves - start
DEBUG:    0.7 KoboTouch:delete_empty_bookshelves - end
DEBUG:    0.7 KoboTouch:update_device_database_collections - Number of series set=0 Number of books=0
DEBUG:    0.7 KoboTouch:update_device_database_collections - Finished 
DEBUG:    0.7 USBMS: starting sync_booklists
DEBUG:    0.8 USBMS: finished sync_booklists
DEBUG:    0.8 KOBO:sync_booklists - end
DeviceJob: 4 Send metadata to device done, calling callback
DeviceJob: 4 Send metadata to device callback returned
Job: 4 Send metadata to device finished
No details available.
Started up in 33.24 seconds with 1 books
DEBUG:   39.4 Have a Kobo device connected connected
DeviceJob: set_books_in_library: books to process= 25
DeviceJob: set_books_in_library finished: time= 0.230999946594
Job: 5 Send metadata to device started
DEBUG:   30.8 KOBO:sync_booklists - start
DEBUG:   30.8 KoboTouch:update_device_database_collections - oncard='main'
DEBUG:   30.8 KoboTouch:update_device_database_collections - set_debugging_title to ''
DEBUG:   30.8 KTCollectionsBookList:get_collections - start - collection_attributes= [u'tags', u'seriel']
DEBUG:   30.8 KTCollectionsBookList:get_collections - collection_attributes= [u'tags', u'seriel']
DEBUG:   30.8 KTCollectionsBookList:get_collections - end
DEBUG:   30.8 KoboTouch:update_device_database_collections - category='China' books=6
DEBUG:   30.8 KoboTouch:update_device_database_collections - end for category='China'
DEBUG:   30.8 KoboTouch:update_device_database_collections - category='Philosophy' books=1
DEBUG:   30.8 KoboTouch:update_device_database_collections - end for category='Philosophy'
DEBUG:   30.8 KoboTouch:update_device_database_collections - category='Fiction' books=3
DEBUG:   30.8 KoboTouch:update_device_database_collections - end for category='Fiction'
DEBUG:   30.8 KoboTouch:update_device_database_collections - category='History' books=5
DEBUG:   30.9 KoboTouch:update_device_database_collections - end for category='History'
DEBUG:   30.9 KoboTouch:update_device_database_collections - managing bookshelves and series.
DEBUG:   30.9 KoboTouch:update_device_database_collections - about to clear empty bookshelves
DEBUG:   30.9 KoboTouch:delete_empty_bookshelves - start
DEBUG:   30.9 KoboTouch:delete_empty_bookshelves - end
DEBUG:   30.9 KoboTouch:update_device_database_collections - Number of series set=0 Number of books=9
DEBUG:   30.9 KoboTouch:update_device_database_collections - Finished 
DEBUG:   30.9 KoboTouch:update_device_database_collections - oncard='carda'
DEBUG:   30.9 KoboTouch:update_device_database_collections - set_debugging_title to ''
DEBUG:   30.9 KTCollectionsBookList:get_collections - start - collection_attributes= [u'tags', u'seriel']
DEBUG:   30.9 KTCollectionsBookList:get_collections - collection_attributes= [u'tags', u'seriel']
DEBUG:   30.9 KTCollectionsBookList:get_collections - end
DEBUG:   30.9 No Collections - reseting ReadStatus
DEBUG:   30.9 KoboTouch:update_device_database_collections - managing bookshelves and series.
DEBUG:   30.9 KoboTouch:update_device_database_collections - about to clear empty bookshelves
DEBUG:   30.9 KoboTouch:delete_empty_bookshelves - start
DEBUG:   31.0 KoboTouch:delete_empty_bookshelves - end
DEBUG:   31.0 KoboTouch:update_device_database_collections - Number of series set=0 Number of books=0
DEBUG:   31.0 KoboTouch:update_device_database_collections - Finished 
DEBUG:   31.0 USBMS: starting sync_booklists
DEBUG:   31.0 USBMS: finished sync_booklists
DEBUG:   31.0 KOBO:sync_booklists - end
DeviceJob: 5 Send metadata to device done, calling callback
DeviceJob: 5 Send metadata to device callback returned
Job: 6 Set library information started
DeviceJob: 6 Set library information done, calling callback
DeviceJob: 6 Set library information callback returned
Job: 6 Set library information finished
No details available.
Job: 5 Send metadata to device finished
No details available.
Job: 7 Delete books from device started
DEBUG:   35.1 KoboTouch:get_content_type_from_extension - start
DEBUG:   35.1 KoboTouch:get_content_type_from_extension - V2 firmware
DEBUG:   35.1 delete_via_sql: ContentID:  file:///mnt/onboard/books/Er Shi Si Shi - Duo Ren.epub ContentType:  6
DeviceJob: 7 Delete books from device done, calling callback
database disk image is malformed

Traceback (most recent call last):
  File "site-packages\calibre\gui2\device.py", line 87, in run
  File "site-packages\calibre\gui2\device.py", line 608, in _delete_books
  File "site-packages\calibre\devices\kobo\driver.py", line 496, in delete_books
  File "site-packages\calibre\devices\kobo\driver.py", line 2009, in delete_via_sql
  File "site-packages\calibre\devices\kobo\driver.py", line 428, in delete_via_sql
DatabaseError: database disk image is malformed

DeviceJob: 7 Delete books from device callback returned
Job: 7 Delete books from device finished
database disk image is malformed
	
	Traceback (most recent call last):
	  File "site-packages\calibre\gui2\device.py", line 87, in run
	  File "site-packages\calibre\gui2\device.py", line 608, in _delete_books
	  File "site-packages\calibre\devices\kobo\driver.py", line 496, in delete_books
	  File "site-packages\calibre\devices\kobo\driver.py", line 2009, in delete_via_sql
	  File "site-packages\calibre\devices\kobo\driver.py", line 428, in delete_via_sql
	DatabaseError: database disk image is malformed

[JSWolf]

Log out of your H2O and log back in. Then the database will be rebuilt and you can go from there. Suggestion is to install the Kobo Utilities plugin that will allow you to have your database backed up every time you connect your H2O to Calibre. So if you do run into an issue with a corrupt database, you can copy back the last good database.

[Lilarcor]

Originally Posted by JSWolf

Log out of your H2O and log back in. Then the database will be rebuilt and you can go from there. Suggestion is to install the Kobo Utilities plugin that will allow you to have your database backed up every time you connect your H2O to Calibre. So if you do run into an issue with a corrupt database, you can copy back the last good database.

Thanks for your info.
Hope Calibre likes Kobo in future.

[JSWolf]

Originally Posted by Lilarcor

Thanks for your info.
Hope Calibre likes Kobo in future.

Calibre like Kobo now. I use an H2O with Calibre with no issues.

[Lilarcor]

do you use koreader, I am not sure if koreader made the issue

[JSWolf]

Originally Posted by Lilarcor

do you use koreader, I am not sure if koreader made the issue

I use RMDSK (ADE) and it works. I also use Calibre to load eBooks. None of the eBooks I load have DRM.

[Lilarcor]

i found sometime I got missing collecting when discounted from PC/calibre, after a reboot , collections will be back. When I got database issue , I restore the last good backup.

[Lilarcor]

i think I found the reason. It's caused by KSM. I am using koreader which need to install KSM first. Before I installed KSM, everything was fine even I use Calibre to send books with many times.Collection was updated once I unplugged the kobo. After I installed KSM, things went wrong. Collection wasn't updated at once till a reboot. And sometime calibre has problem to talk with kobo.

[Lilarcor]

just tried it again on MAC, warning again.

*** in database main ***
On tree page 11067 cell 0: Rowid 66 out of order (max larger than parent max of 61)
On tree page 11068 cell 1: Rowid 49 out of order (min less than parent min of 66)
row 108 missing from index sqlite_autoindex_AnalyticsEvents_1
wrong # of entries in index sqlite_autoindex_AnalyticsEvents_1
row 7 missing from index sqlite_autoindex_Event_1
wrong # of entries in index sqlite_autoindex_Event_1

Traceback (most recent call last):
  File "site-packages/calibre/gui2/device.py", line 87, in run
  File "calibre_plugins.koboutilities.jobs", line 184, in do_device_database_backup
Exception: 
*** in database main ***
On tree page 11067 cell 0: Rowid 66 out of order (max larger than parent max of 61)
On tree page 11068 cell 1: Rowid 49 out of order (min less than parent min of 66)
row 108 missing from index sqlite_autoindex_AnalyticsEvents_1
wrong # of entries in index sqlite_autoindex_AnalyticsEvents_1
row 7 missing from index sqlite_autoindex_Event_1
wrong # of entries in index sqlite_autoindex_Event_1

[Lilarcor]

Originally Posted by Lilarcor

just tried it again on MAC, warning again.

*** in database main ***
On tree page 11067 cell 0: Rowid 66 out of order (max larger than parent max of 61)
On tree page 11068 cell 1: Rowid 49 out of order (min less than parent min of 66)
row 108 missing from index sqlite_autoindex_AnalyticsEvents_1
wrong # of entries in index sqlite_autoindex_AnalyticsEvents_1
row 7 missing from index sqlite_autoindex_Event_1
wrong # of entries in index sqlite_autoindex_Event_1

Traceback (most recent call last):
  File "site-packages/calibre/gui2/device.py", line 87, in run
  File "calibre_plugins.koboutilities.jobs", line 184, in do_device_database_backup
Exception: 
*** in database main ***
On tree page 11067 cell 0: Rowid 66 out of order (max larger than parent max of 61)
On tree page 11068 cell 1: Rowid 49 out of order (min less than parent min of 66)
row 108 missing from index sqlite_autoindex_AnalyticsEvents_1
wrong # of entries in index sqlite_autoindex_AnalyticsEvents_1
row 7 missing from index sqlite_autoindex_Event_1
wrong # of entries in index sqlite_autoindex_Event_1

It's a wrong alarm I think. Kobo utility reported the error because of the failure DB backup while plugging the Kobo. Don't know if it's a bug in MAC. In PC it's fine.

[Lilarcor]

after a reboot of MAC, the error was gone. I will keep an eye on it.
For the collection , here are what I found.
1. follow the instruction of the topic to set up calibre
2.
a.kobo not install KSM , everything is fine.Send books kobo through Calibre, then eject the device, kobo will re-index and show books and collections correctly.
b. install KSM, something strange.After sending kobo through Calibre and eject device, sometime, kobo will not re-index, and although kobo show the new books at once but not time the collection was updated. I have to make a reboot or use exit_nickel then run nickel again, collection will be updated. And sometime, Calibre has problem to talk to Kobo.
Above tests are based on several times to reset to factory/ log out account on kobo for repairing the DB.

So I am pretty sure KSM cause the issue.

[jnelbee]

Thank you so much! This helped tons.

[Shopaholic]

I'm having difficulty with this. I can use Calibre to put books into my Sony readers and they go into shelves but not with the Kobo Touch. They just load on the reader and I have to manually put them in shelves. Ok for a few books. Not ok for many.

I have both the Kobo Touch and Kobo Touch Extended plugins. I did copy and paste of the formatting in OP but no go.

Assistance would be appreciated.

Thanks