Books not in device database

davidfor · 11-07-2012, 01:21 AM

For the Kobo devices, all books put onto the device have their details loaded into a database. The calibre driver reads this plus the metadata.calibre to find the books on the device and show which are in the calibre library. The driver doesn't scan the devices drive to look for books.

This means that any books that have been copied to the device before calibre was started will not be shown anywhere until the device is disconnected and the book is processed. I'm happy with that and don't plan to change it.

But, there are two circumstances where a book can be in the metadata.calibre file, on the devices drive but not in the devices database.

The first of these is when you use calibre to send the the book to the device. Until the device is disconnected, it is in this state but it is shown in the device list as on the device. If calibre and stopped and started without disconnecting, the book disappears from the device list. This is because the database and metadata.calibre are reread and as the book isn't in the database, it is removed from the metadata.calibre.

The other circumstance is if the device couldn't process the book when it was disconnected. This happens if there is a problem with the book. When it happens, processing for all new books stops so others will not be added to the database as well.

The problem that has come up is that I had to make a change to the driver to solve problems with showing the books were on the device. One change meant that for the circumstances above, I am no longer removing the book from the metadata.calibre. This also means that the book shows in the device list and will be shown as on the device in the library. In the first case above, that is OK as the book should be added to the devices database at next disconnect. In the second case, it isn't as good.

So, with all that, what I am asking is:

What has been done for other devices in a similar circumstance? Are the books that are not in the devices database shown in the device list? If so, are they marked in different way?

I am considering using a fake collection for these books. That way they can be easily found. The suggestion has been to call this "Processing_Pending" but if someone has already done something like this, I would like to be consistent. Or if there is a better way to mark these books, I would be very happy to hear it.

David

kovidgoyal · 11-07-2012, 01:50 AM

The only other driver that manages an on device database is the sony driver. At least for the older devices (before the T1) the driver scans both the actual files and reads the device db and metadata.calibre. It then populates the metadata from those three sources. IIRC the device db has the lowest priority. So files on the device are always shown as on device regardless of their status in the device db. This has never been a problem with the SONYs as they are a lot more robust than the Kobos.

There are two possibilities that come to mind:

1) Use your custom collection

2) Add some way for the device driver system to signal to the GUI that the book is not recognized by the device, the gui could then use an exclamation mark instead of a tick to indicate such problem books.

(1) you can do yourself, (2) will require some work either from chaley or me. I cant speak for Charles, but I am rather busy at the moment, so if you want to pursue (2) you might have to wait a few days.

Personally, I prefer (2)

chaley · 11-07-2012, 05:59 AM

Quote:

Originally Posted by kovidgoyal

2) Add some way for the device driver system to signal to the GUI that the book is not recognized by the device, the gui could then use an exclamation mark instead of a tick to indicate such problem books.

This is possible, but there are a few things to watch out for.

The in_library column, where the check marks are, is currently populated when doing book matching. This opens the question: what should happen if the book is both a problem book and in the library? We could:
- add two new icons, one meaning in_library & problem, and the other meaning not in_library and problem.
- add text to the field similar to what happens when DEBUG is turned on. In this case the field would contain text but might or might not contain a check.

Another question: how does someone search for these? Would we allow text searches on the nominally boolean column?

Third question: how does the column sort? Does the text change the nominal value so that they sort together?

None of these are really hard to deal with.

As for how to signal that a book has a problem, this can be done by agreeing on an attribute name for the property. This name would not be added to the list of standard metadata items, so it wouldn't end up in metadata.calibre or in OPF files. The sony driver does something like this for new books and ?_sort values to pass information to the collection builder. I suggest something like "_book_not_in_db" if we want a bool value, or "_device_book_problem" if we want a text value that will be displayed in the GUI. Because this value crosses from device land to gui land, we would want to document its existence in interface.py.

I can do the non-driver work. Probably not this week, but certainly by next week's calibre release. Let me know how you want to proceed.

davidfor · 11-07-2012, 06:10 AM

Quote:

Originally Posted by kovidgoyal

The only other driver that manages an on device database is the sony driver. At least for the older devices (before the T1) the driver scans both the actual files and reads the device db and metadata.calibre. It then populates the metadata from those three sources. IIRC the device db has the lowest priority. So files on the device are always shown as on device regardless of their status in the device db. This has never been a problem with the SONYs as they are a lot more robust than the Kobos.

This isn't a case of lack of robustness. It is either the device hasn't had a chance to look at the book, or the device finding a problem with the book and not loading it.

kovidgoyal · 11-07-2012, 06:35 AM

I'd say go for a different colored tick, say red instead of green for a problem book.

IIRC in the library view i.e. in the on device column, there is already text indicating where the book is, that text can be special cased for this as well.

In the device view, i.e. the in_library column we can add text as well for problem books.

Sorting: problem books should sort first, then non problem books then not on device books.

Search: Not sure about that one, given the sorting above, I dont think its particularly important. Given that there is text, i'd say allow searching for that, but, the devil is in the details.

As for the attribute, I suggest

_device_book_problem = "A long message from the driver indicating the problem" or None if there is no problem.

The long message can be used as a tooltip in the GUI, the short indicator text in the column can be a standard string, rather than coming from the driver.

I wont be able to get to this until about two weeks from now, so if you want to do it before then, go ahead.

davidfor · 11-07-2012, 06:59 AM

I'm typing to slow tonight. I'd just about finished something like Kovid's comments.

About the only extra thing was to suggest using the tool tip for sorting.

I'm in no hurry and would prefer to leave it in the back of everyone brain for a few days to see if a better idea or problems turn up. I do have changes for the driver this week which did include a line for the collection idea. I'll leave that out.

Thanks for the help and ideas.

11-07-2012, 01:21 AM	#1
davidfor Grand Sorcerer Posts: 24,907 Karma: 47303748 Join Date: Jul 2011 Location: Sydney, Australia Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos	Books not in device database For the Kobo devices, all books put onto the device have their details loaded into a database. The calibre driver reads this plus the metadata.calibre to find the books on the device and show which are in the calibre library. The driver doesn't scan the devices drive to look for books. This means that any books that have been copied to the device before calibre was started will not be shown anywhere until the device is disconnected and the book is processed. I'm happy with that and don't plan to change it. But, there are two circumstances where a book can be in the metadata.calibre file, on the devices drive but not in the devices database. The first of these is when you use calibre to send the the book to the device. Until the device is disconnected, it is in this state but it is shown in the device list as on the device. If calibre and stopped and started without disconnecting, the book disappears from the device list. This is because the database and metadata.calibre are reread and as the book isn't in the database, it is removed from the metadata.calibre. The other circumstance is if the device couldn't process the book when it was disconnected. This happens if there is a problem with the book. When it happens, processing for all new books stops so others will not be added to the database as well. The problem that has come up is that I had to make a change to the driver to solve problems with showing the books were on the device. One change meant that for the circumstances above, I am no longer removing the book from the metadata.calibre. This also means that the book shows in the device list and will be shown as on the device in the library. In the first case above, that is OK as the book should be added to the devices database at next disconnect. In the second case, it isn't as good. So, with all that, what I am asking is: What has been done for other devices in a similar circumstance? Are the books that are not in the devices database shown in the device list? If so, are they marked in different way? I am considering using a fake collection for these books. That way they can be easily found. The suggestion has been to call this "Processing_Pending" but if someone has already done something like this, I would like to be consistent. Or if there is a better way to mark these books, I would be very happy to hear it. David

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Error communicating with device(Kobo Wireless)/database disk image is malformed	cotej1977	Devices	2	04-26-2011 09:34 PM
Error communicating with device(Kobo Wireless)/database disk image is malformed	cotej1977	Kobo Reader	1	04-24-2011 08:14 AM
Kobo Desktop erased entire device database! - Don't dowload update	MrsJoseph	Kobo Reader	9	03-23-2011 11:38 AM
We are unable to update your device's database. Syncing cannot be completed.	talios	Kobo Reader	1	12-10-2010 03:16 AM
Deleting books from a database	candide	Calibre	6	10-26-2010 03:04 PM

11-07-2012, 01:50 AM	#2
kovidgoyal creator of calibre Posts: 43,838 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	The only other driver that manages an on device database is the sony driver. At least for the older devices (before the T1) the driver scans both the actual files and reads the device db and metadata.calibre. It then populates the metadata from those three sources. IIRC the device db has the lowest priority. So files on the device are always shown as on device regardless of their status in the device db. This has never been a problem with the SONYs as they are a lot more robust than the Kobos. There are two possibilities that come to mind: 1) Use your custom collection 2) Add some way for the device driver system to signal to the GUI that the book is not recognized by the device, the gui could then use an exclamation mark instead of a tick to indicate such problem books. (1) you can do yourself, (2) will require some work either from chaley or me. I cant speak for Charles, but I am rather busy at the moment, so if you want to pursue (2) you might have to wait a few days. Personally, I prefer (2)

11-07-2012, 06:35 AM	#5
kovidgoyal creator of calibre Posts: 43,838 Karma: 22666666 Join Date: Oct 2006 Location: Mumbai, India Device: Various	I'd say go for a different colored tick, say red instead of green for a problem book. IIRC in the library view i.e. in the on device column, there is already text indicating where the book is, that text can be special cased for this as well. In the device view, i.e. the in_library column we can add text as well for problem books. Sorting: problem books should sort first, then non problem books then not on device books. Search: Not sure about that one, given the sorting above, I dont think its particularly important. Given that there is text, i'd say allow searching for that, but, the devil is in the details. As for the attribute, I suggest _device_book_problem = "A long message from the driver indicating the problem" or None if there is no problem. The long message can be used as a tooltip in the GUI, the short indicator text in the column can be a standard string, rather than coming from the driver. I wont be able to get to this until about two weeks from now, so if you want to do it before then, go ahead.

11-07-2012, 06:59 AM	#6
davidfor Grand Sorcerer Posts: 24,907 Karma: 47303748 Join Date: Jul 2011 Location: Sydney, Australia Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos	I'm typing to slow tonight. I'd just about finished something like Kovid's comments. About the only extra thing was to suggest using the tool tip for sorting. I'm in no hurry and would prefer to leave it in the back of everyone brain for a few days to see if a better idea or problems turn up. I do have changes for the driver this week which did include a line for the collection idea. I'll leave that out. Thanks for the help and ideas.