Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Devices

Notices

Reply
 
Thread Tools Search this Thread
Old 03-01-2013, 08:16 PM   #121
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 24,908
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
There isn't any date checking going on in the driver. It simple copies the covers based on the options. Pre 0.9.21, if you had the "cover upload" option but not the "always upload" option, the covers were not sent when you first sent the book.

As PeterT said, each of the 2.x.x firmware versions have had a bug with the covers. As you are on 2.3.1, the wrong cover is used if you open a new book and let the device go to sleep before returning to the home screen. If you return to the home screen before the device goes to sleep, the correct cover will be used.
davidfor is offline   Reply With Quote
Old 03-08-2013, 05:04 AM   #122
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 24,908
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
Kobo collection changes in calibre 0.9.22

With Kobo adding Archiving of kepubs in firmware 2.4.0, the calibre driver will now display this as a collection in the device list. Along with this I have done some big fixing on the Kobo collections.

The changes include:
  • Archived books will show in the device list with a collection of "Archived".
  • The "Recommendation" and "Preview" options didn't work properly.
  • Some previews showed as recommendations. This would have been most noticeable with fw 2.4.0.
  • If a recommendation matched a book in the calibre library, it could end up on a shelf. The books on the device would show with a "Buy now" button and could not be removed from the shelf. This will no longer happen.

If you currently have recommendations appearing on a shelf, to remove them:
  1. Make sure the "Show Recommendations" is selected in the driver options and metadata management is set to automatic.
  2. Connect the device.
  3. Let calibre sync to the device.
  4. Eject the device and check the shelves. If any recommendations are still on shelves, complain to me.
davidfor is offline   Reply With Quote
Advert
Old 03-09-2013, 09:40 AM   #123
ichrispa
Enthusiast
ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.
 
Posts: 40
Karma: 8604
Join Date: Dec 2012
Location: Germany
Device: Kobo Touch
Hello Davidfor,

I have my Kobo back and will try to run a complete check to find out where exactly the problem with updating the onboard database lies.

Initial test conditions

I will start by establishing that the initial database is ok before connecting to the PC.

Code:
kobo login: root
[root@kobo /]# cd /mnt/onboard/.kobo/
[root@kobo .kobo]# echo "pragma integrity_check;" | sqlite3 KoboReader.sqlite 
ok
[root@kobo .kobo]#
Note: root@kobo is the telnet commandline prompt on the KT.

A backup of that database version is present on the KT.

Given that the original DB is ok, I updated my calibre to version 0.9.22. Driver setting are:
  • KoboReader Driver is disabled.
  • KoboTouch Driver:
    • Read metadata from device.
    • Use sub directories.
    • Save_template = {author_sort}_-_{title}
    • Tags for bookshelve management = #shelves, series
    • Create bookshelves.
    • Set series information.

The KT Firmware Version is 2.4.0, database version is 69.

PC Connection

I now connect the KT to the PC using the USB cable and mount the onboard FS. I will establish that the filesystem is ok...

Code:
Cassandra:/home/ichrispa/bin # fsck /dev/sdd
fsck from util-linux-ng 2.17.2
dosfsck 3.0.9, 31 Jan 2010, FAT32, LFN
/dev/sdd: 6742 files, 127647/348630 clusters
...and that the databse is still intact.

Code:
Cassandra:/home/ichrispa/bin # echo "pragma integrity_check;" | sqlite3 /media/KOBOeReader/.kobo/KoboReader.sqlite 
ok
Cassandra:/home/ichrispa/bin #
In the next step, I will unmount the device and startup calibre, which runs it's own device detection and should mount the KT.

Calibre correctly recognizes the device, mounts it and the driver scans the database. The contents of the main memory are shown correctly (no sd card is being used).

Rechecking the database still yields no database errors, everything is still ok.


Database manipulation

I will now copy a couple of ebooks to main memory. Specifically, I will copy the Song of Ice and Fire books by George R.R. Martin, as these have series information and shelves set. The books are in ebup format. I have reconverted them from epub to epub to make sure that they do not contain formatting errors. Metadata information is correctly displayed both by calibre as well as by the ebook-viewer and FBReader.

Sending the ebooks to main memory completes with no errors. Each book was send seperately, in contrast to selecting all books and then clicking "send to device".

Database integrity check is still ok.

Code:
Cassandra:/home/ichrispa/bin # echo "pragma integrity_check;" | sqlite3 /media/KOBOeReader/.kobo/KoboReader.sqlite 
ok
Cassandra:/home/ichrispa/bin #
I eject the device using calibre's eject function.

The OS (Linux) confirms the FS has been unmounted:

Code:
Cassandra:/home/ichrispa/bin # mount | grep KOBO
Cassandra:/home/ichrispa/bin #
Unfortunately, calibre apparently terminates the USB device connection on a hardware level. /dev/sdd is still there, but it is not mountable or readable. lsusb also shows the device by ID, but is unable to retrieve any device information. There is no way to remount the device at this point in order to check the database, safe for resetting the USB, which will cause the KT to scan the DB.

Effects on device

Disconnecting the KT takes some time for scanning the new contents, then the device shows all uploaded covers on the home screen.

Series information is only set for the first book in the calibre list (Series book 5).
A shelve has been created, but only the first book of the series has been added to it.

Checking the onboard database yields...

Code:
[root@kobo .kobo]# echo "pragma integrity_check;" | sqlite3 KoboReader.sqlite 
*** in database main ***
On tree page 1758 cell 1: Rowid 2782 out of order (min less than parent min of 2782)
rowid 1689 missing from index content_date_last_read_index
wrong # of entries in index content_mime_type
wrong # of entries in index content_bookid_index
wrong # of entries in index content_date_last_read_index
wrong # of entries in index content_title_index
wrong # of entries in index content_attribution_index
wrong # of entries in index sqlite_autoindex_content_1
For now, there are no errors when reading books or using the KT. However from experience I can tell that in several days, when using the KT regulary, the behavior will become more and more erratic.

Conclusion

The database is not corrupted from the calibre driver as such. The error is created when nickel scans the modified database after disconnecting the device.

If you need them, I can send you both the healthy initial db as well as the broken version via PM.

Thanks for your help davidfor,

regards,

ichrispa
ichrispa is offline   Reply With Quote
Old 03-09-2013, 08:46 PM   #124
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 24,908
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
That's a comprehensive test and results. Unfortunately, I don't know what is happening. But, it makes it sound like the database might still be opened when calibre ejects the device.

Some things I can think of to try:

- After sending the book to the device, close calibre, check the database and then eject using the OS.

- After sending the book to the device, close calibre, check the database and then reopen calibre and do the eject from there.

- Don't use calibre but manually copy the book and eject using the OS.

That might help to work out the cause.
davidfor is offline   Reply With Quote
Old 03-11-2013, 03:07 AM   #125
ichrispa
Enthusiast
ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.
 
Posts: 40
Karma: 8604
Join Date: Dec 2012
Location: Germany
Device: Kobo Touch
Hello Davidfor,

I can confirm that the third method, copy-pasting to the KT, works. That's how I manage my ebooks.

I will run some tests on your suspision that the database is not properly closed. I will also take a closer look at my suspision that by manually upgrading the KT Firmware the database version does not agree with the schema expectations of the new firmware.

I'll get back to you.

EDIT:

I have run a couple more tests. As a premise, I took as granted that my database is somehow corrupted or the firmware garbled, so I performed a hard reset (restored firmware to 2.1.0) and then reupdated to 2.4.0. I then deleted the database and had the firmware create a blank one.

First of, the database version after that was 71, confirming that a manual updating in the order 2.1.0 to 2.3.1 to 2.3.2 to 2.4.0 somehow ends up having the database version getting stuck at 69.

Starting with a blank database, I uploaded a series of books with series information and shelves set.

In no case was the KT ejected via calibre during these tests. It was always ejected from the OS using HAL.

Here's what I found:

Database closing

After starting calibre and uploading the books, the database was always closed after the job was completed. That applies to both a running instance of calibre as well as having calibre closed. In all cases, the following was returned when checking for open instances:

Quote:
ichrispa@Cassandra:/> lsof | grep -E "/sqlite/ && /KOBO/"
ichrispa@Cassandra:/>
The database is always closed correctly.

Database corruption

The database corruption scenario is the same as described above. It might however be ebook specific, as it appears to only happen with certain books and not with others. While I was testing around, I used the Harry Potter series this time. No corruption occured. When using A Song of Ice and Fire, the database got busted. I will later try to find out what attribute of an epub triggers this behavior.

I did try to restore my old database by reinserting all values from the backup into the fresh database:

Quote:
ichrispa@Cassandra:/media/KOBOeReader/.kobo> echo .dump | sqlite3 KoboReader-sqlite-20130310.bak \
awk '/INSERT/' | sqlite3 KoboReader.sqlite
Error: near line 1: PRIMARY KEY must be unique
Error: near line 2: column ContentID is not unique
Error: near line 3: column ContentID is not unique
Error: near line 1195: near "file": syntax error
Error: near line 2078: near "file": syntax error
Error: near line 2880: columns volumeId, shortcoverId are not unique
...
Although the "pragma integrity_check;" returns "ok" for the backup, when I tried to "dump" the table contents of the backup into the fresh database, numerous errors occured. That includes mutliple instances of primary key violations and non unique values where required errors. However this might be attributed to the different database version, since the backup was version 69 and the fresh instance was version 71.

Setting series/shelve information

Here's the strange part. The behavior of this is very similar to what I described in my previous test: The books transfered show up, but the series/shelves information appears not to be set. Shelves appear empty if newly created.

Since I did not eject using calibre I inspected the changes the driver made to the database.

After adding the books using calibre, the contents table does not include the series and bookshelve information. In fact, the books are not present in the contents table at all!

After unmounting/disconnecting the device the KT goes into the "Processing contents" mode. No shelves or series information show up. When reconnecting the KT and reinspecting the database, the books are in the contents database, but have (obviously) no series/shelves information set.

Restarting calibre and running "Tranfer to device" again (on the books already on the device), then disconnecting the KT, results in skipping the "Processing contents" dialog, since the books are already in the database.

After that, both shelve information and series are shown properly.

(My) conclusion

I cannot conclude to what the database corruption is attributed. But it appears to be related to specific epubs.

When the KT processes new content, all information is derived from the metadata included in the epub. Any manual changes to the database are effectively overwritten by "Processing contents". Only if an ebook is already in the database, the series and shelves information are kept.

I would conclude that the driver does not add entries to the database when transferring books. It only runs UPDATE, not INSERT. Only if a book is already in the database by having run "Processing Contents" the driver is capable to set the proper information.

ReEDIT:

While finishing up the tests I made a little mistake that proved to be very conclusive: I deleted all file/folders on the kobo, then unplugged it.

Since there was no database present and no more contents to be scanned, the KT should not show any books. However, the contents show by the KT where identical to what was there before I accidentally deleted everything. Books, shelves, metainfo - everything was accessable.

I would deduce that the KT does not close its handle of the database when connecting to a PC.

Last edited by ichrispa; 03-11-2013 at 06:39 AM.
ichrispa is offline   Reply With Quote
Advert
Old 03-11-2013, 09:16 AM   #126
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 24,908
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
Quote:
Originally Posted by ichrispa View Post
I can confirm that the third method, copy-pasting to the KT, works. That's how I manage my ebooks.

I have run a couple more tests. As a premise, I took as granted that my database is somehow corrupted or the firmware garbled, so I performed a hard reset (restored firmware to 2.1.0) and then reupdated to 2.4.0. I then deleted the database and had the firmware create a blank one.

First of, the database version after that was 71, confirming that a manual updating in the order 2.1.0 to 2.3.1 to 2.3.2 to 2.4.0 somehow ends up having the database version getting stuck at 69.
That's strange that it didn't happen. I have been through this with the beta firmware and it always updates. The differences between the two database versions are a new table, "SyncQueue" which is only used for the new archiving function, and in "user" renaming column "DeviceID" to "___DeviceID". I can see the latter causing problems, but I would have expected errors like not opening books at all.
Quote:
Starting with a blank database, I uploaded a series of books with series information and shelves set.

In no case was the KT ejected via calibre during these tests. It was always ejected from the OS using HAL.

Here's what I found:

Database closing

After starting calibre and uploading the books, the database was always closed after the job was completed. That applies to both a running instance of calibre as well as having calibre closed. In all cases, the following was returned when checking for open instances:

The database is always closed correctly.
That makes me happy
Quote:

Database corruption

The database corruption scenario is the same as described above. It might however be ebook specific, as it appears to only happen with certain books and not with others. While I was testing around, I used the Harry Potter series this time. No corruption occured. When using A Song of Ice and Fire, the database got busted. I will later try to find out what attribute of an epub triggers this behavior.
I don't think you mentioned that some books didn't cause problems. I had was think it was all. But, someone reported something similar to today. I'm waiting to get a copy of the book to look at.
Quote:

I did try to restore my old database by reinserting all values from the backup into the fresh database:

Although the "pragma integrity_check;" returns "ok" for the backup, when I tried to "dump" the table contents of the backup into the fresh database, numerous errors occured. That includes mutliple instances of primary key violations and non unique values where required errors. However this might be attributed to the different database version, since the backup was version 69 and the fresh instance was version 71.
That is interesting. Can you send me a copy of the database?
Quote:

Setting series/shelve information

Here's the strange part. The behavior of this is very similar to what I described in my previous test: The books transfered show up, but the series/shelves information appears not to be set. Shelves appear empty if newly created.

Since I did not eject using calibre I inspected the changes the driver made to the database.

After adding the books using calibre, the contents table does not include the series and bookshelve information. In fact, the books are not present in the contents table at all!

After unmounting/disconnecting the device the KT goes into the "Processing contents" mode. No shelves or series information show up. When reconnecting the KT and reinspecting the database, the books are in the contents database, but have (obviously) no series/shelves information set.

Restarting calibre and running "Tranfer to device" again (on the books already on the device), then disconnecting the KT, results in skipping the "Processing contents" dialog, since the books are already in the database.

After that, both shelve information and series are shown properly.
That is how it works. Adding the book to the devices database is left up to the device. It adds a set of rows to the contents and the volume_shortcovers tables. The driver doesn't create these rows as I am not 100% sure what to put in them. Plus, Kobo has changed them a few times over the year I have been looking at this. Because of this, setting the series info is done on the next connect to calibre. You don't need to resend the book, just connect.

The shelves should have been set when the book was sent. This is info in different tables and is simple to maintain. I will have to recheck this in case something changed.

The automatic bits of this happen if "Metadata management" is set to automatic. If it is either manual or on send, it only happens when the book is sent. That would need a resend when the device is connected again.
Quote:

(My) conclusion

I cannot conclude to what the database corruption is attributed. But it appears to be related to specific epubs.

When the KT processes new content, all information is derived from the metadata included in the epub. Any manual changes to the database are effectively overwritten by "Processing contents". Only if an ebook is already in the database, the series and shelves information are kept.

I would conclude that the driver does not add entries to the database when transferring books. It only runs UPDATE, not INSERT. Only if a book is already in the database by having run "Processing Contents" the driver is capable to set the proper information.
Yep.
Quote:
ReEDIT:

While finishing up the tests I made a little mistake that proved to be very conclusive: I deleted all file/folders on the kobo, then unplugged it.

Since there was no database present and no more contents to be scanned, the KT should not show any books. However, the contents show by the KT where identical to what was there before I accidentally deleted everything. Books, shelves, metainfo - everything was accessable.

I would deduce that the KT does not close its handle of the database when connecting to a PC.
I haven't done that, but I have seen the same effect. I have seen something like that when the database was corrupted. The device worked OK, I connected to the PC and calibre reported the corrupt database. I ejected and the device kept working OK until I powered off. On restart, tried to read the corrupt database. Sometimes, it showed old data, sometimes it reset the database and prompted to do the set-up.

With your "experiment" it makes me thing that when connected to the PC, the Kobo application is still in memory including the contents of the database. When the disconnect happens, if for some reason it can't read the database, it just continues with the in-memory data. That is a theory at least.

In any case, the important thing is your Touch is working. And it should continue working.
davidfor is offline   Reply With Quote
Old 03-11-2013, 05:51 PM   #127
ichrispa
Enthusiast
ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.
 
Posts: 40
Karma: 8604
Join Date: Dec 2012
Location: Germany
Device: Kobo Touch
Hello Davidfor,

first of all I apologize for not realizing that updating the shelves and series info is a two phase process. Given how volatile the database is across different firmware versions, I fully understand and support you design choice. It's quite reasonable to do it like that, I just didn't expect it.

Quote:
I don't think you mentioned that some books didn't cause problems. I had was think it was all. But, someone reported something similar to today. I'm waiting to get a copy of the book to look at.
How would you like it send to you? I have isolated a wonderful speciment: Clean and freshly created DB, upload the book and it appears on the home screen. Upload it again and the database is broken (becomes noticable after a reboot).

It is imperative that if no "processing contents" screen appears, a reboot of the KT is done. Otherwise DB corruption or even valid changes will not become visible. That should be kind of obvious and I suppose it is old news to you, but it took me some time to figure out the sqlite caching problem before I came up with that one.

Quote:
That is interesting. Can you send me a copy of the database?
Certainly. Sending it to you via PM right away.

Quote:
The shelves should have been set when the book was sent. This is info in different tables and is simple to maintain. I will have to recheck this in case something changed.

The automatic bits of this happen if "Metadata management" is set to automatic. If it is either manual or on send, it only happens when the book is sent. That would need a resend when the device is connected again.
Yes, shelves are created. However the shelves only appear empty after a reboot. And they are only filled with books after the second pass sync

My database management is set to manual right now, because fully automatic updating usually screws up my kobo as soon as it is connected. However updating on send is also a viable alternative. Now that I found out how the driver works, the option becomes much more understandable.

Quote:
With your "experiment" it makes me thing that when connected to the PC, the Kobo application is still in memory including the contents of the database. When the disconnect happens, if for some reason it can't read the database, it just continues with the in-memory data. That is a theory at least.
I believe that it drops the in memory content/sqlite journal when filesystem changes are detected (processing contents) after the device is disconnected. If it cannot detect changes, it appears indeed that it continues using a cached version of the database. Maybe if one could "induce" a trivial filesystem change, such as altering attributes (access/create time?) of an ebook, this behavior could be inhibited.

In any case I will send you the ebook and the database right away. I have a genuine interest in getting the driver to work with my KT, however if you believe this is an isolated problem (as in error 40: "the error is setting 40cm from the screen") I won't take up any more of your time



EDIT:
While describing the procedure for how to "break" the KT using that specific epub in my PM to you, the solution to the problem dawned to me... here's what I believe is happening: The problem is neither the database version nor some specific epub, it is the in cache database instance.

Roughly described, the KT is always using a cached DB instance. Processing contents only updates relevant information, not the entire database.

To demonstrate, here's what I wrote in the PM:
  1. Remove all ebooks and database from KT.
  2. Copy clean/empty/registered database to KT (registered as foo, otherwise as created by reset KT).
  3. Disconnect & reset KT.
  4. Send epub to Main memory.
  5. Eject KT.

At this point, you can see the epub on the home screen without any series information. The shelve is also not visible, although the table that defines shelves is unrelated to the contents processed by the KT and the database entry is there.

As I understand it, there is currently a cached DB version in the KT. The cache itself and parts of the file where updated durch the "processing contents" phase - other parts of the database, such as the shelves, where not updated and are still served from the cache. That explains why the new shelf is not visible yet present in the DB. However the cache and the database are not strictly speaking contradictory, they are just out of sync.

There are now two ways to proceed:
1) Reconnect to calibre straight away again and attempt to update the database again.
2) If you reboot the KT, the shelve appears - ie the cache is flushed to the file and the file reloaded. Cache and DB are now consistent again.

If you choose (1):
When reuploading the book or updating the database, the discrepancy between the cache and the file becomes contradictory. The database is thus corrupted.

If you choose (2):
When reuploading the book or updating the database, Calibre updates the "correct" database. Again: some information is cached and does not become visible until the KT rebooted again.

I did manage to confirm this uploading a single ebook using method 2 to a fresh and clean database. Uploading the same ebook with (1) fails. The procedure still always fails when attempting (2) with multiple ebooks at once, resulting in a corrupted DB after the first reboot. The manifestation of this problem is that the books uploaded before the reboot are all no longer visible (thought the shelves are).

Does that make any sense?

Last edited by ichrispa; 03-11-2013 at 07:22 PM.
ichrispa is offline   Reply With Quote
Old 03-13-2013, 02:56 AM   #128
TicklishOwl
Member
TicklishOwl began at the beginning.
 
TicklishOwl's Avatar
 
Posts: 16
Karma: 10
Join Date: Oct 2012
Location: Chicago
Device: Kobo Aura, Kobo Touch
Shortlist and Series info

Thanks bunches to the Kobo driver devs, you've helped my Glo become the device I wanted it to be!

I was wondering if it's possible to preserve the Shortlist and have Series info. I have metadata management set to Auto, for the Series info. Best feature ever!

But, every time I connect my Glo, the Shortlist is reset. I know this is because I don't tag 'Shortlist' in Calibre. I only add books to the Shortlist on the Glo; usually my current read and a few that are up next.

If I set metadata management to Manual, the Shortlist is preserved, but then Series info isn't available. I'm like a dog chasing its tail!

I'd be forever grateful to have an option to leave the Shortlist unchanged, while still allowing for series info.
TicklishOwl is offline   Reply With Quote
Old 03-13-2013, 08:14 AM   #129
davidfor
Grand Sorcerer
davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.davidfor ought to be getting tired of karma fortunes by now.
 
Posts: 24,908
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
Part of the problem is that when Kobo introduced the shelves, the Shortlist stopped being special. It is just a shelf. Putting in code to handle it differently is a problem. It is just as likely that someone will be maintaining a shortlist in the calibre library. And someone else will want it named something else.

But, having the ability to block a shelf from being changed by the driver isn't an unreasonable idea. One of the problems with this is the user interface for setting it. The simplest is a single name. Or maybe a comma separated list of names of shelve to ignore. And, what to do if a book has one of these shelf names in the shelf column.
davidfor is offline   Reply With Quote
Old 03-13-2013, 06:18 PM   #130
TicklishOwl
Member
TicklishOwl began at the beginning.
 
TicklishOwl's Avatar
 
Posts: 16
Karma: 10
Join Date: Oct 2012
Location: Chicago
Device: Kobo Aura, Kobo Touch
I suspected it was something like that. I miss the old, special Shortlist on the home screen.

The ability to protect user defined shelves would be nice. Although, I haven't seen anyone else asking for that feature, so perhaps the complexity of implementing it outweighs the usefulness.

For now, I will make an effort to remember to quickly tag my current Shortlist in Calibre before I connect the Glo or Touch.
TicklishOwl is offline   Reply With Quote
Old 03-14-2013, 08:56 AM   #131
ichrispa
Enthusiast
ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.
 
Posts: 40
Karma: 8604
Join Date: Dec 2012
Location: Germany
Device: Kobo Touch
Based on my assumption, I believe I have identified both the problem and a valid workaround.

I monitored the behavior of nickel during a USB connection in relation to the sqlite database. The problem is that nickel does not reliquish its control over the sqlite database during a usb connection, keeping the file open. Note that after disconnecting the USB nickel actually closes and reopens the database.

EDIT:
Nickel only closes the DB if it "processes contents". Otherwise a reboot is necessary to get it reopened. sry about that
END EDIT

The problem with this behavior is apparent: SQLite is a single process database implementing filesystem level locks. Obviously on the used FAT filesystem, that does not work properly. The KT driver in calibre should never write to the database at all, as it is opened by nickel while the usb is connected and any UPDATE/INSERT commands risk corrupting the database.

Linux uses filedescriptors based on FS handles. The thing exploited here is that the handle stays the same, even if the filename changes, since the filename is only an attribute of the FS handle. As nickel opens the database before the USB connection is established, it does not really care what the file is called after that - it is safe to rename it (though not to move it to another directory, just renaming). So if the KT driver actually writes to a different SQLite database that is not open (but named KoboReader.sqlite), the problem is solved.

Heres the workaround:
  • Connect the KT to the computer. Do not start calibre!
  • Rename the KoboReader.sqlite database to something like KoboReader.sqlite.open. The filedescriptor held by nickel will be redirected to KoboReader.sqlite.open, so we don't actually care what nickel writes back.
  • Copy KoboReader.sqlite.open to KoboReader.sqlite. This is the file instance manipulated by calibre.
  • Start calibre and have the metadata uploaded.
  • Disconnect and restart the KT (forces nickel to open the KoboReader.sqlite instead of the KoboReader.sqlite.open).
  • Optionally close calibre, reconnect the KT to the computer and remove KoboReader.sqlite.open.

That way the database updated by the KT driver is closed and not being written to by nickel upon disconnecting the device.

Please note: I have only tested this procedure for updating metadata. But I suppose it should also work with sending ebooks to the device, as long the procedure is repeated before every calibre interaction with the database.

Last edited by ichrispa; 03-14-2013 at 10:41 AM.
ichrispa is offline   Reply With Quote
Old 03-14-2013, 09:31 AM   #132
PeterT
Grand Sorcerer
PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.PeterT ought to be getting tired of karma fortunes by now.
 
PeterT's Avatar
 
Posts: 12,119
Karma: 73448614
Join Date: Nov 2007
Location: Toronto
Device: Nexus 7, Clara, Touch, Tolino EPOS
Only problem with that is that Kobo Desktop DOES safely update the database on the Kobo.

Also, I and many others have used this driver for aeons with no such similar problem.
PeterT is offline   Reply With Quote
Old 03-14-2013, 10:35 AM   #133
ichrispa
Enthusiast
ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.
 
Posts: 40
Karma: 8604
Join Date: Dec 2012
Location: Germany
Device: Kobo Touch
Quote:
Originally Posted by PeterT View Post
Only problem with that is that Kobo Desktop DOES safely update the database on the Kobo.
I don't know how the Kobo Desktop application does this. If you automate the steps I described above, then any application could do it. I want to use calibre (and improve it by solving problems I encounter).

Quote:
Originally Posted by PeterT View Post
Also, I and many others have used this driver for aeons with no such similar problem.
That does not change the fact that my (relatively new) KT does have these problems, that it does keep the DB opened and that many manipulations of said database while connected via USB do corrupt the DB beyond usability. Maybe it can be attributed to the fact that it is new and uses an up-to-date firmware (as opposed to being in use for aeons).

I don't want my efforts to be misunderstood as criticizing the KT driver or Davidfors work. Neither am I implying that the malfunction are the drivers fault or that every kobo user has this problem. Quite on the contrary in fact: If I thought the driver was not worth its salt I would not have spent this much time trying to get this to work. I want to make the driver work with my KT too... and possibly help others by doing so.

I assume that other users may be having similar problems (though not as severe hopefully) that might be attributed to the same cause. However I fear that most other users will not go about searching for the cause of a problem in the same way as I and will only report "books gone missing" or "KT crashing". If they report it at all.

This is just my analysis of my problem with my KT and I want to notify other users that if they are facing problems, this is how I tried to solve it. In particular I want Davidfor to know that this problem does exists. If I am an isolated case, I am actually glad about it.

In any case: I have found my workaround as previously posted, which closes this case for me
ichrispa is offline   Reply With Quote
Old 03-14-2013, 11:08 AM   #134
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 43,782
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
The Kobo firmware should never be holding files open when they are exposed over USB. If that is indeed the problem, then the solution is simple, Kobo needs to update their firmware to release the db when connected by USB.

You could work around the problem with the steps you describe, but the problem is that there needs to be an automated way to get the Kobo to close an re-open the db. Without that, users would have to remember to reboot the device on every disconnect which is not a workable solution. Probably the Desktop Application does something to force a db close/re-open. If you can figure out what, then the calibre driver could do the same.
kovidgoyal is offline   Reply With Quote
Old 03-14-2013, 11:48 AM   #135
ichrispa
Enthusiast
ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.ichrispa shines like a glazed doughnut.
 
Posts: 40
Karma: 8604
Join Date: Dec 2012
Location: Germany
Device: Kobo Touch
Quote:
Originally Posted by kovidgoyal View Post
The Kobo firmware should never be holding files open when they are exposed over USB. If that is indeed the problem, then the solution is simple, Kobo needs to update their firmware to release the db when connected by USB.
Agreed. But even when nickel reopens the database it does not update the information displayed, suggesting as Davidfor mentioned that parts of the database are actually cached beyond the sqlite database system. As I mentioned I have to restart my KT to see new shelves, even if they are in the DB after a disconnect. Same goes for series information after they are inserted into the db: they are there, but only visible after a reboot.

In any case this is not a problem of either Calibre or the KT driver. It is up to Kobo to fix close the database properly and reload its information after a disconnect.

Quote:
Originally Posted by kovidgoyal View Post
Probably the Desktop Application does something to force a db close/re-open. If you can figure out what, then the calibre driver could do the same.
Apart from monitoring the USB bus itself (as in "capture with oscilloscope"), I have no idea how to intercept USB data from a virtual machine. Linux won't let me tap the system bus interface. And I definetely don't know how to do this in a native windows environment. Since I haven't managed to get Kobo Desktop to run in WINE, that's out of the question too.

Can somebody advise me on how to monitor what an application sends over USB?
ichrispa is offline   Reply With Quote
Reply

Tags
kobo

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
firmware update for kobo netronix device JDC Netronix 1 08-22-2014 05:33 AM
Error communicating with device - Kobo eReader after 0.8.56 update PoignantTuna Devices 3 06-21-2012 05:48 AM
[Device Interface Plugin] Update for Nook Color Driver jmricker Plugins 0 10-22-2011 10:11 AM
Kobo Desktop erased entire device database! - Don't dowload update MrsJoseph Kobo Reader 9 03-23-2011 11:38 AM
Sync Problems- Device status doesn't update to Kobo desktop... Wipes my bookmarks dashto Kobo Reader 1 11-26-2010 01:35 PM


All times are GMT -4. The time now is 04:52 AM.


MobileRead.com is a privately owned, operated and funded community.