03-01-2013, 08:16 PM | #121 |
Grand Sorcerer
Posts: 24,907
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
|
There isn't any date checking going on in the driver. It simple copies the covers based on the options. Pre 0.9.21, if you had the "cover upload" option but not the "always upload" option, the covers were not sent when you first sent the book.
As PeterT said, each of the 2.x.x firmware versions have had a bug with the covers. As you are on 2.3.1, the wrong cover is used if you open a new book and let the device go to sleep before returning to the home screen. If you return to the home screen before the device goes to sleep, the correct cover will be used. |
03-08-2013, 05:04 AM | #122 |
Grand Sorcerer
Posts: 24,907
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
|
Kobo collection changes in calibre 0.9.22
With Kobo adding Archiving of kepubs in firmware 2.4.0, the calibre driver will now display this as a collection in the device list. Along with this I have done some big fixing on the Kobo collections.
The changes include:
If you currently have recommendations appearing on a shelf, to remove them:
|
Advert | |
|
03-09-2013, 09:40 AM | #123 |
Enthusiast
Posts: 40
Karma: 8604
Join Date: Dec 2012
Location: Germany
Device: Kobo Touch
|
Hello Davidfor,
I have my Kobo back and will try to run a complete check to find out where exactly the problem with updating the onboard database lies. Initial test conditions I will start by establishing that the initial database is ok before connecting to the PC. Code:
kobo login: root [root@kobo /]# cd /mnt/onboard/.kobo/ [root@kobo .kobo]# echo "pragma integrity_check;" | sqlite3 KoboReader.sqlite ok [root@kobo .kobo]# A backup of that database version is present on the KT. Given that the original DB is ok, I updated my calibre to version 0.9.22. Driver setting are:
The KT Firmware Version is 2.4.0, database version is 69. PC Connection I now connect the KT to the PC using the USB cable and mount the onboard FS. I will establish that the filesystem is ok... Code:
Cassandra:/home/ichrispa/bin # fsck /dev/sdd fsck from util-linux-ng 2.17.2 dosfsck 3.0.9, 31 Jan 2010, FAT32, LFN /dev/sdd: 6742 files, 127647/348630 clusters Code:
Cassandra:/home/ichrispa/bin # echo "pragma integrity_check;" | sqlite3 /media/KOBOeReader/.kobo/KoboReader.sqlite ok Cassandra:/home/ichrispa/bin # Calibre correctly recognizes the device, mounts it and the driver scans the database. The contents of the main memory are shown correctly (no sd card is being used). Rechecking the database still yields no database errors, everything is still ok. Database manipulation I will now copy a couple of ebooks to main memory. Specifically, I will copy the Song of Ice and Fire books by George R.R. Martin, as these have series information and shelves set. The books are in ebup format. I have reconverted them from epub to epub to make sure that they do not contain formatting errors. Metadata information is correctly displayed both by calibre as well as by the ebook-viewer and FBReader. Sending the ebooks to main memory completes with no errors. Each book was send seperately, in contrast to selecting all books and then clicking "send to device". Database integrity check is still ok. Code:
Cassandra:/home/ichrispa/bin # echo "pragma integrity_check;" | sqlite3 /media/KOBOeReader/.kobo/KoboReader.sqlite ok Cassandra:/home/ichrispa/bin # The OS (Linux) confirms the FS has been unmounted: Code:
Cassandra:/home/ichrispa/bin # mount | grep KOBO Cassandra:/home/ichrispa/bin # Effects on device Disconnecting the KT takes some time for scanning the new contents, then the device shows all uploaded covers on the home screen. Series information is only set for the first book in the calibre list (Series book 5). A shelve has been created, but only the first book of the series has been added to it. Checking the onboard database yields... Code:
[root@kobo .kobo]# echo "pragma integrity_check;" | sqlite3 KoboReader.sqlite *** in database main *** On tree page 1758 cell 1: Rowid 2782 out of order (min less than parent min of 2782) rowid 1689 missing from index content_date_last_read_index wrong # of entries in index content_mime_type wrong # of entries in index content_bookid_index wrong # of entries in index content_date_last_read_index wrong # of entries in index content_title_index wrong # of entries in index content_attribution_index wrong # of entries in index sqlite_autoindex_content_1 Conclusion The database is not corrupted from the calibre driver as such. The error is created when nickel scans the modified database after disconnecting the device. If you need them, I can send you both the healthy initial db as well as the broken version via PM. Thanks for your help davidfor, regards, ichrispa |
03-09-2013, 08:46 PM | #124 |
Grand Sorcerer
Posts: 24,907
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
|
That's a comprehensive test and results. Unfortunately, I don't know what is happening. But, it makes it sound like the database might still be opened when calibre ejects the device.
Some things I can think of to try: - After sending the book to the device, close calibre, check the database and then eject using the OS. - After sending the book to the device, close calibre, check the database and then reopen calibre and do the eject from there. - Don't use calibre but manually copy the book and eject using the OS. That might help to work out the cause. |
03-11-2013, 03:07 AM | #125 | ||
Enthusiast
Posts: 40
Karma: 8604
Join Date: Dec 2012
Location: Germany
Device: Kobo Touch
|
Hello Davidfor,
I can confirm that the third method, copy-pasting to the KT, works. That's how I manage my ebooks. I will run some tests on your suspision that the database is not properly closed. I will also take a closer look at my suspision that by manually upgrading the KT Firmware the database version does not agree with the schema expectations of the new firmware. I'll get back to you. EDIT: I have run a couple more tests. As a premise, I took as granted that my database is somehow corrupted or the firmware garbled, so I performed a hard reset (restored firmware to 2.1.0) and then reupdated to 2.4.0. I then deleted the database and had the firmware create a blank one. First of, the database version after that was 71, confirming that a manual updating in the order 2.1.0 to 2.3.1 to 2.3.2 to 2.4.0 somehow ends up having the database version getting stuck at 69. Starting with a blank database, I uploaded a series of books with series information and shelves set. In no case was the KT ejected via calibre during these tests. It was always ejected from the OS using HAL. Here's what I found: Database closing After starting calibre and uploading the books, the database was always closed after the job was completed. That applies to both a running instance of calibre as well as having calibre closed. In all cases, the following was returned when checking for open instances: Quote:
Database corruption The database corruption scenario is the same as described above. It might however be ebook specific, as it appears to only happen with certain books and not with others. While I was testing around, I used the Harry Potter series this time. No corruption occured. When using A Song of Ice and Fire, the database got busted. I will later try to find out what attribute of an epub triggers this behavior. I did try to restore my old database by reinserting all values from the backup into the fresh database: Quote:
Setting series/shelve information Here's the strange part. The behavior of this is very similar to what I described in my previous test: The books transfered show up, but the series/shelves information appears not to be set. Shelves appear empty if newly created. Since I did not eject using calibre I inspected the changes the driver made to the database. After adding the books using calibre, the contents table does not include the series and bookshelve information. In fact, the books are not present in the contents table at all! After unmounting/disconnecting the device the KT goes into the "Processing contents" mode. No shelves or series information show up. When reconnecting the KT and reinspecting the database, the books are in the contents database, but have (obviously) no series/shelves information set. Restarting calibre and running "Tranfer to device" again (on the books already on the device), then disconnecting the KT, results in skipping the "Processing contents" dialog, since the books are already in the database. After that, both shelve information and series are shown properly. (My) conclusion I cannot conclude to what the database corruption is attributed. But it appears to be related to specific epubs. When the KT processes new content, all information is derived from the metadata included in the epub. Any manual changes to the database are effectively overwritten by "Processing contents". Only if an ebook is already in the database, the series and shelves information are kept. I would conclude that the driver does not add entries to the database when transferring books. It only runs UPDATE, not INSERT. Only if a book is already in the database by having run "Processing Contents" the driver is capable to set the proper information. ReEDIT: While finishing up the tests I made a little mistake that proved to be very conclusive: I deleted all file/folders on the kobo, then unplugged it. Since there was no database present and no more contents to be scanned, the KT should not show any books. However, the contents show by the KT where identical to what was there before I accidentally deleted everything. Books, shelves, metainfo - everything was accessable. I would deduce that the KT does not close its handle of the database when connecting to a PC. Last edited by ichrispa; 03-11-2013 at 06:39 AM. |
||
Advert | |
|
03-11-2013, 09:16 AM | #126 | |||||||
Grand Sorcerer
Posts: 24,907
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
|
Quote:
Quote:
Quote:
Quote:
Quote:
The shelves should have been set when the book was sent. This is info in different tables and is simple to maintain. I will have to recheck this in case something changed. The automatic bits of this happen if "Metadata management" is set to automatic. If it is either manual or on send, it only happens when the book is sent. That would need a resend when the device is connected again. Quote:
Quote:
With your "experiment" it makes me thing that when connected to the PC, the Kobo application is still in memory including the contents of the database. When the disconnect happens, if for some reason it can't read the database, it just continues with the in-memory data. That is a theory at least. In any case, the important thing is your Touch is working. And it should continue working. |
|||||||
03-11-2013, 05:51 PM | #127 | ||||
Enthusiast
Posts: 40
Karma: 8604
Join Date: Dec 2012
Location: Germany
Device: Kobo Touch
|
Hello Davidfor,
first of all I apologize for not realizing that updating the shelves and series info is a two phase process. Given how volatile the database is across different firmware versions, I fully understand and support you design choice. It's quite reasonable to do it like that, I just didn't expect it. Quote:
It is imperative that if no "processing contents" screen appears, a reboot of the KT is done. Otherwise DB corruption or even valid changes will not become visible. That should be kind of obvious and I suppose it is old news to you, but it took me some time to figure out the sqlite caching problem before I came up with that one. Quote:
Quote:
My database management is set to manual right now, because fully automatic updating usually screws up my kobo as soon as it is connected. However updating on send is also a viable alternative. Now that I found out how the driver works, the option becomes much more understandable. Quote:
In any case I will send you the ebook and the database right away. I have a genuine interest in getting the driver to work with my KT, however if you believe this is an isolated problem (as in error 40: "the error is setting 40cm from the screen") I won't take up any more of your time EDIT: While describing the procedure for how to "break" the KT using that specific epub in my PM to you, the solution to the problem dawned to me... here's what I believe is happening: The problem is neither the database version nor some specific epub, it is the in cache database instance. Roughly described, the KT is always using a cached DB instance. Processing contents only updates relevant information, not the entire database. To demonstrate, here's what I wrote in the PM:
At this point, you can see the epub on the home screen without any series information. The shelve is also not visible, although the table that defines shelves is unrelated to the contents processed by the KT and the database entry is there. As I understand it, there is currently a cached DB version in the KT. The cache itself and parts of the file where updated durch the "processing contents" phase - other parts of the database, such as the shelves, where not updated and are still served from the cache. That explains why the new shelf is not visible yet present in the DB. However the cache and the database are not strictly speaking contradictory, they are just out of sync. There are now two ways to proceed: 1) Reconnect to calibre straight away again and attempt to update the database again. 2) If you reboot the KT, the shelve appears - ie the cache is flushed to the file and the file reloaded. Cache and DB are now consistent again. If you choose (1): When reuploading the book or updating the database, the discrepancy between the cache and the file becomes contradictory. The database is thus corrupted. If you choose (2): When reuploading the book or updating the database, Calibre updates the "correct" database. Again: some information is cached and does not become visible until the KT rebooted again. I did manage to confirm this uploading a single ebook using method 2 to a fresh and clean database. Uploading the same ebook with (1) fails. The procedure still always fails when attempting (2) with multiple ebooks at once, resulting in a corrupted DB after the first reboot. The manifestation of this problem is that the books uploaded before the reboot are all no longer visible (thought the shelves are). Does that make any sense? Last edited by ichrispa; 03-11-2013 at 07:22 PM. |
||||
03-13-2013, 02:56 AM | #128 |
Member
Posts: 16
Karma: 10
Join Date: Oct 2012
Location: Chicago
Device: Kobo Aura, Kobo Touch
|
Shortlist and Series info
Thanks bunches to the Kobo driver devs, you've helped my Glo become the device I wanted it to be!
I was wondering if it's possible to preserve the Shortlist and have Series info. I have metadata management set to Auto, for the Series info. Best feature ever! But, every time I connect my Glo, the Shortlist is reset. I know this is because I don't tag 'Shortlist' in Calibre. I only add books to the Shortlist on the Glo; usually my current read and a few that are up next. If I set metadata management to Manual, the Shortlist is preserved, but then Series info isn't available. I'm like a dog chasing its tail! I'd be forever grateful to have an option to leave the Shortlist unchanged, while still allowing for series info. |
03-13-2013, 08:14 AM | #129 |
Grand Sorcerer
Posts: 24,907
Karma: 47303748
Join Date: Jul 2011
Location: Sydney, Australia
Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos
|
Part of the problem is that when Kobo introduced the shelves, the Shortlist stopped being special. It is just a shelf. Putting in code to handle it differently is a problem. It is just as likely that someone will be maintaining a shortlist in the calibre library. And someone else will want it named something else.
But, having the ability to block a shelf from being changed by the driver isn't an unreasonable idea. One of the problems with this is the user interface for setting it. The simplest is a single name. Or maybe a comma separated list of names of shelve to ignore. And, what to do if a book has one of these shelf names in the shelf column. |
03-13-2013, 06:18 PM | #130 |
Member
Posts: 16
Karma: 10
Join Date: Oct 2012
Location: Chicago
Device: Kobo Aura, Kobo Touch
|
I suspected it was something like that. I miss the old, special Shortlist on the home screen.
The ability to protect user defined shelves would be nice. Although, I haven't seen anyone else asking for that feature, so perhaps the complexity of implementing it outweighs the usefulness. For now, I will make an effort to remember to quickly tag my current Shortlist in Calibre before I connect the Glo or Touch. |
03-14-2013, 08:56 AM | #131 |
Enthusiast
Posts: 40
Karma: 8604
Join Date: Dec 2012
Location: Germany
Device: Kobo Touch
|
Based on my assumption, I believe I have identified both the problem and a valid workaround.
I monitored the behavior of nickel during a USB connection in relation to the sqlite database. The problem is that nickel does not reliquish its control over the sqlite database during a usb connection, keeping the file open. Note that after disconnecting the USB nickel actually closes and reopens the database. EDIT: Nickel only closes the DB if it "processes contents". Otherwise a reboot is necessary to get it reopened. sry about that END EDIT The problem with this behavior is apparent: SQLite is a single process database implementing filesystem level locks. Obviously on the used FAT filesystem, that does not work properly. The KT driver in calibre should never write to the database at all, as it is opened by nickel while the usb is connected and any UPDATE/INSERT commands risk corrupting the database. Linux uses filedescriptors based on FS handles. The thing exploited here is that the handle stays the same, even if the filename changes, since the filename is only an attribute of the FS handle. As nickel opens the database before the USB connection is established, it does not really care what the file is called after that - it is safe to rename it (though not to move it to another directory, just renaming). So if the KT driver actually writes to a different SQLite database that is not open (but named KoboReader.sqlite), the problem is solved. Heres the workaround:
That way the database updated by the KT driver is closed and not being written to by nickel upon disconnecting the device. Please note: I have only tested this procedure for updating metadata. But I suppose it should also work with sending ebooks to the device, as long the procedure is repeated before every calibre interaction with the database. Last edited by ichrispa; 03-14-2013 at 10:41 AM. |
03-14-2013, 09:31 AM | #132 |
Grand Sorcerer
Posts: 12,155
Karma: 73448616
Join Date: Nov 2007
Location: Toronto
Device: Nexus 7, Clara, Touch, Tolino EPOS
|
Only problem with that is that Kobo Desktop DOES safely update the database on the Kobo.
Also, I and many others have used this driver for aeons with no such similar problem. |
03-14-2013, 10:35 AM | #133 | ||
Enthusiast
Posts: 40
Karma: 8604
Join Date: Dec 2012
Location: Germany
Device: Kobo Touch
|
Quote:
Quote:
I don't want my efforts to be misunderstood as criticizing the KT driver or Davidfors work. Neither am I implying that the malfunction are the drivers fault or that every kobo user has this problem. Quite on the contrary in fact: If I thought the driver was not worth its salt I would not have spent this much time trying to get this to work. I want to make the driver work with my KT too... and possibly help others by doing so. I assume that other users may be having similar problems (though not as severe hopefully) that might be attributed to the same cause. However I fear that most other users will not go about searching for the cause of a problem in the same way as I and will only report "books gone missing" or "KT crashing". If they report it at all. This is just my analysis of my problem with my KT and I want to notify other users that if they are facing problems, this is how I tried to solve it. In particular I want Davidfor to know that this problem does exists. If I am an isolated case, I am actually glad about it. In any case: I have found my workaround as previously posted, which closes this case for me |
||
03-14-2013, 11:08 AM | #134 |
creator of calibre
Posts: 43,842
Karma: 22666666
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
The Kobo firmware should never be holding files open when they are exposed over USB. If that is indeed the problem, then the solution is simple, Kobo needs to update their firmware to release the db when connected by USB.
You could work around the problem with the steps you describe, but the problem is that there needs to be an automated way to get the Kobo to close an re-open the db. Without that, users would have to remember to reboot the device on every disconnect which is not a workable solution. Probably the Desktop Application does something to force a db close/re-open. If you can figure out what, then the calibre driver could do the same. |
03-14-2013, 11:48 AM | #135 | ||
Enthusiast
Posts: 40
Karma: 8604
Join Date: Dec 2012
Location: Germany
Device: Kobo Touch
|
Quote:
In any case this is not a problem of either Calibre or the KT driver. It is up to Kobo to fix close the database properly and reload its information after a disconnect. Quote:
Can somebody advise me on how to monitor what an application sends over USB? |
||
Tags |
kobo |
Thread Tools | Search this Thread |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
firmware update for kobo netronix device | JDC | Netronix | 1 | 08-22-2014 05:33 AM |
Error communicating with device - Kobo eReader after 0.8.56 update | PoignantTuna | Devices | 3 | 06-21-2012 05:48 AM |
[Device Interface Plugin] Update for Nook Color Driver | jmricker | Plugins | 0 | 10-22-2011 10:11 AM |
Kobo Desktop erased entire device database! - Don't dowload update | MrsJoseph | Kobo Reader | 9 | 03-23-2011 11:38 AM |
Sync Problems- Device status doesn't update to Kobo desktop... Wipes my bookmarks | dashto | Kobo Reader | 1 | 11-26-2010 01:35 PM |