[GUI Plugin] Kobo Utilities

[davidfor]

Originally Posted by jbaumann

Hi David,

thanks again for the hard work with the plug-in.

Two minor questions, that might be of interest for others as well ...

Do the shelves work with the firmware version 3.5.0 again i.e., is it safe to update from 3.1.1, or is it better to stay with that firmware version?

Unfortunately not. You will hear me singing the praises of Kobo when it hapens

The plug-in allows to substitute some general css for every book (has been discussed for orphans and widows). Is there a step-by-step instruction somewhere that can be used?

The KoboTouch driver can do that when sending the book to the device. The instructions are here. You can resend books to the device safely if you don't make any changes to the structure. Changing the CSS, fixing spelling, grammar or code inside the pages is OK. Changes to the structure of the internal files or table of contents can cause problems.

[jbaumann]

Hi David,

thanks for the answers. I will stay with my current firmware version then.

Cheers, Joe

[davidfor]

I have just uploaded version 1.7.2 of the plugin. This has no changes in it except to support the upcoming Qt5 version of calibre. The changes have been made in a way that the plugin can be used with either version of calibre.

Note: I release a beta of the plugin as version 1.201.0. If you installed the beta, you will not be notified of the new version. You can install the released version in the same way you first installed the plugin.

[arspr]

One possible issue (or not) with your plugin, sideloaded kepubs (through KoboTouchExtended) and covers.

Root issue:

• Delete a previously sideloaded kepub. Deletion within Kobo device. (I mean not from Calibre, although it doesn't seem to matter. I've not fully tested this later case).
• Send it again but in a different updated version with also a different cover.
• The tile shown by Kobo is not updated. It shows the old cover version.

Possible problem with your plugin:

• Deletion of the old book (old cover) in the same way, (through Kobo Aura HD GUI).
• Plug AuraHD to PC.
• In Calibre I use your Clean Images Directory (+ Kobo epubs tick) feature.
• It perfectly runs and says that no extra images have been found.
• Send the new version of the book (with new cover).
• Unplug.
• "Processing new content" is shown as expected.
• Voilà, the OLD cover is shown in the tile...

If I want the new cover to be effectively shown I have to follow the next procedure:

• Deletion of the old book (old cover) in the same way, (through Kobo Aura HD GUI).
• Plug AuraHD to PC.
• In Calibre I use your Clean Images Directory (+ Kobo epubs tick) feature.
• It perfectly runs and says that no extra images have been found.
• Unplug.
• Turn off (not sleep but really turn off).
• Turn on and re-plug.
• Send the new version of the book (with new cover).
• Unplug.
• "Processing new content" is shown as expected.
• Now the tile is correctly updated with the new cover.

So:

• Is this the expected behaviour? I mean, is the needed turn off/on cycle normal?
• If affirmative, why does the plugin says "no extra images have been found"?

[PeterT]

I think it has been commented on before that nickle (the main application on the Kobo) caches some items like images in memory.

[davidfor]

arspr: Yes, as PeterT said, this is caused by some caching in nickel. Until something happens to clear an image out of the cache, it doesn't get updated. Unfortunately, deleting a book doesn't remove it's cover from the cache.

[arspr]

Thanks guys,

Fully confirmed. Kobo Utilities has nothing to do with it. If I delete the book, I do a turn on/off cycle and I reload the book with the new cover it is perfectly updated...



I was looking for the complicated explanation (something weird in Kobo Utilities) and, as always, the solution is the simplest and dumbest one: And yet another bug in Kobo! (Why the hell do you keep in memory anything about something that does not exist? More over when, in fact, you have already decided it is useless because if you perform a turn on/off cycle you do not store it in your permanent memory, database or whatever...)

There's an idiom in Spanish, (I suppose in English too), which, literally translated, is more or less: The ways of Lord are inscrutable... And it doesn't only have a religious meaning...

EDITED (TWICE)----
What follows is wrong. Period. But if I delete it you won't understand the next post by davidfor and my following one.

WTF! This is what also happens with annotations!

If you delete your current reading without deleting its annotations and you resend it again those annotations are also cached. But if you perform a turn on/off cycle in the middle they are lost!!!

@davidfor, sometime in the past, I pointed out that behaviour, (saving annotations of books that doesn't exist any more, which could be completely modified when you resend them -- in fact I use annotations/highlights in order to mark the typos I find in the books...), as some kind of bug. And you answered me they could be somehow something GOOD, that some people actually preferred that behaviour. Then if you, (Kobo, I mean), have decided they are good, (a decision I do not agree with, but a decision I could understand), WHY do you throw them away with a turn on/off cycle?

More over, I suppose the memory of the device is limited, so, how many annotations, covers, whatever, do you "temporally" store? Do you only do this for the current reading?, for the last 3 deletions?, till that "caching memory" (whichever it is) is full?, till when?

Then, if the book has not been loaded/opened for a very long time, so the annotations are not in your memory when it is deleted, do you force their loading in order to do this absurd "temporal" storage? Or don't you?

Am I the only one seeing a completely absurd incoherency in the design here? Either you store information or you don't, but playing something in the middle with no clear criteria...

[davidfor]

The two things are very different.

The cover caching is for performance. The device reads covers from disk and keeps them in a cache for a while. It does this the first time the book is accessed. And as the book is likely to be seen for a while as you read it, putting the cover it in the memory cache makes sense. The cover will be removed from the cache when it hasn't been used for a while and another cover needs to be loaded. I don't know how big the cache is. For the older style home screen, it appeared to hold five covers - enough for the home screen. The cache is cleared when the device is powered off as everything in memory is dropped and restarted. Not removing the cover from the cache when deleting a book is a bug. In general, not a serious bug as deleting and putting a book back on the device with a different cover before the cover is removed from the cache is not expected to be a common action.

The bookmarks is different. The device stores bookmarks for epubs in the main memory in two places; the database and an annotations file. The annotations file is created by the RMSDK. ADE creates the same file on the PC. And if you are careful, you can copy the file between the two places. When you delete a book on the device, the bookmarks are deleted from the database, but the file is not deleted. So, if you send a new book to the device with exactly the same file name and path, it will reuse the existing annotations file. As this is a file on the internal SD card, it is not affected by powering the device off. Or at least it wasn't whenever I have tested, including before writing this post.

I don't believe there is the same in-memory caching for the annotations as there is for the covers. I can't see any and advantage in having them in memory when the book isn't opened.

At the moment, there is no way to clean-up the leftover annotation files. Occasionally, I manually delete them. But, I generally do that because of the bug when reading them messes up the location and percent into chapter. Having this as part of the utilities plugin makes sense. I'll add it to my todo list.

[arspr]

As this is a file on the internal SD card, it is not affected by powering the device off. Or at least it wasn't whenever I have tested, including before writing this post.

Oops I'll retest because just before writing mine I made an explicit test (adding a highlight, deleting the book, turn off on cycle and resending the very same book) and the highlight was gone... So I must have made some mistake...


EDITED----

Tests done with sideloaded kepubs (I'll re-make them with epubs)

Sideloaded Kepub.

• Highlight > Delete the book within device > Re-send: Highlight also deleted. (PERFECT)
• Highlight > Delete the book from Calibre > Re-send: Highlight is kept. (A bug in Calibre/KoboTouchExtended? I mean, Calibre should do something more than it is actually doing in order to fully mimic Kobo deletion process).
• Highlight > Delete the book from Calibre > Unplug / re-plug > Re-send: Highlight is kept.
• Highlight > Delete the book from Calibre > Unplug > Turn off/on cycle > Replug > Re-send: Highlight is kept.

So, just for a single time, Kobo is doing the things properly. I don't know about databases, cached memory, files on SD, or whatever, but when a book is killed through the device, their annotations are killed. The left-overs, at least with kepubs, happen with external tools (Calibre).

So as a summary:

• This is a more complex issue than it looks at first glance. I was completely wrong in my previous post.
• Kobo seem to have a bug with covers of deleted books being absurdly cached. But Kobo doesn't seem to have that problem with annotations.
• So it really seems that the same apparent problem with annotations is an external one.

EDITED (2) ------

Now test done with Epubs.

• Highlight in Sideloaded Kepub > Delete the book within device > Re-send (as epub): Highlight also deleted. (PERFECT)
• Highlight > Delete the book within device > Re-send: Highlight is kept.
• ... I haven't tested anymore...

So it really seems that:
Deletion of annotations is "buggy" within epubs, (the technical reason being that file leftover from ADE or whatever), but it works fine within kepubs. But, OTOH, Calibre or its driver should be modified to fully mimic the deletion procedure of the device.

[bernard.ryefield]

After adding a (valid) book in my Kobo Glo I run suddenly in all sorts of problems (freezing) so I deleted the book on the device. Then suddenly all books were unreadable (so the error message said). I run a database check through the KU plugin and there's lots of errors:

Result of running 'PRAGMA integrity_check' on database on the Kobo device:

Spoiler:

*** in database main ***
Page 180330: btreeInitPage() returns error code 11
On tree page 180306 cell 66: Child page depth differs
On tree page 180306 cell 67: Child page depth differs
On tree page 180413 cell 0: 2nd reference to page 329890
On tree page 180413 cell 1: 2nd reference to page 329891
On tree page 180413 cell 2: 2nd reference to page 329892
On tree page 180413 cell 3: 2nd reference to page 329893
On tree page 180413 cell 4: 2nd reference to page 329894
On tree page 180413 cell 5: 2nd reference to page 329895
On tree page 180413 cell 6: 2nd reference to page 329896
On tree page 180413 cell 7: 2nd reference to page 329897
On tree page 180413 cell 8: 2nd reference to page 329898
On tree page 180413 cell 9: 2nd reference to page 329899
On tree page 180413 cell 10: 2nd reference to page 329900
On tree page 180413 cell 11: 2nd reference to page 329901
On tree page 180413 cell 12: 2nd reference to page 329902
On tree page 180413 cell 13: 2nd reference to page 329903
On tree page 180413 cell 14: 2nd reference to page 329904
On tree page 180413 cell 15: 2nd reference to page 329905
On tree page 180413 cell 16: 2nd reference to page 329906
On tree page 180413 cell 17: 2nd reference to page 329907
On tree page 180413 cell 18: 2nd reference to page 329908
On tree page 180413 cell 19: 2nd reference to page 329909
On tree page 180413 cell 20: 2nd reference to page 329910
On tree page 180413 cell 21: 2nd reference to page 329911
On tree page 180413 cell 22: 2nd reference to page 329912
On tree page 180413 cell 23: 2nd reference to page 329913
On tree page 180413 cell 24: 2nd reference to page 329914
On tree page 180413 cell 25: 2nd reference to page 329915
On tree page 180413 cell 26: 2nd reference to page 329916
On tree page 180413 cell 27: 2nd reference to page 329918
On tree page 180413 cell 28: 2nd reference to page 329919
On tree page 180413 cell 29: 2nd reference to page 329920
On tree page 180413 cell 30: 2nd reference to page 329921
On tree page 180413 cell 31: 2nd reference to page 329922
On tree page 180413 cell 32: 2nd reference to page 329923
On tree page 180413 cell 33: 2nd reference to page 329924
On tree page 180413 cell 34: 2nd reference to page 329925
On tree page 180413 cell 35: 2nd reference to page 329926
On tree page 180413 cell 36: 2nd reference to page 329927
On tree page 180413 cell 37: 2nd reference to page 329928
On tree page 180413 cell 38: 2nd reference to page 329929
On tree page 180413 cell 39: 2nd reference to page 329930
On tree page 180413 cell 40: 2nd reference to page 329931
On tree page 180413 cell 41: 2nd reference to page 329932
On tree page 180413 cell 42: 2nd reference to page 329933
On tree page 180413 cell 43: 2nd reference to page 329934
On tree page 180413 cell 44: 2nd reference to page 329935
On tree page 180413 cell 45: 2nd reference to page 329936
On tree page 180413 cell 46: 2nd reference to page 329937
On tree page 180413 cell 47: 2nd reference to page 329938
On tree page 180413 cell 48: 2nd reference to page 329939
On tree page 180413 cell 49: 2nd reference to page 329940
On tree page 180413 cell 50: 2nd reference to page 329941
On tree page 180413 cell 51: 2nd reference to page 329942
On tree page 180413 cell 52: 2nd reference to page 329943
On tree page 180413 cell 53: 2nd reference to page 329944
On tree page 180413 cell 54: 2nd reference to page 329945
On tree page 180413 cell 55: 2nd reference to page 329946
On tree page 180413 cell 56: 2nd reference to page 329947
On tree page 180413 cell 57: 2nd reference to page 329948
On tree page 180413 cell 58: 2nd reference to page 329949
On tree page 180413 cell 59: 2nd reference to page 329950
On tree page 180413 cell 60: 2nd reference to page 329951
On tree page 180413 cell 61: 2nd reference to page 329952
On tree page 180413 cell 62: 2nd reference to page 329953
On tree page 180413 cell 63: 2nd reference to page 329954
On tree page 180413 cell 64: 2nd reference to page 329955
On tree page 180413 cell 65: 2nd reference to page 329956
On tree page 180413 cell 66: 2nd reference to page 329957
On tree page 180413 cell 67: 2nd reference to page 329958
On tree page 180413 cell 68: 2nd reference to page 329959
On tree page 180413 cell 69: 2nd reference to page 329960
On tree page 180413 cell 70: 2nd reference to page 329961
On tree page 180413 cell 71: 2nd reference to page 329962
On tree page 180413 cell 72: 2nd reference to page 329963
On tree page 180413 cell 73: 2nd reference to page 329964
On tree page 180413 cell 74: 2nd reference to page 329965
On tree page 180413 cell 75: 2nd reference to page 329966
On tree page 180413 cell 76: 2nd reference to page 329967
On tree page 180413 cell 77: 2nd reference to page 329968
On tree page 180413 cell 78: 2nd reference to page 329969
On tree page 180413 cell 79: 2nd reference to page 329970
On tree page 180413 cell 80: 2nd reference to page 329971
On tree page 180413 cell 81: 2nd reference to page 329972
On tree page 180413 cell 82: 2nd reference to page 329973
On tree page 180413 cell 83: 2nd reference to page 329974
On tree page 180413 cell 84: 2nd reference to page 329975
On tree page 180413 cell 85: 2nd reference to page 329976
On tree page 180413 cell 86: 2nd reference to page 329977
On tree page 180413 cell 87: 2nd reference to page 329978
On tree page 180413 cell 88: 2nd reference to page 329979
On tree page 180413 cell 89: 2nd reference to page 329980
On tree page 180413 cell 90: 2nd reference to page 329981
On tree page 180413 cell 91: 2nd reference to page 329982
On tree page 180413 cell 92: 2nd reference to page 329983
On tree page 180413 cell 93: 2nd reference to page 329984
On tree page 180413 cell 94: 2nd reference to page 329985
On tree page 180413 cell 95: 2nd reference to page 329986
On tree page 180413 cell 96: 2nd reference to page 331984

Also "KOBO EREADER" appeared at the top of the screen (book's title location).
I have the 3.8.0 Kobo desktop , 3.3.0 FW (no 3.5.0 yet) and most of my books are on a SD card. How do I solve these problems?

[davidfor]

bernard.ryefield: That says the database on the device is corrupt. The only ways to recover are to restore a recent backup of the database or do a logout on the device. With the logout, the device will restart, create a new database and then prompt you to login. It will then process the books on the device back into the database.

[davidfor]

Originally Posted by arspr

So it really seems that:
Deletion of annotations is "buggy" within epubs, (the technical reason being that file leftover from ADE or whatever), but it works fine within kepubs. But, OTOH, Calibre or its driver should be modified to fully mimic the deletion procedure of the device.

Your analysis agrees with mine. I don't remember why I didn't delete the bookmarks from the database when deleting the book in calibre. I will put it on my TODO list. Its a simple change, but I'll probably wait until after calibre v2 has been released and settled down.

[bernard.ryefield]

Originally Posted by davidfor

bernard.ryefield: That says the database on the device is corrupt. The only ways to recover are to restore a recent backup of the database or do a logout on the device. With the logout, the device will restart, create a new database and then prompt you to login. It will then process the books on the device back into the database.

Thanks for the reply. I tried the logout/login option however the database rebuild process is stuck at 66%. I began with the Glo fully charged but I suspect the 20 Go on the SD card takes too long to process even on a fully charged device.

I intend to resume this process this time without the SD card inserted, delete all books on the SD card and add back my 4100+ books, a hundred at a time, through Calibre (I run into another problem previously and remember this is sort of a limit).

As it will take a lot of time, any more convenient option is welcomed.

[chrisridd]

Originally Posted by bernard.ryefield

Thanks for the reply. I tried the logout/login option however the database rebuild process is stuck at 66%. I began with the Glo fully charged but I suspect the 20 Go on the SD card takes too long to process even on a fully charged device.

I intend to resume this process this time without the SD card inserted, delete all books on the SD card and add back my 4100+ books, a hundred at a time, through Calibre (I run into another problem previously and remember this is sort of a limit).

As it will take a lot of time, any more convenient option is welcomed.

All I can suggest is that when it is finished you use Calibre's KoboUtilities plugin to automatically backup the device database, so the next time it gets corrupted you can easily restore a recent copy.

[bernard.ryefield]

Originally Posted by chrisridd

All I can suggest is that when it is finished you use Calibre's KoboUtilities plugin to automatically backup the device database, so the next time it gets corrupted you can easily restore a recent copy.

The last backup was on July 24, but I may be willing to try a restore. I wonder how to do it properly now. The database rebuild will not finish (still 66% after an hour).