[GUI Plugin] Kobo Utilities

[davidfor]

When you disconnect the Kobo ereader from the PC, it looks for changes in the books. Any new books will be processed into the internal database and hence the library. Any books that have been deleted, will have nearly all evidence of them removed from the database. Any changed books will be removed from the database and then processed as a new book.

The behaviour of the last one has changed over time. Early firmware didn't do anything. Current firmware does the above.

For this, what the firmware considers as changed is the file size. That means you could unpack an epub and repack it without any other changes and get a different size. Or open it, change one letter somewhere in the text and repack it and the device will see it as a new book.

Because of this, when you send a book to a Kobo device that is already on the device, the driver will update the files size stored in the database. I chose to do this in the driver because that kept older behaviour, and my impression from discussion here is that how people wanted it to work. And to be honest, it is how I wanted it to work. I frequently edit books I read to fix spelling and layout. As long as I don't fiddle with the internal file structure or the ToC, everything will work OK.

If you make changes to the ToC or the internal file structure, the book can probably still be read, but, there might be problems. One of the things that the processing of the book does is to store the ToC in the database. If you open the ToC when reading the book, the database is used. If it doesn't match the book, selecting an entry will probably take you to the start of the book.

If the file structure changes, again the book can probably be read. But, the ToC won't work and the reading position won't be stored properly. That means it will probably reopen to the start of the book.

None of this has anything to do with the utilities plugin. It's how the device works and how the driver works.

[iostrym]

Hi, thanks for your precise answer.

you said :

" I chose to do this in the driver because that kept older behaviour, and my impression from discussion here is that how people wanted it to work."

If this is not related to kobo utilities plug-in, then why are you implicated in the driver ? What driver are we talking about ? windows driver ? calibre driver to access kobo ?

[PeterT]

davidfor is both one of the co-authors of the Kobo calibre device driver, the author of Kobo Utilities, and is involved with a myriad of other plugins too.

Probably he has more knowledge of the Kobos and their database than anyone outside of Kobo itself.

[iostrym]

OK it was what i was thinking about but wasn't sure. Thanks for explanation. So we are talking about calibre kobo driver :-)

But there is one point that I'm missing and I'm sorry because it is then no more related to kobo utilities. Why it is important that kobo don't refresh it database when the same (modified or not) epub is loaded. For processing time ? Or for another purpose ?

And big question, Is the behavior you described with epub the same with PDF ?

[jackie_w]

Originally Posted by PeterT

...Probably he has more knowledge of the Kobos and their database than anyone outside of Kobo itself...

... I wouldn't be surprised if it was also many inside Kobo itself

[davidfor]

Originally Posted by iostrym

OK it was what i was thinking about but wasn't sure. Thanks for explanation. So we are talking about calibre kobo driver :-)

Yes, this is about the KoboTouch driver plugin that is part of calibre.

But there is one point that I'm missing and I'm sorry because it is then no more related to kobo utilities. Why it is important that kobo don't refresh it database when the same (modified or not) epub is loaded. For processing time ? Or for another purpose ?

Strictly, there is no reason to reprocess unchanged books and because it takes a long time. If you add 100 books to the device, it takes at least ten minutes. If you have 100 books on the device and add 1 more but all the the books get reprocessed, then you have ten wasted minutes until the device is ready to use. I have about 1400 books on my H2O, so don't want all of them reprocessed each time I add a book. Especially if I'm disconnecting the H2O intending to read for a little bit before bed.

And big question, Is the behavior you described with epub the same with PDF ?

It is the same for all book types.

[iostrym]

Thanks a lot.

So, kobo driver modifie filesize so that kobo don't see the epub/pdf as modified to save kobo processing time. But what if epub is really modified, kobo driver can't know it so we have to remove it manually from kobo, disconnect and then reconnect and add the new one. right ?
And what if the new modified epub is added in calibre as the new epub (with same title and author as the old one, toc may be modified, and metadata that indicate version number), is the behavior the same ? Because I do modification in pdf and epub outside calibre and then add the new modified file in calibre.
do you prefer me to ask those question in private messages so that this thread is not "polluted" ?

[davidfor]

Originally Posted by iostrym

So, kobo driver modifie filesize so that kobo don't see the epub/pdf as modified to save kobo processing time. But what if epub is really modified, kobo driver can't know it so we have to remove it manually from kobo, disconnect and then reconnect and add the new one. right ?
And what if the new modified epub is added in calibre as the new epub (with same title and author as the old one, toc may be modified, and metadata that indicate version number), is the behavior the same ? Because I do modification in pdf and epub outside calibre and then add the new modified file in calibre.

It's a choice. The firmware uses the full file path to identify a book. If there is a file there, it is assumed to be the same book. But, if the file size changes the book is reprocessed. It doesn't matter where the replacement copy comes from and it doesn't matter what is in the file. If it is the same size as in the database, it's left alone. If the file size is changed, it is completely reprocessed.

The driver updates the file size when the book is replaced so that it won't be removed. It doesn't matter what was done with the book. It could have been edited using the calibre editor, Sigil or something else. Or a new copy downloaded from somewhere. Doing this is a risk. If the book has changed to much, then it could cause problems. But, if it isn't done, there is no way to replace a book and not lose the reading status, bookmarks and collections. You can get around this by removing the book first.

do you prefer me to ask those question in private messages so that this thread is not "polluted" ?

This is much better in the forum where others can see it and join in if they want to. Though it would be more appropriate for the KoboTouch driver thread.

[iostrym]

Hi, thanks a lot this is pretty clear.
So concerning books that are not modified but only tags modified in calibre, If i want the creation in kobo of collection for new tag, I only have to reload to the kobo all books. I will have a to wait USB transfer (lot of PDF) but maybe no processing time in the kobo and no reading status deletion. Right ? There is nothing in kobo driver that detect that Filesize is the same in database and in calibre so there is no need to send the file again ?
By the way where is the thread concerning kobo driver? In this forum ? In another one ? I believe I must check kobo driver support somewhere before upgrading kobo firmeware...

[davidfor]

Originally Posted by iostrym

Hi, thanks a lot this is pretty clear.
So concerning books that are not modified but only tags modified in calibre, If i want the creation in kobo of collection for new tag, I only have to reload to the kobo all books.

The driver does the management of collections independently of sending books. Assuming the configuration is correct, each time you connect, it checks and updates the collections as appropriate. So, if you are using tags for the collections and change them in the library, they will automatically be created or deleted on the device at the next connection.

I will have a to wait USB transfer (lot of PDF) but maybe no processing time in the kobo and no reading status deletion. Right ? There is nothing in kobo driver that detect that Filesize is the same in database and in calibre so there is no need to send the file again ?

Comparing the file size of the book in the library with that on the device doesn't really work. One of the things that calibre does when a book is sent to the device is to update the metadata to the latest in the library. This means that the file size will probably change during the send. All the KoboTouch driver does is update the file size in the database to the same as the file after it has been copied to the device.

By the way where is the thread concerning kobo driver? In this forum ? In another one ? I believe I must check kobo driver support somewhere before upgrading kobo firmeware...

The driver thread is https://www.mobileread.com/forums/sho...d.php?t=192863. It doesn't get much action as most of the discussion happens elsewhere when someone is having a problem. I usually only push things there that are very specific to the driver. Plus any announcements of changes in function.

As to firmware upgrades, I keep the driver up to date and will announce anything problems as soon as possible after the firmware is released. That usually happens on the same day of the release. Part of why I can do this is that I am on the firmware beta testing list and get them to test. I test this with calibre and look for anything new we might want to use. The other reason is that Kobo hasn't made many changes recently that affect how calibre interacts with the device. Generally, I test and bump the supported firmware version number in the driver and that's it.

[iostrym]

thanks a lot for so much details.

[davidfor]

I have just updated the plugin to version 2.0.6. This has two small changes

• Fix: Handle missing files in the backup properly.
• Change: Add select/clear all buttons to reading status update dialog.

As usual, calibre will notify you of the update in the next hour or so. And if there are any problems, please report them here.

[Grady Cole]

Hi David, I just updated calibre to 2.26, and the Kobo Utilities plugin to 2.0.6 and received the following error when backing up the database and the page positions:

[Errno 32] Broken pipe

Traceback (most recent call last):
  File "site-packages/calibre/gui2/device.py", line 87, in run
  File "calibre_plugins.koboutilities.jobs", line 145, in do_device_database_backup
  File "calibre_plugins.koboutilities.jobs", line 42, in debug_print
  File "site-packages/calibre/utils/logging.py", line 154, in prints
  File "site-packages/calibre/utils/logging.py", line 45, in prints
  File "site-packages/calibre/__init__.py", line 227, in prints
IOError: [Errno 32] Broken pipe

I ran a check on the database:

Result of running 'PRAGMA integrity_check' on database on the Kobo device:


ok

Any idea what happened, or how it can be fixed?


Ugh... it may be more of a problem. Tried to transfer a book to the Kobo and got this:

calibre, version 2.26.0
ERROR: Error: Error communicating with device

Failed to find obfuscation key

Traceback (most recent call last):
  File "site-packages/calibre/gui2/device.py", line 87, in run
  File "site-packages/calibre/gui2/device.py", line 584, in _upload_books
  File "calibre_plugins.kobotouch_extended.device.driver", line 285, in upload_books
  File "site-packages/calibre/devices/kobo/driver.py", line 1913, in upload_books
  File "calibre_plugins.kobotouch_extended.device.driver", line 179, in _modify_epub
  File "site-packages/calibre/ebooks/oeb/polish/container.py", line 977, in __init__
  File "site-packages/calibre/ebooks/oeb/polish/container.py", line 1093, in process_encryption
ObfuscationKeyMissing: Failed to find obfuscation key

[davidfor]

Originally Posted by Grady Cole

Hi David, I just updated calibre to 2.26, and the Kobo Utilities plugin to 2.0.6 and received the following error when backing up the database and the page positions:

[Errno 32] Broken pipe

Traceback (most recent call last):
  File "site-packages/calibre/gui2/device.py", line 87, in run
  File "calibre_plugins.koboutilities.jobs", line 145, in do_device_database_backup
  File "calibre_plugins.koboutilities.jobs", line 42, in debug_print
  File "site-packages/calibre/utils/logging.py", line 154, in prints
  File "site-packages/calibre/utils/logging.py", line 45, in prints
  File "site-packages/calibre/__init__.py", line 227, in prints
IOError: [Errno 32] Broken pipe

Sorry, that's my fault. I've been trying to get the logging working better and it looks like I left an extra line in. I'll fix it tonight. But, what OS are you running? And 32bit or 64bit? That's so I can see if I can work out why it works for me, but not you.

I ran a check on the database:

Result of running 'PRAGMA integrity_check' on database on the Kobo device:


ok

Any idea what happened, or how it can be fixed?

Ugh... it may be more of a problem. Tried to transfer a book to the Kobo and got this:

calibre, version 2.26.0
ERROR: Error: Error communicating with device

Failed to find obfuscation key

Traceback (most recent call last):
  File "site-packages/calibre/gui2/device.py", line 87, in run
  File "site-packages/calibre/gui2/device.py", line 584, in _upload_books
  File "calibre_plugins.kobotouch_extended.device.driver", line 285, in upload_books
  File "site-packages/calibre/devices/kobo/driver.py", line 1913, in upload_books
  File "calibre_plugins.kobotouch_extended.device.driver", line 179, in _modify_epub
  File "site-packages/calibre/ebooks/oeb/polish/container.py", line 977, in __init__
  File "site-packages/calibre/ebooks/oeb/polish/container.py", line 1093, in process_encryption
ObfuscationKeyMissing: Failed to find obfuscation key

That's in the extended driver. It looks like the epub has embedded fonts that have been obfuscated. The extended driver is opening the epub and uses other calibre code to do this. That is failing while trying to find the key to unobfuscate the font. I think it means the DRM hasn't been removed properly from the epub. Or one of the internal files is damaged. I think you need to find the original file and removed the DRM again to fix this.

[Grady Cole]

I'm on 10.8.5, and I'm running in 64-bit.

Also just got this error when I opened the book in the calibre reader:

calibre, version 2.26.0
ERROR: Unhandled exception: <b>IOError</b>:[Errno 32] Broken pipe

calibre 2.26  isfrozen: True is64bit: True
Darwin-12.6.0-x86_64-i386-64bit Darwin ('64bit', '')
('Darwin', '12.6.0', 'Darwin Kernel Version 12.6.0: Wed Dec 17 19:11:40 PST 2014; root:xnu-2050.48.15~1/RELEASE_X86_64')
Python 2.7.9
OSX: ('10.8.5', ('', '', ''), 'x86_64')
Successfully initialized third party plugins: DeDRM && KePub Output && Kobo Utilities && KoboTouchExtended && Modify ePub
Traceback (most recent call last):
  File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 209, in main
    return run_entry_point()
  File "/Applications/calibre.app/Contents/Resources/Python/lib/python2.7/site.py", line 114, in run_entry_point
    return getattr(pmod, func)()
  File "site-packages/calibre/utils/ipc/worker.py", line 199, in main
IOError: [Errno 32] Broken pipe

It's all falling apart! hahaha