[Device Plugin] Kobo Touch Extended Driver

[AnemicOak]

I'm using this driver with my Aura HD and I think today is the first time I've sent a book to it since Calibre was updated to 0.9.42. When I send a book to the device now I get a copy on the device and a second copy of the book is being generated on my computer's desktop in a directory structure of kepub-->author name-->epub.kepub file of book. Is there some setting I'm not seeing that's causing this copy to be generated?

My device is having other issues and this may be part of that?

Thanks

[jgoguen]

Shelves are entirely the work of the mainline driver. If you look at the list of options, anything after the debugging title is mine and anything else is the main driver.

For the book copy, do you have anything in the option "Copy generated KePub files to a directory"? If so, that, your problem. If not, please submit a bug report cause that's my problem

[jgoguen]

If anyone is interested in testing out some new code, I'm looking for some guinea pigs kind and friendly people(1) to try out the code on GitHub. There's a few things I'm interested in some intrepid testers taking a look at.

• The Windows Powershell script to create the plugin ZIP files is untested and needs to be verified by someone running Windows with Powershell 2 or later installed. Sorry(2) Windows XP users, I think that may exclude you, but you can manually create the plugin ZIP files or download the attached ZIP files.
• Some files were moved around to make it easier to separate this plugin from the conversion plugin and still share code. The KoboTouchExtended driver plugin should work exactly as before except for the missing options (next point).
• Some driver options were removed (see some previous-but-recent post). The last option now should be the option to copy a generated KePub file to some directory and it should be a text field.
  • The three database options at the end of the options list in the current version will be moved to the Kobo Utilities plugin, which is a much better home for them.
  • The option to delete files not in the manifest became more trouble than I felt it was worth when I switched to the standard calibre OEB container implementation
• In the formats list at the top of the KoboTouchExtended plugin preferences should include a 'kepub' entry (in alphabetical order, not selected by default). This only makes sense in the context of the new conversion output plugin, detailed later.

As you probably figured out by now, there's also a new conversion output format plugin to try out.
WARNING: THE CONVERSION OUTPUT FORMAT PLUGIN GENERATES FILES WHOSE INTERNAL STRUCTURE IS STILL UNDER DEVELOPMENT. THE KEPUB FILE STRUCTURE IS NOT YET FULLY UNDERSTOOD. DO NOT, UNDER ANY CIRCUMSTANCES, DELETE THE FILES FOR THE SOURCE FORMAT. ALWAYS HAVE A NON-KEPUB FORMAT FOR YOUR BOOKS.

• The 'create-plugin' scripts should create the ZIP files for both plugins.
• When you choose to convert a book, you can now choose 'KEPUB' from the output format list (top-right of the conversion window) and convert your books to Kobo's KePub format directly.
  • Please don't delete the non-KePub formats for any books you convert. Just trust me, humour me even, and please don't do it. No matter how many times someone tells you it'll be fine. Especially not if you find yourself or someone else saying "what's the worst that could possibly happen" or "nothing could possibly go wrong"
• If you convert a book to KePub and then send the book to the device, the upload process should be much faster as the driver skips the entire conversion pipeline for books in the 'kepub' format.
• I haven't tried converting from any source format other than ePub, but as this leverages calibre's conversion routines it should Just Work for any supported input format.
  • The list of input formats, in order of calibre's preference, is: LIT, MOBI, AZW, EPUB, AZW3, FB2, DOCX, HTML, PRC, ODT, RTF, PDB, TXT, PDF

Because calibre works with single extensions only, the files generated by the conversion output plugin have the extension 'kepub'. This shouldn't cause any problems when uploading to Kobo devices.

You'll notice that there's no conversion input format plugin. This is deliberate, and shouldn't cause any issues whatsoever because you're definitely going to heed my warning and always going to have a non-KePub format for your book. Right?

As part of this, the location of reference.kepub.epub has changed. Previously, it went in the KoboTouchExtended folder inside calibre's plugins folder, now it goes directly in calibre's plugins folder. Nothing uses the KoboTouchExtended folder anymore and it can be safely removed after updating to plugin version 2.1.1.

Any bugs can be reported in the usual way. The README file on GitHub (but not the copy in the first post) has been updated to be current for the new code, anything wrong or missing in the README should be considered a bug and reported accordingly.

(1) Not so kind and/or friendly people are also welcome. I'm an equal-opportunity tester seeker.
(2) Honestly, about 95% sorry that you're running Windows XP. You're a couple months away from running a 12 year old OS

[Leonatus]

Awsome indeed!
I'm using the hyphenation feature for a long time, and it's always becoming better. Congratulations!
But there are still some strange results, and therefore I would like to install the hyphenation dictionary again. And then I read the Usage Advice in the README, and here are my problems (kindly asked for forbearance, I'm 50+):

"Hyphenation
Both - (? I only know the Extended)

plugins includes the ability to add a CSS file to each book enabling KePub hyphenation. The standard hyphenation dictionaries provided on Kobo devices are somewhat deficient for some languages; fixing this (or adding your own dictionary) requires a little work and requires that you can create gzipped tarballs (.tgz files). - Do I need to create one? How do I do it?

Somewhere - (o. k. I guess I'll find a place)
on your computer, create the directory structure usr/local/Kobo/hyphenDicts - How do I create this? Do I need to create folders with those names?

Download the LibreOffice (or OpenOffice) hyphenation dictionary for your language
This will be a file with the 'oxt' extension. This is just a ZIP archive with a different name. - (o. k.)
Unzip the OXT file and look for the file named as hyph\_[language].dic. (o. k.)
Copy the hyphenation dictionary to the hyphenDicts folder without changing the name.
Add the usr folder to KoboRoot.tgz - (where do I find this? Indeed, on my KT there is no KoboRoot.tgz file - you may call me an idiot, but I didn't find)
UNIX users (Linux, Solaris, BSD, Mac) can, from the command line, run tar czf KoboRoot.tgz usr/ from wherever you put the usr directory.
Windows users, you're on your own. Contributions of reliable Windows directions are welcome! - (Hm, indeed, I'm Windows user. Help! Please!)
Once you have created KoboRoot.tgz copy it to the .kobo directory on your Kobo device, - (see above)
unmount/eject the drive, and unplug the device. You will see a screen indicating that the Kobo device is updating; it is not, but this method takes advantage of the Kobo update mechanism to load the necessary dictionary files. Make sure you keep your version of KoboRoot.tgz around, you will need to re-apply it after every Kobo software update! - (well, I hope I would come around with this)

Someone has an instruction for Dummies?

[BobC]

I have tried to add the Conversion output plugin, both by using the attached zip and by creating one from the Github files and in both cases I get an error :

calibre, version 0.9.42
ERROR: Unhandled exception: <b>ImportError</b>:cannot import name KEPubOutput

Traceback (most recent call last):
File "site-packages\calibre\gui2\preferences\plugins.py", line 310, in add_plugin
File "site-packages\calibre\customize\ui.py", line 361, in add_plugin
File "site-packages\calibre\customize\ui.py", line 53, in load_plugin
File "site-packages\calibre\customize\zipplugin.py", line 167, in load
File "site-packages\calibre\customize\zipplugin.py", line 150, in load_module
File "calibre_plugins.kobotouch_extended.__init__", line 7, in <module>
ImportError: cannot import name KEPubOutput

I notice that in your attachment the sql folder is included though it isn't called for in your "roll your own" recipe. I guess this is just an oversight in your build.

As I'm having connection issues to my Glo I don't want to update the device driver just yet which is why I wanted to try out the conversion one.

BobC
(Win XP user)

[jgoguen]

"Hyphenation
Both - (? I only know the Extended)[/QUOTE]
As per my last post, the current README is for the new code being tested. The first post has the README you'll want to read hidden behind a SPOILER tag.

Originally Posted by Leonatus

plugins includes the ability to add a CSS file to each book enabling KePub hyphenation. The standard hyphenation dictionaries provided on Kobo devices are somewhat deficient for some languages; fixing this (or adding your own dictionary) requires a little work and requires that you can create gzipped tarballs (.tgz files). - Do I need to create one? How do I do it?

If you want to update the hyphenation dictionary, then you must be able to create .tgz files. If you don't want to update the hyphenation dictionary, then it doesn't matter.

Originally Posted by Leonatus

Somewhere - (o. k. I guess I'll find a place)

Specifically where isn't important. You can do it on your Desktop, in your Downloads folder, under My Documents, in your temporary files directory, on a USB key... literally anywhere you want, just pick a spot that makes sense to you.

Originally Posted by Leonatus

on your computer, create the directory structure usr/local/Kobo/hyphenDicts - How do I create this? Do I need to create folders with those names?

Yes.

Originally Posted by Leonatus

Add the usr folder to KoboRoot.tgz - (where do I find this? Indeed, on my KT there is no KoboRoot.tgz file - you may call me an idiot, but I didn't find)

The 'usr' folder that you just created above, that gets added to a new file 'KoboRoot.tgz'. This is where you need to be able to properly create .tgz files. As I state in the README:

Originally Posted by Leonatus

UNIX users (Linux, Solaris, BSD, Mac) can, from the command line, run tar czf KoboRoot.tgz usr/ from wherever you put the usr directory.
Windows users, you're on your own. Contributions of reliable Windows directions are welcome!

So...

Originally Posted by Leonatus

- (Hm, indeed, I'm Windows user. Help! Please!)

You'll need to wait for someone else who's a Windows user to chime in, I'm afraid I'm not much help. Perhaps Google could help you, a quick search I did suggests that a program called 7Zip may be of use to you.

Originally Posted by Leonatus

Someone has an instruction for Dummies?

Not really, but anyone who can make the README easier to understand is welcome to contribute

Originally Posted by BobC

I have tried to add the Conversion output plugin, both by using the attached zip and by creating one from the Github files and in both cases I get an error :

calibre, version 0.9.42
ERROR: Unhandled exception: <b>ImportError</b>:cannot import name KEPubOutput

Traceback (most recent call last):
File "site-packages\calibre\gui2\preferences\plugins.py", line 310, in add_plugin
File "site-packages\calibre\customize\ui.py", line 361, in add_plugin
File "site-packages\calibre\customize\ui.py", line 53, in load_plugin
File "site-packages\calibre\customize\zipplugin.py", line 167, in load
File "site-packages\calibre\customize\zipplugin.py", line 150, in load_module
File "calibre_plugins.kobotouch_extended.__init__", line 7, in <module>
ImportError: cannot import name KEPubOutput

Interesting... I'll try removing the plugin and re-adding it. As it stands right now, it works fine for me. Which __init__.py did you add? Did you add the empty one that's there or did you rename conversion_init.py to __init__.py and then add a non-empty __init__.py?

Originally Posted by BobC

I notice that in your attachment the sql folder is included though it isn't called for in your "roll your own" recipe. I guess this is just an oversight in your build.

Actually the SQL folder should be gone. It shouldn't get added to any ZIP files, it shouldn't be in the attached ZIP files, and I should have deleted it from the Git repo.

[BobC]

[QUOTE=jgoguen;2589223]

Interesting... I'll try removing the plugin and re-adding it. As it stands right now, it works fine for me. Which __init__.py did you add? Did you add the empty one that's there or did you rename conversion_init.py to __init__.py and then add a non-empty __init__.py?[/QUOTE}

I renamed the conversion one (it took a few tries before I realised that there were double underscores on each side of "init") - the contents I now have in my zip file for installation is :

# vim:fileencoding=UTF-8:ts=4:sw=4:sta:et:sts=4:ai

__license__ = 'GPL v3'
__copyright__ = '2013, Joel Goguen <jgoguen@jgoguen.ca>'
__docformat__ = 'markdown en'

from calibre_plugins.kobotouch_extended.conversion.kepub_output import KEPubOutput

Both the one built from the Github zip and the attachment appear to be the same in this respect (and they both gave the same error).

I see there is a zero byte __init__.py in the conversion folder but perhaps that is normal.

Actually the SQL folder should be gone. It shouldn't get added to any ZIP files, it shouldn't be in the attached ZIP files, and I should have deleted it from the Git repo

.

A naive question from someone who doesn't really understand Git and similar but does the SQL stay in the repo in case you need to roll-back to a previous version ? When I built my copy I left the sql out as your recipe didn't call for it and AIUI it was used for manipulating tiles, not something needed in a conversion utility.

BobC

[jgoguen]

Originally Posted by BobC

I see there is a zero byte __init__.py in the conversion folder but perhaps that is normal.

That's normal, Python needs an __init__.py file in a folder to tell it to treat the folder as part of a package. Doesn't matter if the file is empty, there just needs to be a file with that name. The only special __init__.py is the one at the top level of the plugin, that tells calibre what class to load as the plugin class.

Originally Posted by BobC

A naive question from someone who doesn't really understand Git and similar but does the SQL stay in the repo in case you need to roll-back to a previous version ?

It stays in the previous versions. If you go to the repository (https://github.com/jgoguen/calibre-kobo-driver/) you'll see above the list of files a drop-down that says "branch: master". Click that and choose "tags" and you'll see the list of tags, which represent a snapshot of the point in time that I released the tagged version. The sql folder won't be in the 'master' branch, but if it's needed for some reason you can always check out (or download) any tag to get each file as it was at that point in time.

Originally Posted by BobC

When I built my copy I left the sql out as your recipe didn't call for it and AIUI it was used for manipulating tiles, not something needed in a conversion utility.

That's pretty much it, the SQL files were used to create the database triggers for hiding various tiles. Those are moved into the Kobo Utilities plugin now.

[Leonatus]

@jgoguen: Thank you Joel! Perhaps I deem things more complicated than they are, because of not wholy capturing the sense of some English words. Besides- my technical abilities are rather constrained.

[JQW]

I notice that books whose chapters stem two HTML files within the .epub result in the chapters being split into two when rendered as a .kepub.

Is there any fix to this other than manually editing the book within something like Sigil?

[jgoguen]

I don't believe so. It appears to me (having no idea how the rendering engine actually processes books) that there's an assumption that each file contains precisely one chapter. Granted, with the complete lack of any standard structure to an eBook "chapter" I don't actually see a better way to handle it, and it's a reasonable enough assumption...

[Leonatus]

Quote: You will see a screen indicating that the Kobo device is updating; it is not, but this method takes advantage of the Kobo update mechanism to load the necessary dictionary files. /Quote.

Well, I succeeded with all that you recommended. the only thing is, that, after unplugging the device, there is no indication of updating. the KoboRoot.tar file but is still in the .Kobo folder. Does the lack of the indication matter? - I should mention that on the KT runs fw. 2.5.2.

[JQW]

Originally Posted by jgoguen

I don't believe so. It appears to me (having no idea how the rendering engine actually processes books) that there's an assumption that each file contains precisely one chapter. Granted, with the complete lack of any standard structure to an eBook "chapter" I don't actually see a better way to handle it, and it's a reasonable enough assumption...

Thanks. I'll just have to process everything manually, then. Unfortunately the books that I'm most bothered about are both of multi-volume history works with many chapters.

[BobC]

@jgoguen Regarding the problem installing the KePub Conversion plug-in. I've now worked out what was wrong.

The problem seems to be that it needs at least Ver 2.1.0 of the extended driver to be installed to work. I was running with an earlier version. I'd made the assumption that the two were stand-alone and not interdependent.

And thanks for the explanation about Github and how to get at previous versions.

BobC

[jgoguen]

Originally Posted by Leonatus

the KoboRoot.tar file

It has to be KoboRoot.tgz. You can't just change the extension, you actually have to properly create it as a gzipped TAR archive. If the file you have is a proper TAR archive, you can just GZip it and name the resulting file KoboRoot.tgz. If you created a gzipped TAR archive and just named it wrong, then you should be able to change the extension.

To be absolutely certain though, I would create the KoboRoot.tgz file from scratch to make sure it's right. I don't know how the device will handle it if you give it a file in the wrong format. I know my Glo needed a factory reset when I gave it a file in the wrong format

Originally Posted by BobC

The problem seems to be that it needs at least Ver 2.1.0 of the extended driver to be installed to work.

That's bad, they should be completely independent. Each plugin file has all the files it needs to be fully independent. I'm pretty sure I know what it is though, I'll verify and post a new set of plugin files tonight sometime (I'm UTC-3).