Aura H2O Turn pages by voice

[Ecallan]

I am new to mobile reads. My principal reason for joining is to seek help. I am paralysed from my neck down and currently use my Kobo e-reader using a mouth stick. However I'd really like to be able to use my e-reader by voice, ie to be able to turn pages by saying "next page" or "previous page" to go back. I did see one possibility with a Kindle using HID commands in conjunction with a Raspberry pi. However I would prefer to stay with Kobo. So dear readers, any bright ideas out there?

In case anybody is thinking of a tablet, such as an iPad, while I know a software solution to this probably exists, it doesn't have the same readability as an E Ink reader, particularly outside so I'm discounting the tablet route. I am also not particularly interested in text to speech options such as audible. I want to read the written word.

Obviously, my E reader came as a transformative development in my ease of reading. For many years I struggled with great difficulty to turn pages using my mouthstick on regular books. While in hospital originally, I read Kane and Abel, The Day of the Jackal and many other books by asking somebody passing my bed to turn the page. As you can imagine, this required the book to be pretty good to persevere! Thankfully these days, my e-reader gives me the scope to read that I had longed for.

I am based in Dublin, Ireland. Many thanks in advance for at least reading my call for help.

Eugene

[tshering]

Hi Ecallan,

Welcome to the forum!

Would it be ok if your Kobo device had to be connected to the usb port of a pc/laptop?

The basic idea is to use a speech recognition program and telnet to send commands to the Kobo device.
Since I am a windows user, I started experimenting a little with visual basic script and the windows inbuild speech recognition program sapisvr.exe.
Page turning per telnet does work, so I did not include this part in my current experiment.

What I have now is a visual basic script that

• starts a telnet session and connects to the Kobo device
• waits for speech input and sends (depending on the speech input) commands via telnet to the Kobo device.

Currently, my main problem is that the speech recognition is completely unreliable. If you are a windows user and have already some acquaintance with sapisvr.exe you might possibly get better results than me.

Now my question. Do you think this is something worth continuing?

[tshering]

Just want to report that I can now page forward and backward by voice commands. I will give the details when I have got a chance.

[Ecallan]

Hi Tshering

Thank you very much for investigating the telnet approach via Windows and the program sapisvr.exe. It is a very novel approach and certainly demonstrates that page turning commands can be sent via the usb port which is a important step forward.

Using a windows pc not ideal although perhaps using a usb stick computer such as the Intel Compute Stick might be ideal if combined with external battery sounds like a definite possibility. I'd be grateful if you would give me the code tto test it out on my current PC please.

Thanks again and I really do appreciate your work on this.

Eugene

[Ecallan]

......

[tshering]

Originally Posted by Ecallan

I'd be grateful if you would give me the code tto test it out on my current PC please.

Sure. I am currently in my office. I will post the code this night when I am at home (after cleaning the code a little bit, and preparing some explanations) or maybe tomorrow. But do not expect to much. It is working, but it is far from perfect. And there is still some preparatory work to do. I give you the details:

One can (or has to) "record" the necessary touch events, so that one can later "play" them. I did this for the Kobo Touch, but cannot do it for the H2O, since I do not have access to one. When I post the code here, I will also ask whether somebody can record the touch events. I hope some people might be interested in this too. This recorded touch events will still be useful even if we decide to change other parts of our approach.

On the pc

• you will have to enable the windows speech recognition, if you did not do this already (Since then I learned that you are using Nuance NaturallySpeaking. I have still to google for it. I just read "the program transcribes the words into the active window at the location of the cursor." Can it also close the input box by confirming [saying "OK"] or cancelling [saying "Cancel"]? Or something like that? If yes you could just use this instead of the Windows inbuilt speech recognition.)
• you will also have to enable telnet
• and you will also have to prepare your pc for usbnet (If you wish we could avoid this for the moment, and do our first tests with wife instead of telnet. But this has some disadvantages.)
• I think all windows pcs can run visual basic scripts out of the box. Right?

On the Kobo side, you would have to

• enable telnet (You can simply copy the KoboRoot.tgz that you can find here to the directory .kobo of you device and restart the device)
• when you decide for using usbnet, I recommend installing the Kobo start menu (KSM) for easily enabling/disabling usbnet.
• copy the files with the recorded touch events to the kobo (once we have them)

A short description of the script:

• It reads a text file that associates certain commands like "next page", "previous page" with the names of files that contain the appropriate recorded touch events.
• It waits for your input (by speaking or typing [so that people can also use it without having to enable speech recognition, or the English language package]).
• • In test mode, if the scripts can identify the input phrase with a command, it confirms this, or it tells you that it does not recognize the command (both acoustically with the Windows inbuilt text to speech).
  • In normal mode, the script sends a command with telnet to the Kobo device in order to trigger the touch event, and gives acoustic information (you can disable this).
• You terminate the script by saying/typing "exit."

Things that do not work so well:

• 1) Once you start the script (in normal mode) you should not change to another window or application. If you do, the script looses its connection with the telnet window.
• 2) After terminating the script, you have to close the telnet session, exit telnet and close the cmd window yourself.
• 3) One has to get used to a certain way of speaking to make the speech recognition recognize what one is saying.

So maybe as a first step you could just try the script in test mode to get a feeling of how it works, and see whether you think it is usable at all. To do this no changes are necessary on the Kobo device, it is not involved at all.

At a later stage, I might try to replace the use of the vb script interpreter and the windows inbuilt telnet program by a stand alone application. That should make some things easier (points 1 and 2 from above) and be easier portable to other platforms. I am however not sure about the integration of the speech recognition.

What is currently working:

• page forwards and backwards in a book
• open the main menu
• go into the library or the collections, page through them and open a book
• close a book
• open a book from "home"

This is done with the more recent firmwares that have the "new home." Having the menu easily available makes things much easier.

[tshering]

I would like to ask whether some people could help with this project and create data files of touch events for different devices, especially the H2O.

You would need telnet access to your device.

This is how it works:

Spoiler:

make (from your telnet window) a directory for the files that you will create:

mkdir -p /mnt/onboard/.adds/pressdata

change to the newly created directory

cd /mnt/onboard/.adds/pressdata

initiate the record of a touch event

cat /dev/input/event1 > testtouch.dat

now (on the reader with nickel running) touch the screen (for instance somewhere in the middle), and watch whether the reader responds as expected
again from the telnet window: type Ctrl-C to get the telnet prompt again

To test whether the created file works:

cat testtouch.dat > /dev/input/event1

See whether the reader responses as expected.

We would need the following simple touch events (I give the list of the filenames. Together with the attached images you will certainly figure out what is meant:

Spoiler:

home_open_book1.dat
home_open_book2.dat
home_open_book3.dat
home_open_book4.dat
home_open_group1.dat
home_open_group2.dat
library_open_book1.dat
library_open_book2.dat
library_open_book3.dat
library_open_book4.dat
library_open_book5.dat
open_main_menu.dat
open_main_menu_home.dat
open_main_menu_mybooks.dat
press_middle.dat
swipe_bottom_to_left.dat
swipe_bottom_to_right.dat

Note that all but the last two are simple touch events, the last two are swipes.

I attach the files for the Kobo Touch.

[tshering]

Here is the script.

For testing on the pc only:
Double click on "no_telnet.bat" to run the script in test mode.
Double click on "no_telnet_dont_speak.bat" to run the script in test mode, and without acustic responses.

For real usage (needs pc and Kobo device beeing correctly prepared)
Double click on "kobo_voice_control.vbs" to run the script in normal mode.
Double click on "dont_speak.bat" to run the script in normal mode, without acoustic information.

Be aware that "speaking" (text to voice) slows the script down considerably.

The text file vc_commands.txt contains among other things the currently available commands. Each line contains columns separated by tabs. The third column contains the commands and their variations (the are between "|").

Read also post #6.

[frostschutz]

Aaah, the full blown solution in the making

If you want it to cover everything and be resilient against firmware upgrades that move stuff around, you'd have to let the user define arbitrary events and do their own touch recordings for them. But that's probably overthinking it.

[tshering]

Originally Posted by frostschutz

Aaah, the full blown solution in the making

If you want it to cover everything and be resilient against firmware upgrades that move stuff around, you'd have to let the user define arbitrary events and do their own touch recordings for them. But that's probably overthinking it.

Thank you for the recordings. Corncerning defining arbitrary commands, with the current approach there is a text file that associates commands with file names. It should not prove to be too difficult for a user edit this file. But, I think I do not like the vbscript approach anyway.

[tshering]

@Ecallan


Maybe we should think about other possibilities to trigger the touch events. I guess you browse the web somehow with speech commands. If you do, can you directly address buttons by speaking out their caption? For instance, if there is a button with the caption "page next" on a web page, can you press it by saying "next page?"

The idea is to run a web server on the Kobo that delivers a page with buttons. "Pressing" a button on the page (in your web browser on the pc) will trigger a touch event.

You can see a similar project here. The point there was to make a mouse click in a web browser and by this trigger a touch at the corresponding position of the Kobo reader. For our current project, we would limit the possibilities to a number of pre-recorded events to make things easier, because we want to give our commands without having to look at the screen of the browser and without moving around the cursor. Before I try to find out about voice controlling a web page by myself, I wait for your answer, since you will already have the necessary informations.

[Ecallan]

Thank you for this fantastic amount of work. I don't really have anybody to assist me with this project at the moment, but I will follow directions step-by-step and get there in the end!

A question on installing the kobo start menu (KSM) when I followed the link in your post, there was a line in red at the top saying that the ksm 08 doesn't work on the kobo aura H2O. Could you provide a link directly to the file that I need to download and copy to the .kobo folder please?

Eugene

Originally Posted by tshering

Sure. I am currently in my office. I will post the code this night when I am at home (after cleaning the code a little bit, and preparing some explanations) or maybe tomorrow. But do not expect to much. It is working, but it is far from perfect. And there is still some preparatory work to do. I give you the details:

One can (or has to) "record" the necessary touch events, so that one can later "play" them. I did this for the Kobo Touch, but cannot do it for the H2O, since I do not have access to one. When I post the code here, I will also ask whether somebody can record the touch events. I hope some people might be interested in this too. This recorded touch events will still be useful even if we decide to change other parts of our approach.

On the pc

• you will have to enable the windows speech recognition, if you did not do this already (Since then I learned that you are using Nuance NaturallySpeaking. I have still to google for it. I just read "the program transcribes the words into the active window at the location of the cursor." Can it also close the input box by confirming [saying "OK"] or cancelling [saying "Cancel"]? Or something like that? If yes you could just use this instead of the Windows inbuilt speech recognition.)
• you will also have to enable telnet
• and you will also have to prepare your pc for usbnet (If you wish we could avoid this for the moment, and do our first tests with wife instead of telnet. But this has some disadvantages.)
• I think all windows pcs can run visual basic scripts out of the box. Right?

On the Kobo side, you would have to

• enable telnet (You can simply copy the KoboRoot.tgz that you can find here to the directory .kobo of you device and restart the device)
• when you decide for using usbnet, I recommend installing the Kobo start menu (KSM) for easily enabling/disabling usbnet.
• copy the files with the recorded touch events to the kobo (once we have them)

A short description of the script:

• It reads a text file that associates certain commands like "next page", "previous page" with the names of files that contain the appropriate recorded touch events.
• It waits for your input (by speaking or typing [so that people can also use it without having to enable speech recognition, or the English language package]).
• • In test mode, if the scripts can identify the input phrase with a command, it confirms this, or it tells you that it does not recognize the command (both acoustically with the Windows inbuilt text to speech).
  • In normal mode, the script sends a command with telnet to the Kobo device in order to trigger the touch event, and gives acoustic information (you can disable this).
• You terminate the script by saying/typing "exit."

Things that do not work so well:

• 1) Once you start the script (in normal mode) you should not change to another window or application. If you do, the script looses its connection with the telnet window.
• 2) After terminating the script, you have to close the telnet session, exit telnet and close the cmd window yourself.
• 3) One has to get used to a certain way of speaking to make the speech recognition recognize what one is saying.

So maybe as a first step you could just try the script in test mode to get a feeling of how it works, and see whether you think it is usable at all. To do this no changes are necessary on the Kobo device, it is not involved at all.

At a later stage, I might try to replace the use of the vb script interpreter and the windows inbuilt telnet program by a stand alone application. That should make some things easier (points 1 and 2 from above) and be easier portable to other platforms. I am however not sure about the integration of the speech recognition.

What is currently working:

• page forwards and backwards in a book
• open the main menu
• go into the library or the collections, page through them and open a book
• close a book
• open a book from "home"

This is done with the more recent firmwares that have the "new home." Having the menu easily available makes things much easier.

[tshering]

Originally Posted by Ecallan

Thank you for this fantastic amount of work. I don't really have anybody to assist me with this project at the moment, but I will follow directions step-by-step and get there in the end!

A question on installing the kobo start menu (KSM) when I followed the link in your post, there was a line in red at the top saying that the ksm 08 doesn't work on the kobo aura H2O. Could you provide a link directly to the file that I need to download and copy to the .kobo folder please?

Eugene

I am not sure whether you are aware that there are two models with "H2O" in the name. One is the Kobo Aura H2O, and the other is the Kobo Aura H20 Edition 2 (also called New Kobo Aura H20). KSM does not work on the new one, the Kobo Aura H20 Edition 2. If you do not know which H2O you have, the most evident difference might be that the "old" H2O has a slot for an external SD card (covered by a plug), whereas the new, H2O Edition 2 does not have a slot for an external SD; and the blue power button on the back side of the H2O Edition 2.

Concerning installing KSM 8 (supposing you have the "old" H2O), I am not sure whether it is necessary or a good idea to do this now, especially since you don't have anybody to assist you at the moment with this project. I realize only now, that if we can control the reader by key input or voice from a pc (or something similar), we do not need KSM at all. If you still want to install KSM, tell me, and I will prepare a new package to make the installation easier. (But frankly, I would not recommend doing this at the moment.)

Did you see my post #11 of this thread? Can you tell me your thought on this? Especially on interacting with a web page by voice commands?

[Ecallan]

Sorry to be only getting back to you now. I went to town today to go to a new exhibition that is on in our national Gallery!

Re your post #11,

Yes I can say "click name" or if there is a button with a caption eg "Next Page" I can simply say "Next Page" to initiate that action.

My Nuance NaturallySpeaking needs significant resources so may not be the best option to use on my planned Intel Compute usb stick. It might be better for me to use Microsoft Windows 10 inbuilt voice recognition. As far as I can tell, it will also action button with a caption under it.

My Kobo model is the original Aura H2O not the model 2.

I've successfully installed KoboRoot.tgz before I actually read your post saying that it isn't necessary. It is providing the menu with a range of options.

I'll go ahead and installed the other files that you've created.

Just another thought that might be useful. I have Teamviewer on my PC which would enable you to logon directly to my PC and through it link directly to my kobo reader if that would be useful.

Eugene

Originally Posted by tshering

@Ecallan


Maybe we should think about other possibilities to trigger the touch events. I guess you browse the web somehow with speech commands. If you do, can you directly address buttons by speaking their caption? For instance, if there is a button with the caption "page next" on a web page, can you press it by saying "next page?"

The idea is to run a web server on the Kobo that delivers a page with buttons. "Pressing" a button on the page (in your web browser) will trigger a touch event.

You can see a similar project here. The point there was to make a mouse click in a webbrowser and by this trigger a touch at the corresonding position of the Kobo reader. For our current project, we would limit the possibilities to a number of pre-recorded events to make things easier, because we want to give our commands without having to look at the screen of the browser and without moving around the coursor. Before I try to find out about voice controlling a web page by myself, I wait for your answer, since you will already have the necessary informations.

[tshering]

Originally Posted by Ecallan

Sorry to be only getting back to you now. I went to town today to go to a new exhibition that is on in our national Gallery!

It seems you had a great day!

Originally Posted by Ecallan

My Nuance NaturallySpeaking needs significant resources so may not be the best option to use on my planned Intel Compute usb stick. It might be better for me to use Microsoft Windows 10 inbuilt voice recognition. As far as I can tell, it will also action button with a caption under it.

I thought Nuance NaturallySpeaking would be preferable, since you are already used to it. I did however not think about the resources it needs. I will play around a little more with the Microsoft Windows 10 inbuilt voice recognition.

Originally Posted by Ecallan

I've successfully installed KoboRoot.tgz before I actually read your post saying that it isn't necessary.

Since you installed it already, it might make things easier on the way.

Originally Posted by Ecallan

I'll go ahead and installed the other files that you've created.

Which files? Do you mean the files that contain the recorded touch events? You need those created by Frostschutz for the H20. Wait a moment. I will pack them into a KoboRoot.tgz so that you can install them.
In the meantime you could install the telnet packaged I linked to above (here is the link again.). Is telnet enabled on your PC? One has to go into the settings and enable it (it is disabled by default).