Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Readers > Amazon Kindle > Kindle Developer's Corner

Notices

Reply
 
Thread Tools Search this Thread
Old 01-26-2011, 06:39 AM   #1
Acorn
Connoisseur
Acorn is on a distinguished road
 
Posts: 56
Karma: 74
Join Date: Oct 2010
Device: Kindle 3
Question Voice Control for Accessibility (simulating key presses)

I'm hoping to try and get some kind of a working form of voice control going for the Kindle.

This would be a great help to anyone who doesn't have the use of their hands and can't control their Kindle by pressing buttons.

I specifically wanted to try and get this functionality working for a friend of mine who has Multiple Sclerosis. This would allow her to read on her own, which she hasn't been able to do for a long time.

The practical stuff:

I'm not going to attempt to use the built in microphone, or anything like that. I'm just going to try and get remote execution of key input so that the voice recognition stuff can happen on a computer.

So the main thing I need to work out is, how do you simulate key presses via the command line?

I have a feeling that h1uke's Launchpad hack does exactly this, but I'm having trouble working out exactly what part of his code is responsible for sending the actual command, and how it is doing this.

If anyone could explain to me how one simulates a key press, then I could have a script on the computer trigger those keypresses over SSH when it receives the correct voice commands.

Thanks!

UPDATE:

I've got working remote execution of Kindle key commands that can be interacted with using voice recognition software. Still a work in progress, but give it a go and let me know what you think!

Just remember to put the right settings in config.ini, and make sure you have the usbnetwork hack installed with ssh over wifi.
Attached Files
File Type: zip KindleControl-0.1.zip (5.88 MB, 919 views)

Last edited by Acorn; 01-31-2011 at 08:15 PM.
Acorn is offline   Reply With Quote
Old 01-26-2011, 08:40 AM   #2
lrizzo
Member
lrizzo can tie a knot in a cherry stem with his or her tonguelrizzo can tie a knot in a cherry stem with his or her tonguelrizzo can tie a knot in a cherry stem with his or her tonguelrizzo can tie a knot in a cherry stem with his or her tonguelrizzo can tie a knot in a cherry stem with his or her tonguelrizzo can tie a knot in a cherry stem with his or her tonguelrizzo can tie a knot in a cherry stem with his or her tonguelrizzo can tie a knot in a cherry stem with his or her tonguelrizzo can tie a knot in a cherry stem with his or her tonguelrizzo can tie a knot in a cherry stem with his or her tonguelrizzo can tie a knot in a cherry stem with his or her tongue
 
Posts: 23
Karma: 22222
Join Date: Oct 2010
Device: kindle3
Quote:
Originally Posted by Acorn View Post
I'm hoping to try and get some kind of a working form of voice control going for the Kindle.

This would be a great help to anyone who doesn't have the use of their hands and can't control their Kindle by pressing buttons.

I specifically wanted to try and get this functionality working for a friend of mine who has Multiple Sclerosis. This would allow her to read on her own, which she hasn't been able to do for a long time.

The practical stuff:

I'm not going to attempt to use the built in microphone, or anything like that. I'm just going to try and get remote execution of key input so that the voice recognition stuff can happen on a computer.

So the main thing I need to work out is, how do you simulate key presses via the command line?

I have a feeling that h1uke's Launchpad hack does exactly this, but I'm having trouble working out exactly what part of his code is responsible for sending the actual command, and how it is doing this.

If anyone could explain to me how one simulates a key press, then I could have a script on the computer trigger those keypresses over SSH when it receives the correct voice commands.

Thanks!
the way to simulate keypresses is to write the string "send NNN" to /proc/keypad or one of the other devices, where NNN is the appropriate keycode. For instance,

echo "send 104" > /proc/keypad

on a Kindle3 is the same as moving to next page.

You can find the device names and keycodes in keydefs.ini and launchpad.ini from my "Kindle Terminal" at http://info.iet.unipi.it/~luigi/kindle/

Finally, the microphone is available e.g. arecord test.wav records a file (you can stop with ctrl-c) that you can later play with aplay test.wav (and in C you can access the device using the ALSA API)

cheers
luigi
lrizzo is offline   Reply With Quote
Advert
Old 01-26-2011, 08:58 AM   #3
Acorn
Connoisseur
Acorn is on a distinguished road
 
Posts: 56
Karma: 74
Join Date: Oct 2010
Device: Kindle 3
Wow, thanks for the help!

If I'm going to start by just having the voice recognition side of things on a computer, I assume the easiest way would be to just send the commands over SSH?

The idea of getting simple speech recognition working with the built in microphone is really exciting, but it's way over my head. I have no knowledge of C

If it was possible to get Python working on the Kindle, I might be able to give it more of a try, but that's probably a crazy way to go about it.

Thanks again!

Edit:

Is there a simpler way to send commands to the Kindle than SSH? It seems a little bit overkill. It also has the downside that you need to fiddle around with drivers to get SSH over USB working.

Telnet?

Last edited by Acorn; 01-26-2011 at 10:19 AM.
Acorn is offline   Reply With Quote
Old 01-26-2011, 11:23 AM   #4
h1uke
Zealot
h1uke can do the Funky Gibbon.h1uke can do the Funky Gibbon.h1uke can do the Funky Gibbon.h1uke can do the Funky Gibbon.h1uke can do the Funky Gibbon.h1uke can do the Funky Gibbon.h1uke can do the Funky Gibbon.h1uke can do the Funky Gibbon.h1uke can do the Funky Gibbon.h1uke can do the Funky Gibbon.h1uke can do the Funky Gibbon.
 
Posts: 121
Karma: 82565
Join Date: Aug 2010
Location: Maryland, USA
Device: dxg, k3w,k4nt,kpw
Quote:
Originally Posted by Acorn View Post
Is there a simpler way to send commands to the Kindle than SSH? It seems a little bit overkill. It also has the downside that you need to fiddle around with drivers to get SSH over USB working.
Telnet?
provided the usbNetwork hack installed, I can easily add a "network" part to the existing
launchpad daemon. This addition will be listening to incoming UDP multicast packets
sent by your desktop application and translate them directly to whatever Kindle shell
commands you want.
You can also use Kindle serial console, this solution will not require any hacks at all.
Some mechanical/soldering work will be required in this case. But this approach can noticeably
simplify the whole thing.
h1uke is offline   Reply With Quote
Old 01-26-2011, 04:36 PM   #5
kindle3zeng
Enthusiast
kindle3zeng once ate a cherry pie in a record 7 seconds.kindle3zeng once ate a cherry pie in a record 7 seconds.kindle3zeng once ate a cherry pie in a record 7 seconds.kindle3zeng once ate a cherry pie in a record 7 seconds.kindle3zeng once ate a cherry pie in a record 7 seconds.kindle3zeng once ate a cherry pie in a record 7 seconds.kindle3zeng once ate a cherry pie in a record 7 seconds.kindle3zeng once ate a cherry pie in a record 7 seconds.kindle3zeng once ate a cherry pie in a record 7 seconds.kindle3zeng once ate a cherry pie in a record 7 seconds.kindle3zeng once ate a cherry pie in a record 7 seconds.
 
Posts: 35
Karma: 1566
Join Date: Nov 2010
Device: kindle 3wifi
Quote:
Originally Posted by Acorn View Post
Wow, thanks for the help!

If I'm going to start by just having the voice recognition side of things on a computer, I assume the easiest way would be to just send the commands over SSH?

The idea of getting simple speech recognition working with the built in microphone is really exciting, but it's way over my head. I have no knowledge of C

If it was possible to get Python working on the Kindle, I might be able to give it more of a try, but that's probably a crazy way to go about it.

Thanks again!

Edit:

Is there a simpler way to send commands to the Kindle than SSH? It seems a little bit overkill. It also has the downside that you need to fiddle around with drivers to get SSH over USB working.

Telnet?
Yes, it's possible to run python on kindle, I copied python from ubuntu arm port, and from my limited tests, it worked fine.
kindle3zeng is offline   Reply With Quote
Advert
Old 01-26-2011, 04:52 PM   #6
Acorn
Connoisseur
Acorn is on a distinguished road
 
Posts: 56
Karma: 74
Join Date: Oct 2010
Device: Kindle 3
Quote:
Originally Posted by h1uke View Post
provided the usbNetwork hack installed, I can easily add a "network" part to the existing
launchpad daemon. This addition will be listening to incoming UDP multicast packets
sent by your desktop application and translate them directly to whatever Kindle shell
commands you want.
You can also use Kindle serial console, this solution will not require any hacks at all.
Some mechanical/soldering work will be required in this case. But this approach can noticeably
simplify the whole thing.
That's an awesome offer h1uke! Do you think it would be fine to not have any authentication, or would there need to be some kind of signing of the commands sent to the Kindle? Do you think it would make sense to have it as part of launchpad? Or as a separate packaged mod (not sure how much effort that takes)? Even just a script that gives this functionality would be really cool.
Acorn is offline   Reply With Quote
Old 01-26-2011, 04:53 PM   #7
Acorn
Connoisseur
Acorn is on a distinguished road
 
Posts: 56
Karma: 74
Join Date: Oct 2010
Device: Kindle 3
Quote:
Originally Posted by kindle3zeng View Post
Yes, it's possible to run python on kindle, I copied python from ubuntu arm port, and from my limited tests, it worked fine.
That's interesting to know. I'll look into if I ever want to do something more serious on the Kindle itself.
Acorn is offline   Reply With Quote
Old 01-26-2011, 05:06 PM   #8
lrizzo
Member
lrizzo can tie a knot in a cherry stem with his or her tonguelrizzo can tie a knot in a cherry stem with his or her tonguelrizzo can tie a knot in a cherry stem with his or her tonguelrizzo can tie a knot in a cherry stem with his or her tonguelrizzo can tie a knot in a cherry stem with his or her tonguelrizzo can tie a knot in a cherry stem with his or her tonguelrizzo can tie a knot in a cherry stem with his or her tonguelrizzo can tie a knot in a cherry stem with his or her tonguelrizzo can tie a knot in a cherry stem with his or her tonguelrizzo can tie a knot in a cherry stem with his or her tonguelrizzo can tie a knot in a cherry stem with his or her tongue
 
Posts: 23
Karma: 22222
Join Date: Oct 2010
Device: kindle3
Quote:
Originally Posted by Acorn View Post
That's an awesome offer h1uke! Do you think it would be fine to not have any authentication, or would there need to be some kind of signing of the commands sent to the Kindle? Do you think it would make sense to have it as part of launchpad? Or as a separate packaged mod (not sure how much effort that takes)? Even just a script that gives this functionality would be really cool.
I am missing the point for doing this. ssh (over wifi) is already nicely packaged as part of the usbnetwork package, and you can configure it to have ssh always active. This gives you authentication and reliability without the need of reinventing everything from scratch.
lrizzo is offline   Reply With Quote
Old 01-26-2011, 05:24 PM   #9
yifanlu
Kindle Dissector
yifanlu ought to be getting tired of karma fortunes by now.yifanlu ought to be getting tired of karma fortunes by now.yifanlu ought to be getting tired of karma fortunes by now.yifanlu ought to be getting tired of karma fortunes by now.yifanlu ought to be getting tired of karma fortunes by now.yifanlu ought to be getting tired of karma fortunes by now.yifanlu ought to be getting tired of karma fortunes by now.yifanlu ought to be getting tired of karma fortunes by now.yifanlu ought to be getting tired of karma fortunes by now.yifanlu ought to be getting tired of karma fortunes by now.yifanlu ought to be getting tired of karma fortunes by now.
 
Posts: 662
Karma: 475607
Join Date: Jul 2010
Device: Amazon Kindle 3
Doesn't the kindle 3 have a microphone? Anyone know how to access it?
yifanlu is offline   Reply With Quote
Old 01-26-2011, 05:29 PM   #10
Acorn
Connoisseur
Acorn is on a distinguished road
 
Posts: 56
Karma: 74
Join Date: Oct 2010
Device: Kindle 3
Quote:
Originally Posted by lrizzo View Post
I am missing the point for doing this. ssh (over wifi) is already nicely packaged as part of the usbnetwork package, and you can configure it to have ssh always active. This gives you authentication and reliability without the need of reinventing everything from scratch.
I think UDP would make things a lot simpler. Authentication is probably completely unnecessary. All you need is a way to send simple commands to the Kindle. The fact that Multicast UDP means you don't even need to know the ip address of the kindle is even better. Quite nice to be able to have SSH set-up separately as well so you don't have to have SSH running all the time if you don't need/want it.

I don't know much about networking/socket programming/ssh, so these thoughts are just based on my limited knowledge and what I've read.
Acorn is offline   Reply With Quote
Old 01-26-2011, 05:31 PM   #11
Acorn
Connoisseur
Acorn is on a distinguished road
 
Posts: 56
Karma: 74
Join Date: Oct 2010
Device: Kindle 3
Quote:
Originally Posted by yifanlu View Post
Doesn't the kindle 3 have a microphone? Anyone know how to access it?
Irizzo explained this further up:
Quote:
Originally Posted by lrizzo View Post
Finally, the microphone is available e.g. arecord test.wav records a file (you can stop with ctrl-c) that you can later play with aplay test.wav (and in C you can access the device using the ALSA API)
Acorn is offline   Reply With Quote
Old 01-27-2011, 03:51 AM   #12
vdp
Enthusiast
vdp is clearly one to watchvdp is clearly one to watchvdp is clearly one to watchvdp is clearly one to watchvdp is clearly one to watchvdp is clearly one to watchvdp is clearly one to watchvdp is clearly one to watchvdp is clearly one to watchvdp is clearly one to watchvdp is clearly one to watch
 
Posts: 45
Karma: 10842
Join Date: Aug 2010
Device: Kindle DXG
Quote:
Originally Posted by Acorn View Post
That's interesting to know. I'll look into if I ever want to do something more serious on the Kindle itself.
Why do you think you need to have python in order to do SR on the device?
I think the "command and control" type of application you describe is perfectly doable using CMU's PocketSphinx with a finite state grammar. As far as I know there are already versions for IPhone and Android that can be used as a starting point.

Last edited by vdp; 01-27-2011 at 04:02 AM.
vdp is offline   Reply With Quote
Old 01-27-2011, 11:50 AM   #13
Acorn
Connoisseur
Acorn is on a distinguished road
 
Posts: 56
Karma: 74
Join Date: Oct 2010
Device: Kindle 3
Quote:
Originally Posted by vdp View Post
Why do you think you need to have python in order to do SR on the device?
I think the "command and control" type of application you describe is perfectly doable using CMU's PocketSphinx with a finite state grammar. As far as I know there are already versions for IPhone and Android that can be used as a starting point.
I was just saying that Python was the only language I have knowledge of. PocketSphinx looks interesting, I'll definitely look into it! Thanks for the tip.
Acorn is offline   Reply With Quote
Old 01-29-2011, 09:35 PM   #14
Acorn
Connoisseur
Acorn is on a distinguished road
 
Posts: 56
Karma: 74
Join Date: Oct 2010
Device: Kindle 3
It turns out that having to send the commands over SSH is making things a huge headache. I'm wishing I could just send a packet
Acorn is offline   Reply With Quote
Old 01-31-2011, 01:06 PM   #15
Acorn
Connoisseur
Acorn is on a distinguished road
 
Posts: 56
Karma: 74
Join Date: Oct 2010
Device: Kindle 3
Okay, I have a few more questions.

1. What is the command to make the Kindle wake up from sleep? Is it safe to use if the kindle is already awake?

2. How does the Kindle get assigned its IP address? My script needs to know the IP address of the Kindle, so it will need to be static. How can I achieve this?
Acorn is offline   Reply With Quote
Reply

Tags
accessibility, key-press, simulate, voice, voice-control

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Accessories DIY Kindle Cover djulian Amazon Kindle 11 02-26-2011 08:12 PM
iPad Enable voice control on iPad (jailbreak) scottjl Apple Devices 0 05-03-2010 06:17 PM
The Clerkenwell Presses ShortNCuddlyAm Writers' Corner 16 01-30-2010 06:21 PM
Free on Kindle - The Voice koland Deals and Resources (No Self-Promotion or Affiliate Links) 0 12-09-2009 02:06 PM
DIY Kindle Case EnginerdLisa Amazon Kindle 3 09-27-2008 10:56 PM


All times are GMT -4. The time now is 06:39 AM.


MobileRead.com is a privately owned, operated and funded community.