pocketsphinx Audio Recognition and Programming API

twobob · 03-23-2013, 10:18 PM

It was supposed to be my 4500'th post... but close enough

Pocketsphinx: Audio Recognition and Programming API, built and tested.

Upstream: https://sourceforge.net/projects/cmusphinx/

GITHUB: https://github.com/twobob/pocketsphinx (testing extensions)

Developer API info http://cmusphinx.sourceforge.net/wiki/develop/
(how sexy is that!!!)

Building with 2012.03:

Spoiler:

do much wangling...

Testing on a kindle touch:

SAYS TESTING TO KINDLE TOUCH:

Spoiler:

Quote:

Listening...
Recording is stopped, start recording with ad_start_rec
Stopped listening, please wait...
INFO: cmn_prior.c(121): cmn_prior_update: from < 18.05 -5.35 1.02 -0.73 -0.23 -0.56 -0.04 0.09 -0.23 0.01 0.03 -0.05 -0.05 >
INFO: cmn_prior.c(139): cmn_prior_update: to < 18.14 -5.15 1.14 -0.72 -0.32 -0.59 0.05 0.13 -0.25 0.03 0.04 -0.12 -0.02 >
INFO: ngram_search_fwdtree.c(1549): 3517 words recognized (60/fr)
INFO: ngram_search_fwdtree.c(1551): 200694 senones evaluated (3402/fr)
INFO: ngram_search_fwdtree.c(1553): 400108 channels searched (6781/fr), 25135 1st, 105110 last
INFO: ngram_search_fwdtree.c(1557): 5882 words for which last channels evaluated (99/fr)
INFO: ngram_search_fwdtree.c(1560): 37237 candidate words for entering last phone (631/fr)
INFO: ngram_search_fwdtree.c(1562): fwdtree 1.69 CPU 2.864 xRT
INFO: ngram_search_fwdtree.c(1565): fwdtree 1.71 wall 2.898 xRT
INFO: ngram_search_fwdflat.c(302): Utterance vocabulary contains 170 words
INFO: ngram_search_fwdflat.c(937): 2139 words recognized (36/fr)
INFO: ngram_search_fwdflat.c(939): 94138 senones evaluated (1596/fr)
INFO: ngram_search_fwdflat.c(941): 191492 channels searched (3245/fr)
INFO: ngram_search_fwdflat.c(943): 9414 words searched (159/fr)
INFO: ngram_search_fwdflat.c(945): 8057 word transitions (136/fr)
INFO: ngram_search_fwdflat.c(948): fwdflat 0.87 CPU 1.475 xRT
INFO: ngram_search_fwdflat.c(951): fwdflat 0.87 wall 1.477 xRT
INFO: ngram_search.c(1214): </s> not found in last frame, using testing.57 instead
INFO: ngram_search.c(1266): lattice start node <s>.0 end node testing.20
INFO: ngram_search.c(1294): Eliminated 25 nodes before end node
INFO: ngram_search.c(1399): Lattice has 310 nodes, 68 links
INFO: ps_lattice.c(1365): Normalizer P(O) = alpha(testing:20:57) = -478384
INFO: ps_lattice.c(1403): Joint P(O,S) = -481774 P(S|O) = -3390
INFO: ngram_search.c(888): bestpath 0.01 CPU 0.017 xRT
INFO: ngram_search.c(891): bestpath 0.01 wall 0.011 xRT
000000037: testing
READY....

Quote:

INFO: ngram_search_fwdtree.c(430): TOTAL fwdtree 200.77 CPU 3.022 xRT
INFO: ngram_search_fwdtree.c(433): TOTAL fwdtree 206.75 wall 3.112 xRT
INFO: ngram_search_fwdflat.c(174): TOTAL fwdflat 57.46 CPU 0.865 xRT
INFO: ngram_search_fwdflat.c(177): TOTAL fwdflat 57.92 wall 0.872 xRT
INFO: ngram_search.c(317): TOTAL bestpath 74.00 CPU 1.114 xRT
INFO: ngram_search.c(320): TOTAL bestpath 74.66 wall 1.124 xRT

Special thanks to knc1 and nhsm from the support irc without whom this would never had got built.

YAY!

https://github.com/twobob/sphinxbase-kindle

https://github.com/twobob/pocketsphinx-kindle

expect a KUAL extension RSN EDIT: And now it is completed for kindle touch.

And a build for ALL KINDLES

EDIT:

In order to EXTEND THE VOCABULARY of the application.

1) Grab this corpus file. (or just make you own)

2) push it to the lmtool webservice

3) grab the tgz. Extract out the lm and dic files... Rename them gui.lm and gui.dic
replace the default ones in /mnt/us/extension/ with yours... That should be all..

twobob · 03-23-2013, 11:51 PM

Couldn't sleep.

Now it's built for all all kindles.

Meh : )

https://github.com/twobob/sphinxbase...d753b9a6805f07

and

https://github.com/twobob/pocketsphi...25123056071895

Both of which are insanely large commits and will probably explode your browser

say BUTT to a kindle 3

Spoiler:

Quote:

Listening...
Recording is stopped, start recording with ad_start_rec
Stopped listening, please wait...
INFO: cmn_prior.c(121): cmn_prior_update: from < 56.00 -3.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >
INFO: cmn_prior.c(139): cmn_prior_update: to < 17.19 -3.76 -1.19 -0.65 -0.07 -0.15 -0.15 -0.03 -0.05 0.03 -0.12 0.02 0.06 >
INFO: ngram_search_fwdtree.c(1549): 1453 words recognized (21/fr)
INFO: ngram_search_fwdtree.c(1551): 245085 senones evaluated (3552/fr)
INFO: ngram_search_fwdtree.c(1553): 409829 channels searched (5939/fr), 29705 1st, 52007 last
INFO: ngram_search_fwdtree.c(1557): 3597 words for which last channels evaluated (52/fr)
INFO: ngram_search_fwdtree.c(1560): 45307 candidate words for entering last phone (656/fr)
INFO: ngram_search_fwdtree.c(1562): fwdtree 3.17 CPU 4.594 xRT
INFO: ngram_search_fwdtree.c(1565): fwdtree 3.70 wall 5.356 xRT
INFO: ngram_search_fwdflat.c(302): Utterance vocabulary contains 25 words
INFO: ngram_search_fwdflat.c(937): 623 words recognized (9/fr)
INFO: ngram_search_fwdflat.c(939): 29138 senones evaluated (422/fr)
INFO: ngram_search_fwdflat.c(941): 27021 channels searched (391/fr)
INFO: ngram_search_fwdflat.c(943): 1473 words searched (21/fr)
INFO: ngram_search_fwdflat.c(945): 1313 word transitions (19/fr)
INFO: ngram_search_fwdflat.c(948): fwdflat 0.22 CPU 0.319 xRT
INFO: ngram_search_fwdflat.c(951): fwdflat 0.22 wall 0.320 xRT
INFO: ngram_search.c(1214): </s> not found in last frame, using but.67 instead
INFO: ngram_search.c(1266): lattice start node <s>.0 end node but.14
INFO: ngram_search.c(1294): Eliminated 97 nodes before end node
INFO: ngram_search.c(1399): Lattice has 114 nodes, 2 links
INFO: ps_lattice.c(1365): Normalizer P(O) = alpha(but:14:67) = -550709
INFO: ps_lattice.c(1403): Joint P(O,S) = -550709 P(S|O) = 0
INFO: ngram_search.c(888): bestpath 0.01 CPU 0.015 xRT
INFO: ngram_search.c(891): bestpath 0.01 wall 0.014 xRT
000000000: but

Close enough

Quote:

INFO: ngram_search_fwdtree.c(430): TOTAL fwdtree 3.17 CPU 4.662 xRT
INFO: ngram_search_fwdtree.c(433): TOTAL fwdtree 3.70 wall 5.435 xRT
INFO: ngram_search_fwdflat.c(174): TOTAL fwdflat 0.22 CPU 0.324 xRT
INFO: ngram_search_fwdflat.c(177): TOTAL fwdflat 0.22 wall 0.324 xRT
INFO: ngram_search.c(317): TOTAL bestpath 0.01 CPU 0.015 xRT
INFO: ngram_search.c(320): TOTAL bestpath 0.01 wall 0.014 xRT

NICE!

twobob · 03-24-2013, 12:49 AM

https://github.com/twobob/pocketsphinx

Will probably be refactoring this into Kual-system but there is a raw build with a testing wrapper...

OBVIOUSLY DONT USE THESE AS EXTENSIONS AS-IS
They wouldn't output anything useful.

Cheers.

We will probably have to think about TRAINING this thing at some point

pspvhp · 03-24-2013, 01:59 AM

nice job. how to use this for kt5

twobob · 03-24-2013, 05:14 AM

Quote:

Originally Posted by pspvhp

nice job. how to use this for kt5

http://cmusphinx.sourceforge.net/wiki/develop/ <-- Either write an application yourself and use the files I provide from the shinxbase... or..

Wait for me to actually do something with this myself. : )

knc1 · 03-24-2013, 09:18 AM

Quote:

Originally Posted by twobob

http://cmusphinx.sourceforge.net/wiki/develop/ <-- Either write an application yourself and use the files I provide from the shinxbase... or..

Wait for me to actually do something with this myself. : )

That is a really big application and there have been requests for voice control here in the past from time to time . . .

Maybe we can find a few pages for it in our structured wiki:
http://wiki.knetconnect.com/twiki/bi...?topic=SiteMap
(Even though it does not have a "button" yet.)

twobob · 03-24-2013, 10:12 AM

Good idea. Done that.

http://wiki.knetconnect.com/twiki/bi...ice_Activation

I'll look at making a useful extension when time/knowledge allows.

twobob · 03-24-2013, 10:09 PM

[root@kindle root]# /mnt/us/extensions/pocketsphinx/usr/local/bin/pocketsphinx_continuous -logfn /dev/null -lm /mnt/us/extensions/pocketsphinx/usr/local/share/pocketsphinx/model/gui/gui.lm -dict /mnt/us/extensions/pocketsphinx/usr/local/share/pocketsphinx/model/gui/gui.dic | cut -c 12-

Quote:

SCREEN SAVER
HOME
PAGE FORWARD
PAGE RIGHT
CLOSE THAT
WEB BROWSER
RUN COOL
SELECT SEVEN
CONNECT U_S_B NETWORKING
NETWORK DOWN
SCREEN SAVER
REBOOT
TAKE SCREENIE

It got them all right...

twobob · 03-24-2013, 10:35 PM

PHP Code:


			
SETPROP=/usr/bin/lipc-set-prop

# Whitespace == :Space:Tab:Line Feed:Carriage Return:
readonly WSP_IFS=`printf "\x20\x09\x0A\x0D"`
# No Whitespace == :Line Feed:Carriage Return:
readonly NO_WSP=`printf "\x0A\x0D"`
# Whitespace == :Space:Tab:
readonly WSP=`printf "\x20\x09"`

IFS=${NO_WSP}

# Now however you write a read command block:
read C
case $C in
  'HOME' ) "$SETPROP com.lab126.appmgrd start app://com.lab126.booklet.home" ;;
  'SCREEN SAVER' ) "$SETPROP com.lab126.appmgrd start app://com.lab126.booklet.settings?diagnosticMode=\;411" ;;
  * ) : # no-op ;
esac

I have no idea why that doesn't work.

I'll sleep on it.

Happily will take advice...

It DOES match the case... then nothing happens?

Sigh

NullNix · 03-25-2013, 09:16 AM

Quote:

Originally Posted by twobob

PHP Code:


			
# Now however you write a read command block:

read C

case $C in

  'HOME' ) "$SETPROP com.lab126.appmgrd start app://com.lab126.booklet.home" ;;

  'SCREEN SAVER' ) "$SETPROP com.lab126.appmgrd start app://com.lab126.booklet.settings?diagnosticMode=\;411" ;;

  * ) : # no-op ;

esac

I have no idea why that doesn't work.

Well for one thing that ? is an unescaped shell metacharacter. It's probably not changing, due to matching nothing, but still, it's probably safer to say

PHP Code:


			
'SCREEN SAVER' ) "$SETPROP" 'com.lab126.appmgrd start app://com.lab126.booklet.settings?diagnosticMode=;411';;

i.e. change quotation style after the command, since you no longer need parameter expansion after that point. Only, even that isn't what you want, because that's passing *one* parameter to setprop, which happens to have spaces in it. setprop isn't going to like that. Try

PHP Code:


			
'SCREEN SAVER' ) "$SETPROP" com.lab126.appmgrd start 'app://com.lab126.booklet.settings?diagnosticMode=;411';;

Shell quoting... it has defeated all of us from time to time.

knc1 · 03-25-2013, 09:16 AM

Quote:

Originally Posted by twobob

PHP Code:


			
SETPROP=/usr/bin/lipc-set-prop

# Whitespace == :Space:Tab:Line Feed:Carriage Return:
readonly WSP_IFS=`printf "\x20\x09\x0A\x0D"`
# No Whitespace == :Line Feed:Carriage Return:
readonly NO_WSP=`printf "\x0A\x0D"`
# Whitespace == :Space:Tab:
readonly WSP=`printf "\x20\x09"`

IFS=${NO_WSP}

# Now however you write a read command block:
read C
case $C in
  'HOME' ) IFS=${WSP_IFS} ; "$SETPROP com.lab126.appmgrd start app://com.lab126.booklet.home" ;;
  'SCREEN SAVER' ) IFS=${WSP_IFS} ; "$SETPROP com.lab126.appmgrd start app://com.lab126.booklet.settings?diagnosticMode=\;411" ;;
  * ) : # no-op ;
esac

I have no idea why that doesn't work.

I'll sleep on it.

Happily will take advice...

It DOES match the case... then nothing happens?

Sigh

Re-enable whitespace word splitting after the case match.
See mark-up above (quote has been changed).

knc1 · 03-25-2013, 09:29 AM

Two questions:

1) can we train this software to distinguish normal vs abnormal heart sounds?
(Do we have any pre-med students in the audience interested in some experimenting?)

2) can we feed this software with the output of geophones and train it to distinguish approaching footsteps from the other low frequency ground sounds?

twobob · 03-25-2013, 09:45 AM

Quote:

Originally Posted by knc1

Two questions:

1) can we train this software to distinguish normal vs abnormal heart sounds?
(Do we have any pre-med students in the audience interested in some experimenting?)

2) can we feed this software with the output of geophones and train it to distinguish approaching footsteps from the other low frequency ground sounds?

Jeebers. I don't know... I would suspect the main "fail" on that is in fact the microphones that are built in.

However... an "external microphone" could well be that accurate.

With regards to the training... irc.freenode.net #cmusphinx is the place to ask such questions I guess. I am still trying to get my head around making even the most basic of "Garbage Dictionaries" right now.

Ill try your other suggestions right now as this is what I am working on today.

twobob · 03-25-2013, 09:49 AM

Quote:

Originally Posted by NullNix

Well for one thing that ? is an unescaped shell metacharacter. It's probably not changing, due to matching nothing, but still, it's probably safer to say

PHP Code:


			
'SCREEN SAVER' ) "$SETPROP" 'com.lab126.appmgrd start app://com.lab126.booklet.settings?diagnosticMode=;411';;

i.e. change quotation style after the command, since you no longer need parameter expansion after that point. Only, even that isn't what you want, because that's passing *one* parameter to setprop, which happens to have spaces in it. setprop isn't going to like that. Try

PHP Code:


			
'SCREEN SAVER' ) "$SETPROP" com.lab126.appmgrd start 'app://com.lab126.booklet.settings?diagnosticMode=;411';;

Shell quoting... it has defeated all of us from time to time.

I will try that too...

Although I am happy to just get even the HOME one working for right now : )

knc1 · 03-25-2013, 09:51 AM

Quote:

Originally Posted by twobob

Jeebers. I don't know... I would suspect the main "fail" on that is in fact the microphones that are built in.

However... and external microphone could well be that accurate.

With regards to the training... irc.freenode.net #cmusphinx is the place to ask such questions I guess. I am still trying to get my head around making even the most basic of "Garbage Dictionaries" right now.

Ill try your other suggestions right now as this is what I am working on today.

Just wanted to get the readers to thinking.
Hey, if I have to think this early in the morning. . . .

Besides, it is time we gave the KUAL development forum members something to talk about.

http://kual.knetconnect.com/forums/forum/kual/

03-24-2013, 12:49 AM	#3
twobob ( ͡° ͜ʖ ͡°){ʇlnɐɟ ƃǝs}Týr Posts: 6,586 Karma: 6299993 Join Date: Jun 2012 Location: uti gratia usura (Yao ying da ying; Mo ying da yieng) Device: PW-WIFI\|K5-3G+WIFI\| K4\|K3-3G\|DXG\|K2\| Rooted Nook Touch	https://github.com/twobob/pocketsphinx Will probably be refactoring this into Kual-system but there is a raw build with a testing wrapper... OBVIOUSLY DONT USE THESE AS EXTENSIONS AS-IS They wouldn't output anything useful. Cheers. We will probably have to think about TRAINING this thing at some point Last edited by twobob; 03-24-2013 at 10:36 PM.

03-24-2013, 10:35 PM	#9
twobob ( ͡° ͜ʖ ͡°){ʇlnɐɟ ƃǝs}Týr Posts: 6,586 Karma: 6299993 Join Date: Jun 2012 Location: uti gratia usura (Yao ying da ying; Mo ying da yieng) Device: PW-WIFI\|K5-3G+WIFI\| K4\|K3-3G\|DXG\|K2\| Rooted Nook Touch	Code case statement refuses to run the apps... PHP Code: SETPROP=/usr/bin/lipc-set-prop # Whitespace == :Space:Tab:Line Feed:Carriage Return: readonly WSP_IFS=`printf "\x20\x09\x0A\x0D"` # No Whitespace == :Line Feed:Carriage Return: readonly NO_WSP=`printf "\x0A\x0D"` # Whitespace == :Space:Tab: readonly WSP=`printf "\x20\x09"` IFS=${NO_WSP} # Now however you write a read command block: read C case $C in 'HOME' ) "$SETPROP com.lab126.appmgrd start app://com.lab126.booklet.home" ;; 'SCREEN SAVER' ) "$SETPROP com.lab126.appmgrd start app://com.lab126.booklet.settings?diagnosticMode=\;411" ;; * ) : # no-op ; esac I have no idea why that doesn't work. I'll sleep on it. Happily will take advice... It DOES match the case... then nothing happens? Sigh Last edited by twobob; 03-24-2013 at 10:37 PM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
calibre's new plugin API	kovidgoyal	Plugins	26	05-07-2011 02:43 PM
Links to API's...?	theranch	General Discussions	0	05-06-2011 09:32 AM
New metadata API in 0.8 questions	kiwidude	Development	38	04-18-2011 10:42 AM
Goodreads has published an API	EricLandes	Calibre	6	01-12-2011 04:39 PM
Blackstone Audio to release audio books as DRM-free MP3s	Donnageddon	News	4	07-07-2008 12:16 PM

03-24-2013, 01:59 AM	#4
pspvhp Member Posts: 20 Karma: 30000 Join Date: Jan 2013 Device: kindle touch 5.3.2	nice job. how to use this for kt5

03-24-2013, 10:12 AM	#7
twobob ( ͡° ͜ʖ ͡°){ʇlnɐɟ ƃǝs}Týr Posts: 6,586 Karma: 6299993 Join Date: Jun 2012 Location: uti gratia usura (Yao ying da ying; Mo ying da yieng) Device: PW-WIFI\|K5-3G+WIFI\| K4\|K3-3G\|DXG\|K2\| Rooted Nook Touch	Good idea. Done that. http://wiki.knetconnect.com/twiki/bi...ice_Activation I'll look at making a useful extension when time/knowledge allows.

03-25-2013, 09:29 AM	#12
knc1 Going Viral Posts: 17,212 Karma: 18210809 Join Date: Feb 2012 Location: Central Texas Device: No K1, PW2, KV, KOA	Two questions: 1) can we train this software to distinguish normal vs abnormal heart sounds? (Do we have any pre-med students in the audience interested in some experimenting?) 2) can we feed this software with the output of geophones and train it to distinguish approaching footsteps from the other low frequency ground sounds?

Advert

Advert