Editorial: Where are the intelligent handhelds?

[Brian]

In a recent User Thought submitted to Palm Addicts, I talked about how our PDAs haven't yet evolved to bring us intelligent features like those of the Apple Newton MessagePad from almost 10 years ago. The Apple Knowledge Navigator (Quicktime Movie), while only a concept, was given as an example of a truly intelligent handheld that has yet to materialize. While the term "Personal Digital Assistant" might evoke images of a handheld that can take an active role in organizing our lives, currently it's the user who is left to do all of the real thinking and work. Where is the intelligence, and why is my PDA still nothing more than a glamorized digital day planner in many regards?

As Evan Light at Palm Addicts points out in a recent Associate Writer Thought, we're starting to see more "smart" features and applications for our smartphones and PDAs, and the upcoming Treo 700w has one example - a "friendly SMS":

"The 700w boasts a powerful and yet simple feature: the ability to, in a small series of button presses, allow you to ignore a call and SMS the caller that you're busy but will get back to them."

While this is going in the right direction, we should expect and demand even more intelligent features in the future. How, when, and where will intelligence come into play, and who will be the innovator to bring it to us? Here are a couple of areas where improvements can and will be made, and I'll discuss more in a future article.

Putting everything into context

Context is an important and powerful concept in human-computer interaction, as it is in our social interaction and perception of the world. Who, what, where, why, when, and how are all important pieces of "metadata" that help us put information and events in the proper context. Software applications and services that have context awareness operate using information about context and react accordingly. The goal is to create a computing environment that is much more seamless, convenient, relevant, and intuitive for the user.

A recent article at EContentMag.com about the MIT Media Lab's Reality Mining Project provides a glimpse at how patterns and context can be used to make more intelligent applications for our smartphones and PDAs that predict our behavior and react accordingly:

"If your cell phone knew what you were going to do at two o'clock, would that change how you planned your day? If your cell phone "predicted" correctly where you would be at a particular time of the week, how would you feel? No longer hypothetical situations, the Reality Mining experiment answers these questions."

GPS receivers will soon become a common feature in handhelds and smartphones, but mapping, navigation, or "location aware" advertisements won't be the killer applications. Where you go on a daily basis, and what you do while you are there as well as everywhere in between are important pieces of contextual information that can be used in intelligent applications. It would be nice to have a device recognize where you are, what your patterns of usage are, and then present information in a more meaningful way and in the proper context. What applications do you use most often at home, and when do you use them? What do you do in the morning before work, when you get home, or on weekends when you don't work? How could his information be used?

The article about MIT's Reality Mining project provides a few hints.

"According to data collected by Eagle and his staff, 35% of the subjects used their phone's clock application regularly, yet opening the clock application on the phone involved 10 keystrokes. The data showed that people who used the phone's clock application used it at their homes rather than at work."

Here are some other examples - use them as a guide and then use your own imagination about other possible scenarios.

• If you have a smartphone and a regular landline, it would be nice to have an option to automatically forward incoming calls to your landline while you're home. It would also be convenient if your device would automatically present you with a to-do list of tasks and reminders that you need when you get home, and switch to a special launcher screen with your "home" applications. If you are one of those people who use the clock application regularly at home, it would be nice if a large clock was always displayed on the screen in the launcher while in your home.
• In the morning when you wake up, Traffic for Treo could automatically check the traffic conditions of your regular route to work and sound an alert if you needed to leave earlier due to traffic congestion. On Saturdays and Sundays - days that you don't usually work - Traffic for Treo wouldn't launch automatically, but you could schedule an exception if you plan on going in.
• When you walk into a movie theater or when there's an entry in your calendar that you're going to a Blue Man Group show, the ringer on your phone should automatically change to vibrate mode.
• When you arrive at the office, your device would automatically switch the ringer to vibrate mode and forward important calls to the phone on your desk. You could have a list of important calls and to-dos presented to you on a special today screen to remind you of what needed to get done that day. If you check certain stock prices regularly at work instead of the clock application, the launcher screen could display the stock prices of your favorite ticker symbols.
• Combine location and context awareness with an application like HandyShopper, and you could create shopping lists for different stores that would pop up whenever you went to those stores, or as reminders the next time you drove near them, say on the way home from work. If your spouse's birthday or your wedding anniversary is approaching, you will get reminded and notified when you are in or near stores for which gift shopping lists have been made.
• Who do you call most often, when do you call them, and from where? Are they a friend, family member, or business contact? This information could automatically generate custom "favorite numbers" lists that change when you're at home, in your car, or at work depending on the time of day and the day of the week.
• If your device has multiple wireless radios, and you are constantly moving in between different wireless networks, your device should detect which network it should join and which network services and applications should be launched, depending on when and where you are, and what you'll be doing.

How?

Intelligent features like those in the Newton's "Intelligent Assistance Architecture" as well as the examples mentioned above will come through software that can accurately recognize patterns, make predictions, and understand context. It just so happens that Jeff Hawkins, one of the minds behind the original Palm Pilot, the Handspring Treo, and who no doubt has a hand in Palm's "third business", is interested in understanding human intelligence and has created a software company called Numenta. His latest startup, which was also co-founded by Donna Dubinsky, is working to commercialize and license technology with "real intelligence" based on the research going on at the
Redwood Center for Theoretical Neuroscience (RCTN). The RCTN was formerly the Redwood Neuroscience Institute (RNI), a research institute founded by and based on the theories of none other than Jeff Hawkins as discussed in his book On Intelligence. He talked about Numenta's products during the same interview in which he first revealed a secret "third business" at Palm:

"Our products are a set of tools that allow you to configure these memory systems, which we call HTM, hierarchical temporal memory. You can interface them with a thing like a camera or a microphone or sonar and it learns about its environment in the same way you learned about your environment when you were a child. It can model the environment, recognize things in it and make predictions about the future. What we're building is actually a platform. It's like a new type of operating system. It's a platform on which people can use our tools to create new applications for solving different types of problems."

As Palm's Chief Technology Officer and the co-founder of Numenta, Jeff Hawkins splits his time between the two companies, a fitting arrangement when you consider his passion for both handheld computing and creating intelligent machines. If his two passions don't influence one another and Jeff Hawkins doesn't plan on combining the two - handheld computers and intelligence machines - I would be extremely surprised and disappointed. Will this mean that Palm could become a licensee of Numenta Technology at some point in the future and Palm's mobile computing products will become much more intelligent? Once Numenta's technology becomes ready for commercialization, it would be logical for Palm to become a licensee. I can't predict the future, so only time will tell. In the meantime, expect more context aware applications, like the Treo 700w's "friendly SMS", to be added to Palm's products.

What intelligent features would you like to see for your PDA or smartphone? Please post your thoughts and ideas about intelligent handhelds.

Related: Palm's "Secret Third Business" isn't the LifeDrive

[cervezas]

Great editorial, Brian. I found myself saying "gee, I wish I'd written that." ;-)

What I most agree with is the opportunity that GPS offers for enabling a device to be somewhat aware of context and respond accordingly. As a fan of Hawkins' work on intelligence, I too believe his Hierarchical Temporal Memory (HTM) is going to have exciting applications for mobile devices in the future--and I think Palm knows this. However, I suspect that we are still many years away from seeing that come to practical fruition in Palm devices.

It's not just that HTM is still in an experimental state, it's also that I think its usefulness is going to depend on the development of complementary technologies that are out of Numenta's or Palm's control. For a computer to understand the context of a situation well enough that it can predict things like intentionality, it seems to me it needs not just good pattern recognition, but access to pretty rich data about the environment. It needs a nervous system, not just a neocortex. The "senses" an intelligent device might have need not correspond to the five human senses (GPS is a good example) but whatever they are they will involve some kind of continuous monitoring and processing. That means intelligent devices will likely include sensor technologies that mobile devices don't have today (maybe ones that don't even exist today) and it also means there are likely to be much larger power demands on a usefully intelligent device. Power has been a bottleneck for mobile device innovation in the past, and is likely to continue to be one for some time.

There's another thing: I expect it to take a while before HMT will be good enough not to be annoying. There may be a certain charm and fascination about your device misinterpreting a situation and making a goofy response to it, but that appeal is shortlived. An "intelligent" device will have to reach a certain threshold of correct interpretations before that kind of charm will be something you're happy to live with in a device you depend on and carry with you everywhere. The nice thing about "dumb" PDAs is that they are predictable. Add just a little intelligence to them and you may get some intriguing new functionality out of them, but you also lose some of the reliability you're accustomed to demanding of them. You may be able to get your puppy to bring you your slippers when you sit in your favorite easy chair, but that same level of mental capability also makes him likely to mistake them for small animals sometimes and chew them into small pieces. :-)

I really do believe that Hawkins is on to something, I just think it may take many years, and even after it comes out it will probably be relegated to novelty status for quite a while. But even before we get to HTM, I wholeheartedly agree that there are big opportunities to make software that better predicts the behavior we really want from it and GPS is one of the best resources to enable this kind of intelligence. You've got me thinking about it!

Originally Posted by Brian

As Evan Light at Palm Addicts points out in a recent Associate Writer Thought, we're starting to see more "smart" features and applications for our smartphones and PDAs....

Putting everything into context

Context is an important and powerful concept in human-computer interaction, as it is in our social interaction and perception of the world. Who, what, where, why, when, and how are all important pieces of "metadata" that help us put information and events in the proper context. Software applications and services that have context awareness operate using information about context and react accordingly. The goal is to create a computing environment that is much more seamless, convenient, relevant, and intuitive for the user.

I wanted to point out a couple of recent features that are quite "context aware" on the Treo 650.

Using a third party software, the Treo 650 is able to alter the brightness of its screen based upon the user's current ambient light level. Using the camera it detects the light level of the users surroundings and increases or decreases the screens brightness to fit the user's needs.

A second, similar feature is the abilty to adjust the phone's volume based on the background noise level. Context awareness takes the form of noting you are at you quite desk and decreasing the ringer's volume or alternatively noting the subway car your riding on is quite loud and you'll need a louder alarm and ringer.

The user also has the option to adjust the future ringer's volume while scheduling a meeting. The volume will automatically be adjusted up and down according to your needs. While it does require your forethought to include an activation code, it could well be added to a future version of a datebook application as a simple check option or setup as a preference for meetings.

These are albeit small, but none the less powerful, steps in the direction of context awareness and intelligent assistance.

[Brian]

Originally Posted by Unregistered

I wanted to point out a couple of recent features that are quite "context aware" on the Treo 650.

Using a third party software, the Treo 650 is able to alter the brightness of its screen based upon the user's current ambient light level. Using the camera it detects the light level of the users surroundings and increases or decreases the screens brightness to fit the user's needs.

A second, similar feature is the abilty to adjust the phone's volume based on the background noise level. Context awareness takes the form of noting you are at you quite desk and decreasing the ringer's volume or alternatively noting the subway car your riding on is quite loud and you'll need a louder alarm and ringer.

The user also has the option to adjust the future ringer's volume while scheduling a meeting. The volume will automatically be adjusted up and down according to your needs. While it does require your forethought to include an activation code, it could well be added to a future version of a datebook application as a simple check option or setup as a preference for meetings.

These are albeit small, but none the less powerful, steps in the direction of context awareness and intelligent assistance.

Thanks for the information. I wasn't aware of those applications. If you or anyone else could provide links or names, that would be great for Treo owners who are interested in making their device more context-aware.

Originally Posted by cervezas

Great editorial, Brian. I found myself saying "gee, I wish I'd written that." ;-)

Thanks David!

For a computer to understand the context of a situation well enough that it can predict things like intentionality, it seems to me it needs not just good pattern recognition, but access to pretty rich data about the environment. It needs a nervous system, not just a neocortex. The "senses" an intelligent device might have need not correspond to the five human senses (GPS is a good example) but whatever they are they will involve some kind of continuous monitoring and processing. That means intelligent devices will likely include sensor technologies that mobile devices don't have today (maybe ones that don't even exist today) and it also means there are likely to be much larger power demands on a usefully intelligent device. Power has been a bottleneck for mobile device innovation in the past, and is likely to continue to be one for some time.

Good points. The examples mentioned in the previous post show how context-aware applications can take advantage the Treo 650's camera and microphone, and they could also use a mesh network of IP-based sensors which are likely to grow over time.

There's another thing: I expect it to take a while before HMT will be good enough not to be annoying. There may be a certain charm and fascination about your device misinterpreting a situation and making a goofy response to it, but that appeal is shortlived. An "intelligent" device will have to reach a certain threshold of correct interpretations before that kind of charm will be something you're happy to live with in a device you depend on and carry with you everywhere. The nice thing about "dumb" PDAs is that they are predictable. Add just a little intelligence to them and you may get some intriguing new functionality out of them, but you also lose some of the reliability you're accustomed to demanding of them. You may be able to get your puppy to bring you your slippers when you sit in your favorite easy chair, but that same level of mental capability also makes him likely to mistake them for small animals sometimes and chew them into small pieces. :-)

Agreed. It's a fine balance that will be difficult to "get right", and it would also be a good idea to give the user the power to over ride these features or turn them off entirely.

I really do believe that Hawkins is on to something, I just think it may take many years, and even after it comes out it will probably be relegated to novelty status for quite a while. But even before we get to HTM, I wholeheartedly agree that there are big opportunities to make software that better predicts the behavior we really want from it and GPS is one of the best resources to enable this kind of intelligence. You've got me thinking about it!

I hold the same belief myself, and based on some of his comments it sounds like the brain research has already paid off, but to what extent, I guess we won't know until products incorporating HTM and Numenta Technology reach the market. Some of the ideas and examples I mentioned in my article could be accomplished through software that mimics intelligence, much like the Newton Intelligent Assistance, and I hope more software developers think of new ways to mimic intelligence and use context in their applications. It's great to hear that you're thinking about it!

[Dick Tracy]

SharkMsg is available for "friendly SMS" functionality on the Treo 600 and 650 now.

OMG i cant wait till such devices come out.

For example I have always wanted a feature that would turn my PDA to silent when in class, and loud when im not

Andrew

[Bob Russell]

Originally Posted by Andrew

OMG i cant wait till such devices come out.

For example I have always wanted a feature that would turn my PDA to silent when in class, and loud when im not

Andrew

That's an awesome feature idea, Andrew. Hadn't thought about that one, but it would really be neat to be able to set quiet zones, based on certain times even, so you could program it to turn off the ringer every time you are in a meeting based on your calendar entries! I'd like to be able to program my home phone ringers that way also!

[Colin Dunstan]

Originally Posted by BobR

That's an awesome feature idea, Andrew. Hadn't thought about that one, but it would really be neat to be able to set quiet zones, based on certain times even, so you could program it to turn off the ringer every time you are in a meeting based on your calendar entries! I'd like to be able to program my home phone ringers that way also!

Are you sure you cannot do this with WM- or Symbian-based smartphones? It's called "Profiles" and it allows you to customize your settings, including ring tones and alarms, according to time schedules.

Of course, if we were speaking of truly "intelligent" handhelds, as referred to in Brian's excellent editorial, the handheld should automatically detect it's present location and change its settings accordingly. For instance, a sender could be installed in the classroom notifying the handheld that it should switch on the "silent" profile.

[Brian]

The application that will automatically adjust the Treo's screen brightness depending on ambient light levels and ringer volume depending on ambient noise levels is Brightcam.

Originally Posted by Brian

The application that will automatically adjust the Treo's screen brightness depending on ambient light levels and ringer volume depending on ambient noise levels is Brightcam.

It is also the same app that allows your databook to adjust the volume too.

[Evan]

Originally Posted by cervezas

Great editorial, Brian. I found myself saying "gee, I wish I'd written that." ;-)

I've had exactly the same reaction to each and every editorial Brian has produced so far. At least he keeps me honest.

Brian: I'll follow-up in a day or two with something, I hope. As I e-mailed you, I'm still playing catch-up with the interesting sources that you've linked to in some of your previous articles.

[Brian]

Thanks Evan, I'm looking forward to your comments and thoughts. Welcome to MobileRead!

[Brian]

Russell Beattie has posted a short article called Knowledge Navigator Redux and asks if intelligent "agents" could be the "Next Big Thing".

I hope so Russ.

[Evan]

I caught that blurb as well. A few of my colleagues and I hope so; we're discussing doing a group project involving intelligent agents as part of our Master's Program in Computer Science--that is assuming that we all get admitted to the program.