Entries from January 2006

I read a nice paper about a project of the Swedish Library of Talking Books about their Flash games for kids who are visually impaired. How about a version of Hark like this? Would Flash or Java-Script be more portable?

Look at Typing the Dead for ideas for how to make a typing tutor fun.

I got to meet Jake and his mom. We believe some interesting switch assessible games would be a great motivator for him to learn to use his devices. Ideas we came up with include:

• Castle of the Winds a simple “adventure” game.
• A basketball game with some simple physics. You set up your player, set the angle, how hard to throw the ball and then you get to see it shoot. 3D could be cool here but not mandatory.
• “Scorched Earth” the old classic game where you aim your cannon and fire hoping to destroy the computer’s cannon before it gets you.
• How about a bot for playing an online multiplayer shooter that allows Jake to exert control over what is happening but frees him from having to quickly aim and shoot? The robotics crowd calls this “supervisory control” and consider it for situations where the operator can’t control a remote robot in real time (for example on Mars). This could make a cool research project.
• Tic-Tac-Toe.
• Nibbles the classic “Snake” game where you steer a snake to “eat” while avoiding running into barriers or himself. Slow this down and it could be fun and help with learning to control the buttons.

Skimming in text for people who are blind or reading help for people with learning disabilities. Automatic summarization or maybe Cliff Notes merged with a book so that you can read at the summary level or down at the detailed level. How can students who are blind skim for answers to questions? Can this help people with learning disabilities? Sort of Tell what you’re going to tell them, tell them, and tell them what you told them.

Many people are permanently or temporarily unable to speak. Examples include:

• hospital patients who have a trachea tube or a broken jaw,
• people who have had a stroke, or
• people with CP, ALS, traumatic brain injury, etc.

It can be particularly stressful for a hospital patient who needs to participate in important health-care and quality-of-life decisions to be unable to communicate.

Devices have been available for many years to enable people to communicate without speech. These range from simple aids such as pointing to words or pictures on a message board and writing by hand to high-end systems with dynamic displays that produce synthetic or recorded voice messages when the user selects a word or symbol.

These devices are rarely available to people when they are lying down (e.g. hospital patients or anyone at night). They are hard to see, hold, and operate while prone.

We could combine a simple communication device with a display that projects onto the ceiling (like those clocks that display the time on the ceiling). In the simplest case it might display only “yes” and “no” and allow a user to select between them. Extension to multiple words, phrases, or even individual letters would be easy. Synthesized or pre-recorded speech and alert signals could be easily added.

This device is about the size of a clock radio and sits beside the bed. It has standard 1/8” mono plugs for connecting a variety of switches. The projector is normally off and is quiet when in operation. It displays single color, high-contrast text on the ceiling. In “scanning” mode it cycles among the choices until the user makes a selection with a single switch. In “step” mode the user has two switches, one to move among choices and another to select one. Upon selection an audible or visual display is generated for the communication partner.

Macros

Windows “macro programs” like Macro Scheduler and Macro Express allow users to record sequences of keyboard and mouse events for later playback. Judging from the number of them on the market I’m guessing they are a pretty popular way to automate recurring actions.

Macros and Python

It occurs to me that we could implement such a thing using pyAA and make it much more powerful and useful. A Python module on top of pyAA that makes it easy to automate GUI‘s would be a really great tool for automating otherwise unscriptable applications. For example, I’d like to convert my midi music files to “Impulse Tracker” files using ModPlug Tracker. Now this app might be scriptable but I don’t really want to learn their scripting language. I also don’t want to repeat the multi-step GUI interaction required for dozens of midi files in Hark the Sound.

What I’d like is a simple way to grab the appropriate GUI handles so that I can run the program from a Python script. Conceptually the script is quite simple. Something like:

for filename in os.listdir():     basename, ext = os.path.splitext(filename) prog.Open(filename) prog.SaveAs(filename + '.it') 

Assuming I had some tools to make it easy to define how to activate the Open and SaveAs actions in this particular program I’m automating. Pete’s clever naming scheme from Clique might work here, or it might work a bit like the wxPython XML resource editor “xrced:http://xrced.sourceforge.net/” in reverse. You start up a program, hook a watcher to it, and then use the GUI to accomplish tasks. The result is some XML that describes the events required for each. Now in your script you have an easy way to hook to the XML to invoke the actions you recorded earlier.

Macros, Python, and Accessibility

Given such a tool, improving the accessibility of common applications might be much simpler. For example I hear that iTunes has a really terrible interface. Adding a few keyboard-based accellerators to it could be quite easy and realy helpful.

Really cool display based on candles

A video of the display in action.