My friend Michelle’s son Alex is autistic. We visited them on Saturday night. While Michelle and the rest of us were doing whatever, Alex was playing with an R2D2 toy.

This toy performs actions based on speech recognition.

Alex: “Hey R2″
R2: affirmative beep (sounds like “Boo-Boop”)
Alex: “Do yamgine?”
R2: negative beep (sounds like “Bee-Bawp”)
Alex: “Do you ‘magine?”
R2: negative beep
Michelle: “Do you remember?”
R2: affirmative beep
Michelle: “Darth Vader?”
R2: scared noise and shaking head and moving in circles (sounds like “wa-a-a-hoo!”)

REPEAT w/o michelle for like an hour

the kid was HIGHLY motivated to say the right stuff, and kept at it.

what if there was a web framework (through flash or silverlight?) and a teacher could program in like the vocab, and a storyline (maybe somehting like tarheel reader could help create some default storylines (TANGENT: we should try to have some of your projects seed other as possible) and then the kid tries to say it, this would be for kids with speech issues.

maybe the kid’s a detective, and has to go around quesitoning people, maybe at first it acknowledges bits and pieces (i.e.”Hey ” “where were you?” “?”) then later it only acknowledges at the end

maybe how tolerant it is of sound deviating depends on the kid’s level and their recent success rate…

I love it! This could make a very interesting student project and would take our work in a different and exciting direction.

Who would benefit? students with visual impairment – and students that need more cuing than they get from visual supports on the printed page

Use vibration feedback in TTS, screen readers, digital auditory text

• To cue for bold text, boxed info, important information
• Software could “read” the text/page and insert vibration to cue the student

But more importantly, to allow the student to tag the auditory text – when they “re-read” the text, the tags trigger vibration. Use different types of vibration (intensity, pattern, etc) for different types of tags.

• Tag as a highlighter, tag a phrase or a paragraph
• Tag specific info they need to capture (main characters, literary action)
• Tag for references
• Tag as notetaking – retrieve only tagged information for later?

Karen suggested long ago and I failed to blog about it, that the WiiMote could be useful for “keeping beat with reading. Kid reads along with program, keeps beat. Shakes the controller on the key words. That way you know the kid is reading.”

Seems to me there is great synergy between these ideas. This would make a great student research project.

How about putting the samples on the 16 pads you get with 2 DDR pads? Then we could have a game that was something like Guitar Hero where you put the samples in the right places. DJ Hero? Or maybe DJ Revolution? I think our blind visitors would have a ton of fun on Maze Day trying to “sing” along with Harder, Better, Faster, Stronger by stepping on the DDR pads. Think Daft Hands or Daft Bodies on the DDR pad.

Or how about making an AAC device more fun by changing the pitch of the voice like this? Or somehow linking a “word wall” to music so when you put words together in appropriate ways they go with the music?

Suppose you could provide a list of words or word combinations to a web site and it would automagically assign them musical pitches like this. What cool games could we make where students play with words while making music? How could we integrate reading into it? Perhaps the “score” (the words to be spoken) is presented something like the Guitar Hero notes so you have to at least recognize the words to know which to play.

How do we make such a game switch accessible?

I’m thinking out loud here. Help me out folks, some of you have to be more musical and creative than I am. Post a comment.

Come see if you can find “it” at Maze Day 2009 on the University of North Carolina Campus at Chapel Hill. What are we looking for? We’ll be looking for bus stops, intersections, building entrances and much more. Using Trekker Breeze, students, teachers, parents and peers will participate in a “Talking GPS” scavenger hunt on campus. During the activity we will use the newest in GPS technology for the blind and visually impaired to find out “What’s around.” Join us to learn about this exciting technology and be ready to have some fun in the process.

We plan to have a wide variety of accessible fun, educational, and exercise activities including:

• Trekker Breeze Scavenger Hunt (click for more details)
• Hark the Sound
• Braille Twister
• Sonic Zoom
• Descent into Madness
• Last Crusade
• Making music on the DDR pad.
• Two player Simon combined with an upper body workout.
• Accessible Guitar Hero Freeplay.
• Life size and computer mazes.
• Rockin’ SafaWii: A rocking-horse (or therapy ball) game that combines exercise and acoustic localization.
• Move to the Music: An accessible version of Dance Dance Revolution for a good exercise workout to music.
• Sweet Beat: an edible drum sequencer.
• SamiSays: a program for recording stories with sound effects.
• And there’s always a little bit more… but you’ll have to attend to see what that is.

Lunch will be provided. Free!

To register send email to MazeDay@cs.unc.edu. Include the number of elementary, middle, and high school students you will be bringing and the number of adults. Let us know if you’ll need parking for a bus or van.

Date: Tuesday April 28, 2009
Time: 9am until 2pm
Location: Sitterson Hall on the UNC Chapel Hill campus (click here for directions)

Also, please complete the following forms and bring them with you:

• Photo Consent – We will only include your children in photographs of the event with your permission. We also have the photo consent form in Spanish.
• Experiment Consent – Coming soon.

So I hacked on CamKeys to add a GUI for configuring all sorts of events. I’ll include a few screen shots below. Now the teachers are recording their own speech and music sounds and hooking them to events on the DDR pad or camera.

The CamKeys roots of this project were funded by the Mozilla foundation.

The source is in CVS at sourceforge uncassist.

This year, in addition to many of the old favorites, we expect to have many amazing new activities, including:

• Composing music on the DDR pad.
• Two player Simon combined with an upper body workout.
• Guitar Hero made accessible.
• Life size and computer mazes.
• Carolina Rocker: A rocking-horse (or therapy ball) game that combines exercise and acoustic localization.
• Carolina Beat: An accessible version of Dance Dance Revolution for a good exercise workout to music.
• Virtual Beep Ball.
• Sweet Beat: an edible drum sequencer.
• A new kind of Braille display.
• SamiSays: a program for recording stories with sound effects.
• And there’s always a little bit more… but you’ll have to attend to see what that is.

Lunch will be provided. Free!

Registration is closed.

Date: Wednesday April 30, 2008
Time: 9am until 2pm
Location: Sitterson Hall on the UNC Chapel Hill campus (click here for directions)

Also, please complete the following forms and bring them with you:

• Photo Consent – We will only include your children in photographs of the event with your permission. We also have the photo consent form in Spanish.
• Experiment Consent – High school and middle school Maze day participants are invited to participate in a fun research study during Maze Day. This study is part of an activity called Metric Olympics where you will estimate the sizes of different things. Not only will you win prizes, but you will help researchers understand how students learn about size and estimation. Interested participants should read and sign the consent letter. Join us in having fun while helping out researchers.