Teenager Plays Mind-Controlled Video Game

space-invader.jpg

I previously reported on BrainGate-the neural interface that connected a paraplegic’s brain to a computer for cursor control.  Researchers at Wash U have reported a sytem that uses surface electrodes to provide similar function. 

“With approval of the patient and his parents and the Washington University School of Medicine Institutional Review Board, Leuthardt and Moran connected the patient to a sophisticated computer running a special program known as BCI2000 (developed by their collaborator Gerwin Schalk at the Wadsworth Center, New York State Department of Health in Albany) which involves a video game that is linked to the ECoG grid. They then asked the boy to do various motor and speech tasks, moving his hands various ways, talking, and imagining. The team could see from the data which parts of the brain and what brain signals correlate to these movements. They then asked the boy to play a simple, two-dimensional Space Invaders game by actually moving his tongue and hand. He was then asked to imagine the same movements, but not to actually perform them with his hands or tongue. When he saw the cursor in the video game, he then controlled it with his brain.”

The boy, a 14-year-old who suffers from epilepsy, is the first teenager to play a two-dimensional video game, Space Invaders, using only the signals from his brain to make movements.   Getting subjects to move objects using only their brains has implications toward someday building biomedical devices that can control artificial limbs, for instance, enabling the disabled to move a prosthetic arm or leg by thinking about it.

I also reported on the monkey with a direct link info here.

In the big picture this is another variation of alternative control systems.  I often speak of “gesture controled surgery” where gestures are converted by video to task controls for surgery.  This another way to potentially control surgical taks without traditional hands or instruments. 

Share

Gesture Control Everywhere

drag-draw-lr.jpg

Engadget just reported on Phillips prototype toy Drag and Draw.  Looks like yet another of what I call “gesture control” systems.  In the photo I see the pen and a control box.  Wonder if its got a video tracking system and projector in the box?  This control technology once mass produced will definitely make it into the operating room (see previous posts on this idea below).

Share

Hand Gesture Control Video

[wpyt_profile1]1qjEiF8orHg[/wpyt_profile1]

I previously wrote on the fogscreen display system. Just as interesting as the display system is a demo of the hand gesture control concept. Here hands point in space and the display responds with electric ink. There must be a video tracking system trnslating the movements into mouse controls. I can forsee systems where control systems like this would follow a surgeon’s hands.  Just like Tom Cruise in Minority Report (before he jumped on couches).

Share

Another Vapor Projector at Nextfest with Gesture Control Video

[wpyt_profile1]khkx3zDMs5w[/wpyt_profile1]

In addition to the fogscreen display overhead as you entered Nextfest there was another water vapor-based projection system.  Heliodisplay from IOtechnology also projects moving images onto a stream of vapor.  They also have an interactive gesture controled version.  Generates the same ideas to me for a gesture control surgical system.  Here my video of me messing with the images.  You can see the mirror projector in the back.

Share

NextFest 2: Hand Gesture Controlled Surgery and Tom Cruise

min-report-1.jpghand-control-1.jpg

Tom Cruise Minority Report                          Raytheon Army Version

In my last post on Fogscreen I shared my vision for a future of alternative display technologies opening up new options for surgical imaging. Let me elaborate on what I have called the “hand gesture control” concept. In traditional surgery there is a direct interaction between surgeon and patient. Laparoscopy (“keyhole surgery”- my specialty) is the revolutionary breakthrough where for the first time there is a technological interface between surgeon and patient. I believe this is a radical enabling step since this interface can now be modified endlessly by technology. Today we remain shackled by the surgeon using his hand to manipulate the tools. The DaVinci robot was one of the first steps whereby the direct line of touch between surgeon and patient has been interrupted by a computer controlled interface.

We should further explore the possibilities of the hand gesture system. Fogscreen employes this in their interactive version where you write with your finger like a cursor in space and the lines appear where you gesture. Clearly, a computer and camera system track movements and convert them into cursor paths. First shown in Minority Report, Tom Cruise gestures in space and pulls, pushes, and manipulats computer images projected in space. Wll folks, prototypes of this already exist.  Raytheon has developed one for the army (they get all the good stuff first)– but the concept directly applies to surgery. Project an image in space- gesture with hand – direct laser or robotic tool- it opens up endless possibilities for free form and novel operative procedures and I predict it will be a major concept for future surgery.

Share

Alternative Displays -Fogscreen:Update 1 from Nextfest

fogscreen-2.jpgfogscreen-3.jpgfogscreen-41.jpg

Projection image                Close-up of fog               projector

I have a series of Nextfest updates coming.  These are the first of my photo/video reports.  An outstanding show at the Javits Center in NYC fromWired with advances in technology and cutting edge developments. The first impresion is the fogscreen display overhead as you pass into the exhibit hall.  Fogscreen generates a water vapor mist with ultrasound (like an ultrasound humidifier) and then projects videon onto the mist-like screen.  Ok Doc cool but isn’t this just a party trick why sould we care?  The more intriguing aspect (besides the techno club rave entrance) is that this is one of the first commercialized alternative video display technologies.  I have been speaking for years that I envision future endoscopy and medical imaging to employ several alternative display technologies.  Today we are using just CRTs, LCDs, and the occasiaonal HMD (head mounted display).  Thinking outside the box a bit — if we can transform the imaging display then we can create new funtionality.  I envision augmented reality displays joining live visio with projected images. Just as fogscreen images on moving water vapor we could someday image on living tissues.  Interactivity has been added by the fogscreen people for entertainment– you draw with your finger in space and the computer “sees” and interprets where you are.  The same could be done for surgery by pointing and drawing in space.  Remember Tom Cruise’s hand movements directing the image recalls in Minority Report…  Its coming…

Share