Virtual Environments Projects - Brent Rossen - Spring 07
Virtual Breast Mass Scene

Wiimote Tools for the Virtual Patient Project

By Brent Rossen

Nintendo Wiimote                        
Download VRPN WiiServer - 1 MB (source and executables included, use start.bat to execute, requires VS 2005 SP1 to compile)

The Purpose
This project was primarily created to allow the Wiimote to be used as a novel interface for a virtual environment. The goals were to create a Wiimote server, allow interaction with a virtual environment, and provide haptic as well as audio feedback. Six degrees of viewing freedom and stereo vision are provided via a Head Mounted Display and optical tracking system. Sound is provided through FMOD. Much of this project is built on top of code from the Virtual Patient Project.

Interaction Instructions
First, put on the wrist strap.
Initiate Tools: say, "I would like to give you a physical exam now"
Hammer: strike by swinging quickly at the knee
Ophthalmoscope: rotate the wrist to rotate the tool around the eye
Switch tools: hold the trigger ("B") button and throw the tool over your shoulder (HOLD ON TO THE WIIMOTE)

The Wiimote Server
To acquire data from the Wiimote, a C# Wiimote API (written by Brian Peek) was used, the original api can be acquired at To create a modular system, VRPN was used to allow network communication with the WiimoteServer. Since VRPN is written in C++ a managed C++ wrapper was necessary for the C# code to communicate with VRPN. The VRPN managed wrapper is included in the WiiServer download above. The interface can be seen to the right. After downloading the WiiServer, just double click the included .bat file to start. To use this system you will need a blue tooth adapter and your own Wiimote. The blue tooth stack Blue Soleil is known to work with Wiimotes, though many others may work as well. Even if you don't plan to use the WiiServer's network capabilities, mouse control and hotkeys for Windows Media Player are included. Use the directional pad to control the mouse pointer, the trigger (B) for left click and A for right click. For more details, email brossen AT cise dot ufl dot edu.

The Virtual Environment
The environment is the Virtual Patient Project (screen shot seen in the header). Please see to read more about the Virtual Patient Project and it's capabilities. The virtual patient project is 400 MB and so a download is not provided. This download may become available in the future.

To summarize, the Virtual Patient Project allows a medical practitioner to interact with a life sized virtual patient in a natural manner (using natural language processing). The system begins as a regular interview, to switch to physical exam mode, simply say, "I would like to give you a physical exam now" and the first exam tool will appear.

The Wii Tools
Two tools, a reflex hammer and an ophthalmoscope were developed to interact with the virtual patient. Interaction with these devices is very intuitive using the wiimote.

To strike with the hammer, simply swing as though striking. You must swing hard enough that it would result in an actual strike, otherwise, it will just touch. The wiimote vibrates when a strike occurs.

The ophthalmoscope can be rotated around the eye of the patient by rotating the wiimote left and right. To switch between the tools, press the trigger ("B") button and throw the tool over your shoulder. This can be thought of as throwing away the tool you are holding.

This project used many different technologies: VRPN, Wiimote, C#, C++, Managed C++, and the Virtual Patient Project (which is itself many combined technologies).

Writing managed code was a significant hurdle, but once it was written, it worked well. My thanks goes to Wang Xiyong, a Graphics Lab colleague, for providing samples of managed VRPN code.

Reflex Hammer              
Most of the work went into creating the tools, creating the interaction, and then fine tuning the behavior of the tools. There are several real time systems interacting, and their 100hz or more worth of data needed to be restricted to discrete inputs. For example, when the A button is pressed on the Wii remote, it sends the press with every update, since that would be too much network traffic, only the first press, and the release are actually sent across the network. Problems similar to this were overcome many times during the project. For example, to detect a strike or throw away, the difference between one reading and the next should be large, that works fine, but with accelerometers, a serious jolt will make the next few readings result in large discrepancies as well as "junk" data, this was accounted for by capturing the first occurrence and handling the rest cleanly.

Hardware Used
Z800 HMD, Bluetooth Adapter, Nintendo Wiimote, Optical Tracking Cameras, speakers, and two desktop computers. An IR Bar was constructed for this system but was not necessary for the implementation of these tools.

Time Worked: ~40 hrs


© Brent Rossen • 2006, All rights are reserved