dyadica.net

Abstract:

The aim of this research is to investigate the possibilities for utilising and adapting contemporary gaming technologies for use as adaptive assistive devices, to facilitate effective interaction for those with physical and/or learning disabilities. An ever evolving user-sensitive design methodology and the evaluation of resulting design iterations have highlighted computer gaming technologies as suitable for these purposes.

The Nintendo Wii remote controller (Wiimote) offers motion sensing capability, enabling the user to interact with the system via movement and pointing. In addition, accompanying software solutions have been identified to provide facility for run time applications to robustly determine the motions which a user is performing from a set of developer-provided examples.

The facility to purpose train an individual device is suggested, which in turn would provide the capability for making potential design solutions truly adaptable. This adaptability is further increased by the volume of physical data which can be obtained from the device itself, allowing for further additional software solutions to interpret and effect system input. In addition, visual, auditory and haptic feedback is also available.

Applications currently being developed involving the Wiimote are a virtual cane/guide dog, a surface free interface, stroke rehabilitation tools, sign language translator, tremor compensation and gesture recognition. A custom Wiimote extension is being designed to further development of additional suggested applications.

1. The Wiimote as an Assistive Device

The Nintendo Wii Remote controller (Wiimote) has been investigated as a feasible platform for the development of assistive technology, by interfacing it with a contemporary personal computer and by detailing the technologies involved. Using this device has many advantages, not least that it is mainstream, easily available and relatively cheap.

1.1 Interfacing a Wiimote with a PC

Investigation commenced by attempting to interface the Wiimote with a standard PC. This was found to be a simple task and achieved via utilising the Wiimote’s Bluetooth capability and an input emulator. The Wiimote contains a Broadcom 2042 chip designed for use with devices that follow the Bluetooth Human Interface Device (HID) standard; examples of this include keyboards and mice. As the Bluetooth HID is directly based upon the USB HID luckily much of the same documentation applies, thus providing great resource for driver development.

Interface is achieved by setting the controller into discoverable mode by either utilising the Wiimote’s sync button or by pressing the Wiimote’s 1 and 2 buttons at the same time. Once in this mode (indicated by flashing LEDs) the hosts Bluetooth HID driver can query the Wiimote.

The Glove Programmable Input Emulator (GlovePIE) acts a bridge between the Wiimote device and Windows applications via use of custom scripts. It was originally developed for the emulation of Joystick and Keyboard Input with the Essential Reality P5 Glove. GlovePIE has been further developed to support the emulation of a variety of devices including the Wiimote.

Viewed as a short term solution, the use of GlovePIE provides facility for rapid prototyping and concept testing to be undertaken, in tandem with the development of a more robust Windows driver, Wii Connect. Following iteration 0.1 (see Battersby, 2007), Wii Connect has been used to facilitate further research and development.

1.1.2 Wii Connect

Utilizing Peek’s (2008) Managed Library for Nintendo’s Wiimote as its heart, Wii Connect is a functional windows-based driver application written in C#, providing facility for keyboard, mouse and joystick mapping.

In addition Wii Connect also provides facility for peripheral input data capture and a basic capacity for the replay and analytics of any captured data.

As with GlovePIE, Wii Connect currently utilizes the Parallel Port Joysticks driver (PPJoy) to provide joystick style functionality, via its virtual joystick interface. However it is aimed to remove this dependency within future iterations.

Wii Connect not only provides facility for the Wiimote to be mapped to the windows system but also supports multiple extensions such as the Wii Classic Controller and Wii Nunchuck which may also be mapped.

Mapping functionality is provided via the use of several panels each of which enable a user to configure each of the Wiimote’s physical inputs to any desired keyboard or mouse input combination. However, Joystick functionality is tied to a defined configuration within version 0.1 (see figure 1).

In effect this enables the Wiimote to be seen by the system as any of the highlighted devices, limited only by the volume of inputs available from the device.

Customizable feedback is provided by the feedback mapping panel, enabling a user to set flags which in turn allow the system to operate any of the Wiimote’s feedback mechanisms, such as the rumble feature.

Figure 1: The Wii Connect Driver

1.2 Evaluation of Wiimote Sensor Technology

With basic Wiimote – PC interface achieved, the Wiimote’s accelerometer and optical sensor technologies were evaluated.

1.2.1 The ADXL330 Accelerometer

The ADXL330 (see figure 2) contains a micro mechanical structure supported by silicon springs. The sensor works by measuring the force exerted on the springs by a test mass. Differential capacitance measurements allow the net displacement of the mass to be converted to a voltage, which in turn is then digitized. Due to the sign convention used, this quantity is proportional to the net force exerted upon the Wiimote. Thus, when at rest on a flat surface, the accelerometer reports vertical force of +g and when in freefall reports a force of nearly zero.

Figure 2: The ADXL330 accelerometer

As the Wiimote’s ADXL330 only measures linear accelerations along three fixed axes it is unable to distinguish between linear motions and rotations, and so it is capable of measuring only pitch and roll angles. The implication of this is that the ADXL330 alone cannot be used to track the (relative) position and orientation of the Wiimote in 3D space.

1.2.2 The PixArt Optical Sensor

The PixArt Optical Sensor is an Infrared camera situated at the front end of the Wiimote. The camera is connected to an integrated image analysis chip that has facility to identify up to four individual Infrared light sources and report position, approximate size and level of intensity. The light sources are provided in the form of two clusters of Infrared LED’s situated at two opposite ends of a stationary bar.

The Wiimote’s image sensor is used to locate each cluster within the Wiimote’s field of view. The light emitted from each cluster is focused onto the image sensor which sees the light as two bright dots separated by a distance. A second distance value is provided in the form of the known fixed separation between each cluster. Triangulation is then used to calculate the distance between the bar and the Wiimote.

The rotation of the Wiimote with respect to the ground is calculated from the relative angle of the two points of light on the image sensor. This system allows the Wiimote to function as an accurate pointing device whilst also providing facility for the much needed yaw value.

1.3 Practical Investigations of Wiimote Sensors:

In order to test the interface functionality of the two Wiimote sensors a simple test environment was created that consisted of a virtual representation of the Wiimote within 3D space. The environment was constructed within Autodesk’s 3D Studio Max application and interactive capability provided by importing the environment into Adobe Director, where responses to system input could then be coded. Input used to trigger these responses was provided by the GlovePIE application using scripts developed to simulate generic mice, keyboard and joystick interactions.

Successful performance was indicated by slaving the motion of the virtual Wiimote to that of a real world counterpart. Results of this initial testing were good but have highlighted a dramatic need for dampening to be applied to the accelerometer output.

Pointing functionality was tested by the development of a GlovePIE script to facilitate the use of the Wiimote as a Windows operating system mouse. Successful control of the windows operating system was demonstrated.

1.3.1 Development of USB Powered Sensor Bar:

In order to facilitate use of the Wiimote’s optical sensor technology, development of a hardware solution was needed to provide the required infrared light sources. This was achieved via the modification of an existing product to be powered either by a USB connection or batteries (providing a wireless and portable solution however can also easily be achieved via use of two infrared LED clusters; see figure 3.1-3.6).

Fig 3.1-3.6: Portable USB powered / Wireless battery powered Wii Sensor Bar

1.3.2 Further Investigation of the Infrared Camera:

As the Wiimote camera system uses infrared light sources to calculate both variables for position and rotation it was conceived that by using movable sources of infrared light, data equivalent to that needed to operate contemporary multi-touch devices, such as the Microsoft surface or the Apple IPhone, could be produced, but without need for physical input as required by touchscreen systems.

In order to determine if the concept is feasible initial tests were conducted using the Wii console’s sensitivity adjustment screen and two movable infrared light sources. When used with a standard Wiimote the sensitivity screen displays two dots, each representing an IR cluster.

Due to the current location of the infrared light sources being fixed the resulting dots are invisibly chained together. Movement towards the console’s sensor bar by a Wiimote results in a greater separation between the dots and an increase in scale.

For this test the Wiimote is located in the position usually occupied by the sensor bar. Then each light source is independently moved within the Wiimote’s field of view. Success is indicated by the independent movement and change of scale of the two dots on the sensitivity screen.

Following this success the concept is currently being expanded further to utilise reflected light, thus removing the need for movable physical light sources being attached to the user. This will result in facility for a surface free application to be produced which would have the capability to utilise hand gestures for operation.

Figure 4: The Wii Connect Infrared Panel

With this in mind the Wii Connect device driver has been modified via inclusion of an additional panel and facilities offering greater control over the Wiimote’s infrared capabilities.

During this enhancement, a need was highlighted for facility to configure the device’s operational range and is currently being investigated.

1.4 Wiimote Sensor Technology – Summary:

The initial testing of the Wiimote sensors highlights the Wiimote and its technologies as more than adequate for the development of assistive concepts and solutions. The important sensor features to note are:

• The ADXL330 accelerometer can be used independently to measure and provide the Wiimote’s pitch and roll.
• The PixArt optical sensor can be used independently to measure and provide the Wiimote’s pitch and yaw.
• Both sensors can be used in conjunction to provide the required 6 degrees of freedom needed to orientate a body within 3D space.
• If the Wiimote contained gyroscopes then tracking based on double integration of linear and angular acceleration values would be possible.
• The Wiimote provides facility to be used as a 2D pointing device via use of the optical sensor and can easily provide 2D mouse-like interaction.
• By varying the position of the infrared light sources within the Wiimote’s field of view positional and rotational data can be produced equivalent to that needed to operate a surface free multi touch system.
• Software developed for performance evaluation within the aforementioned studies could easily be adapted for use with the Wiimote.

2. Feedback Functionality of the Wiimote:

In addition to providing system input the Wiimote is also capable of providing basic audio, visual and haptic feedback in the following forms:

• Auditory – A small low powered speaker
• Visual – 4 blue light emitting diodes
• Haptic – A rumble facility provided by an offset motor

Initially functionality of each was tested via the development of bespoke GlovePIE scripts. Testing was then repeated using the Wii Connect driver with the exception of the speaker.

The speaker was found to emit poor quality sound, especially at high volumes, however does provide facility for audio content to be transmitted from the host to be played back via the Wiimote.

By default the Wiimote indicates its player assignment via lighting the corresponding LED. LEDs are also used to indicate the power level of the Wiimote’s battery upon power up. This is done by lighting the LEDs to represent power increments of 25%.

The rumble device can be set to either an on or off state. Manipulation of the frequency of operation can be used to provide pulses to create the illusion of differing strengths of rumble. Investigation of additional methods for providing strength of rumble is still ongoing. Performance of the rumble device manipulation improved with the transition to the Wii Connect driver.

3. Expansion Capabilities of the Wiimote:

The Wiimote features an expansion port. This provides facility for the connection of additional auxiliary controllers to augment the input capabilities of the Wiimote. The Wiimote’s assistive capabilities can also be expanded via implementation of software solutions.

3.1 Functional Attachments:

All the functional attachments utilise the Bluetooth interface of the Wiimote to communicate with the Wii console, thus by result allowing them to be much simpler and cheaper to implement than the Wiimote itself.

The expansion port consists of a custom connector with six contacts. Two of these contacts are slightly longer than the others, allowing them to make connection first. Attachments can be added and removed from the Wiimote whilst in use without resulting in device damage.

The main attachment to date for the Wiimote is the Nunchuk controller, so called as it gives the Wiimote unit the appearance of a nunchaku once connected. The primary component of the Nunchuk is an analogue thumbstick. Similar to a contemporary joystick the thumbstick utilises potentiometers to output 2D axis data.

In addition to the thumbstick the Nunchuk also has two additional buttons labeled as C and Z and an additional three-axis accelerometer sensor. A large volume of Wii games need the Nunchuk controller to provide navigational input, however this is likely due to requirements resulting from cross platform game development.

Other current attachments include a classic controller which provides the Wii system with facility for backwards compatibility.

3.2 Gesture Recognition Software:

Many games have demonstrated the ability to utilise the Wiimote to issue commands through pre-defined motion patterns. Good examples of this are the use of a fishing rod and the slashing of a sword within Nintendo’s latest Zelda game.

Investigation has identified software solutions which use Context Learning and Fuzzy Pattern matching to provide facility for run time applications to robustly determine the motions which a user is performing from a set of developer-provided examples.

Facility to purpose train an individual device is also suggested, which in turn would provide capability for making potential design solutions truly adaptable.

4. Current Investigations:

Given the capabilities of the Wiimote for use as an assistive device, applications for the following user groups are being developed:

• Visually Impaired
• Physically Impaired
• Deaf
• Stroke rehabilitation

To facilitate the development of all of the concept solutions an advanced Windows Driver for the Wiimote is currently in development and will replace the GlovePIE system upon completion. Current areas of investigation include:

4.1 Wiimote as a Cane within 3D Environments:

The Wiimote’s ability to describe a body within 3D space means that it can be used like a cane, and therefore could provide an interface to 3D environments for the visually impaired. By acting as a cane within 3D space the virtual counterpart uses distance measurements obtained by ray casting to control feedback in the form of vibration and auditory signals.

To explore this concept an additional test environment has been created and accompanying scripts defined. In addition a visually impaired user base has been set up in line with the current user-sensitive design methodology, to provide both design input and evaluation.

Existing games based software solutions have also been evaluated to aid in the development of a suitable navigation system for such interaction.

Testing has highlighted the need for initial orientation and calibration to be facilitated, however performance results are good.

The guide dog analogy has also been explored. This has greater potential for successful navigation, and it is planned to develop the “guide dog” to give verbal feedback and advice. The aim to create accessible Serious Games for the blind is feasible.

4.2 Wiimote as a surface free interface:

Investigation into the development of the surface free interface is ongoing, with focus on developing the system to use the reflected infrared light as an input source (figure 5).

Figure 5: Development of Reflective System

Initial results have proved encouraging however work needs to be undertaken to improve resolution and functionality within well lit surroundings.

The capability of using the Wiimote as a surface free interface has suggested numerous applications. With respect to assistive technology, two possible implementations are being developed:

• Stroke rehabilitation
• Sign Language Translator

Currently a Wiimote add-on, Finger-Mice, has already been prototyped (see figure 6) as a means of aiding with both physical and software development. Primarily consisting of two movable infrared light sources, when used in conjunction with Wii Connect, finger-mice enable a user to operate specially developed multi point software via movement of the index fingers.

Fig 6.0: Finger Mice Prototypes

An additional outcome at this stage is a very simple and cheap head tracking device. When tethered to a fixed position such as a pair of glasses Finger-Mice act as a movable sensor bar and thus by result can be used to provide facility for a basic head tracking unit.

4.2.1 Stroke rehabilitation:

For stroke rehabilitation, it is suggested that a user is tasked with drawing objects within free space. Facility to draw in this manner results in the capability for a full range of upper body exercises to be developed. For example, to create a 3D object such as a tetrahedron, four points at varying depths must be achieved. This naturally results in the use of a full range of arm movements.

In addition gesture recognition could also be used by physiotherapists to define a bespoke suite of motions as exercises, thus enabling a tailor-made course for patients.

4.2.2 Sign Language Translator:

Using the surface free application in conjunction with gesture recognition, a software solution could be trained by a user to recognise a suite of hand movements. Repetition of defined motions by the user could be used to trigger corresponding auditory output in the form of pre recorded speech.

A practical implementation of this is that of a sign language translator. An issue here however is the removal of some of the motion variants used for gesture recognition due to the current systems implementation. This could be remedied by expanding the system to use an additional Wiimote. Investigation into the feasibility of this concept has commenced and is ongoing.

4.2.3 Gesture recognition for the physically disabled:

The idea that the Wiimote could be trained to cater for an individual user’s motor skill ability and even facilitate bespoke methods of input has suggested application use with users suffering from severe physical disability.

Comparisons have been drawn with the EU Equal sponsored study to design assistive/adaptive devices for users with severe physical disabilities and further investigation is ongoing. Initial work to this end includes development of a potential design solution for an assistive switch, controller extension (the Wii Switch).

Functionality of this extension would include ability to perform not only as a contemporary switch but also provide facility to also behave as a mouse and joystick.

4.2.4 Tremor compensation for the elderly and physically disabled:

Investigations using force-feedback technology to assist motion-impaired computer users (Hwang et al, 2001) indicates that implementation of force-feedback as non-directional viscous damping has a beneficial effect for some users, improving time taken to reach a designated target by up to 50%.

It is suggested that accelerometer data provided by a mouse equivalent extension (4.2.3, Switch) be utilized for the real-time evaluation of user performance. By result, required dampening can then be applied to the raw input stream for more effective cursor control.

Use of gesture recognition could also be used to determine data outliers, thus increasing both the robustness and reliability of the system.

5. The Wiimote as an Assistive Device – Summary and Conclusions:

The important features of the Wiimote which make it suitable for use as an assistive device are:

• The Wiimote can be used to provide data needed to describe the 6 degrees of freedom needed to orientate a body within 3D space.
• The Wiimote provides facility to be used as a 2D pointing device and can easily facilitate 2D mouse-like interaction.
• The Wiimote has facility to act as a wireless bridge between developed devices via use of its expansion port.
• The Wiimote has facility to provide auditory, visual and haptic feedback.
• It has been conceived that by utilising software solutions a device could be trained to cater for an individual user’s motor skill ability and even facilitate bespoke methods of input.

Further development of the Wii Connect windows driver is now at its first iteration. New concept solutions using the Wiimote as an assistive device have been explored and developed. These include:

• The Wiimote as a cane and as a virtual guide dog to enable interaction by users with visual impairment.
• The Wiimote as a surface free interface – enabling development for stroke rehabilitation applications, and as a sign language translator.
• Gesture recognition, to enable interaction by users with severe physical disabilities.
• PDS developed for new extension, the Wii Switch
• Dampening for cursor input.
• Simple and cheap head tracking system.

These investigations have demonstrated the enormous potential of using the Wiimote as an assistive device. It is mainstream, easily available and relatively cheap, all of which make it ideal for this purpose.

References:

Battersby, S. (2007) Serious Games and the Wii – a technical report, BCS SGAI AI-2007 Serious Games Workshop, Cambridge, 10-12 December 2007

Hwang F, Langdon P, Keates S, Clarkson PJ (2001) Haptic assistance to improve computer access for motion-impaired users, Proceedings of Eurohaptics 2001, Birmingham, 176-178,
http://rehab-www.eng.cam.ac.uk/papers/eurohaptics2001/, accessed 1.2.08

Peek, B. (2008) Managed library for Nintendo’s Wiimote: A library for using a Nintendo Wii Remote (Wiimote) from .NET, CodePlex, http://www.codeplex.com/WiimoteLib, accessed 1.2.08