Gesture Recognition 

In this project, we developed a gesture recognition system from the ground up for use as a new interface modality.  The current system utilizes real-time camera feed, a max hand size for the user in the x and y dimensions, processes the image through several steps, and then returns the amount of movement in pixels, the orientation of the hand, and the number of fingers that are being held up.  In the interface, this information is used to manipulate a 3D skull.  The current system is implemented in C++ and utilizes the IPL (Image Processing Library) and OpenCV (Open-source Computer Vision) libraries from Intel.  It can process 640x480 video feed or PGM images at 30 fps on a 1.33 Ghz Athlon MP processor with a standard firewire webcam.

Since the system was designed to run in real-time, efficient image processing algorithms were utilized.  Building and updating a 3D model of the hand with stereo vision was deemed to be too slow for this endeavor on practical computer systems.  For this reason, a series of steps are applied to images in order to extract the desired information (namely movement between two frames/images, orientation of the hand, and the number of fingers being held up).  The position of the hand and the number of fingers determines the manipulation of a 3D skull as shown below.

Original placement of skull and hand (left)
Gesture to action mapping for manipulation (right)

Hand moved to right for Yaw Right (left)
Hand showing four fingers for Zoom In (right)

If you would like further information about the processing stages of the gesture recognition system, see the following links:

The following downloads are made available to explain the process of the system in more depth. Please follow any of the links below:

• Video of gesture interface system v.2.0 in use [AVI, 14.4MB].
  - Requires the TechSmith Screen Capture Codec (TSCC)

• PowerPoint presentation of v.1.0 of the system [PDF, 1.6MB].

• PowerPoint presentation of v.2.0 of the system [PDF, 1.2MB].

Relevant Publications

New, J., Hasanbelliu, E., and Aguilar, M. (2003). "Facilitating User Interaction with Complex Systems via Hand Gesture Recognition." To appear in Proceedings of the 2003 Southeastern ACM Conference, Savannah, GA. [PDF]

New, J. (2002). "A Method for Hand Gesture Recognition."  In Proceedings of the ACM Mid-Southeast Chapter Fall Conference, Gatlinburg, TN. [PDF]

Kjeldsen, Frederik. (1997). "Visual Interpretation of Hand Gestures as a Practical Interface Modality"  PhD thesis, Columbia University, New York, NY. System based loosely on the findings presented in this thesis. [link to PDFs]

Related Projects

The gesture recognition system has future plans to be used as an interface modality for skull and/or slice manipulation in the context of Med-LIFE as well as augmented reality visualization.  

Acknowledgements

This work was supported by a Faculty Research Grant awarded to the director of the laboratory by the faculty research committee and Jacksonville State University.  Opinions, interpretations, and conclusions are those of the authors and not necessarily endorsed by the committee or Jacksonville State University.