FaceTrackNoIR and FreeTrack: FOSS head-tracking (like TrackIR) - use any webcam to link to freelook /movement in games

FaceTrackNoIR (download) enables a user to use any webcam,by moving their head very slightly, to control the free-look/xyz movement of any game, through joystick emulation via PPjoy for Windows or directly when the game developer has included support.
Very popular with space/flight simulation community, this technology can be used with FPSs/3rd person RPGs as well. FaceTrackNoir uses the protocol developed for FreeTrack, among others, including FlightGear's own.

Webcams known to work well with FaceTrackNoIR.

Normally this type of control through head movement was very expensive, and required specialised IR cameras/lights/markers to acquire raw data which was processed through closed source software and sent to games via protocols like TrackIR, being aimed at the high end simulation users. The FreeTrack opensource project used cheap but specialised equipment, wrote their own processing software and sent the data to games via reverse engineered TrackIR protocol/FreeTrack protocol. FaceTrackNoIR takes raw data from any webcam, processes it, and sends to games via FreeTrack or other protocols or through emulating a joystick.

Orbiter with FaceTrackNoIR.

Even used in a non-foss pinball game.

Devs note: the quick head movements accentuate depth perception a lot, and shows off the 3d art.

(The last two videos were non-FOSS games, but only a few youtube vids as it is new software, and demonstrates different uses)

This is a new project, started in May 2010, built from FreeTrack source code and using faceAPI to help with imageprocessing. Unfortunately it is not available on Linux as faceAPI is windows only at the moment, and would require substituting with one of the opensource face tracking techs. A simple face-mouse cursor example here called eViacam, 8.9mb.Apparently working well as shown being used in GIMP here.


There is no mouse or keyboard emulation for games that don't support a tracking protocol yet, but it should not be hard to port from FreeTrack.It should be possible to use a mouse/keyboard emulator that accepts joystick inputs if the game does not support joystick look.

With the advent of Kinect and PC hacks for it, movement tracking based control will become more sought after. Given the additional (upto 6) degrees of freedom easily input by the user, perhaps some quite creative uses can be thought of as webcams are very commonplace.

Games with direct support
Note: anything stated as working with FreeTrack will also work.

Any of the free games running under the Freespace 2 open engine.
Flight gear
Combat Simulator Project, when released
Orbiter

Games that (should) benefit through emulation
(If any game benefits particularly well, be sure to post in comments, if any game would benefit if better joystick support was added to certain control elements, be sure to let devs know). Free look can be catered for by assigning axis displacement to angles, very simply.

 Any Spacesim

Pioneer
Vega Strike
Hardwar, when released

Any flightsim

Palmino
A lot of combat sims mentioned in this blog post

3rd person 3d RTS

Political space war
Far colony
A lot of the 3d Spring RTS games

Quite a lot of 3rd person RPG/RTS games have automated and often wonky cameras that would benefit from being made freelook, mouse controlled views could be utilised by linking to tracking and emulated in engines like Open Morrowind.

---

FreeTrack

FreeTrack is similar to FaceTrackNoIR, but requires markers,usually very cheap IR leds attached to headphones, which must be tracked. Mouse/joystick/keyboard emulation for games that don't support tracking protocols available. Linux is not supported although from the forums one or two people might be working on linux projects.

What it involves, without buying TrackIR/mocap specific equipment

• Webcam

• Photo negatives from an old reel to only let IR light through (if your cam doesn't have an IR mode)

• A hat, headphones or similar to stick markers to

• Cheap IR leds to use as markers, probably work less acurately with IR reflective tape/stickers..the idea is it stands out from the rest of the picture, and IR is an easy way to do that

• Costs less than 10 euro according to their website, more info here.

Used to drive a mouse cursor.

All videos courtesy of various Youtube people.

---

Gamers can download and try it immediately, game developers can add direct FreeTrack support, or add joystick/mouse support in a way that better accepts emulation. If you have an interest in interface tech, source code for both are available.