Camera tracking: the secrets of motion capture

In its infancy a decade ago, today motion capture (now more commonly described as performance capture) is a common and regularly used technique in the world of TV, film, VFX and games.

This impressive technology has advanced over the last 10 years to produce state of the art systems, enabling the creation of photoreal, computer generated characters that can be brought to life with realistic emotions and movements - all without the need of makeup, prosthetics, or traditional key animation.

While we celebrate and admire modern movie characters such as Gollum from The Lord of the Rings Trilogy and the cast of Avatar, none of these would have been possible without taking a journey through a pipeline of several stages, each as vitally important as the next, to enable a high quality end result.

01. Pre-production

The performance capture pipeline features many similarities to those carried out on an everyday film set.

Vital pre-production preparations must be carried out to ensure a shoot can run as smoothly as possible and allow an actor - who is covered with special markers - to give their best performance without distractions or disruptions.

More recently, the implementation of real-time visualization systems has enabled a digital mirror for the actor and the director to see an unpolished result of a virtual avatar being driven by the actor's movements.

However, the main importance still lies with the motion capture system and its ability to capture the highest quality data possible.

02. Post-processing

While today's systems are constantly improving the quality of the data captured on stage, like most elements of VFX, post processing is still very much required.

Data reconstruction and tracking is the first stop in the performance capture pipeline, making it an important step to avoid time-disturbing repercussions further along.

03. Reconstruction

Optical data captured from a shoot is stored as multiple raw 2D data and requires a 'reconstruction' process combined with a calibration of the system taken on the day of shooting, to convert it into continuous 3D trajectories.

At this stage the scene will contain a cloud of 3D data. However, there will be no identification as to what each piece of data is. The 'Labelling' process involves labelling these trajectories using a generic predefined character setup, to give each marker an identity associated with the underlying skeleton.

These labels define what role each marker plays in the skeleton of the human body, translating the captured movements onto a primitive rig.

The role of data tracker is to ensure these post-process tasks are carried out correctly. Within the software used, certain scripts or tools can be applied to automate the workflow. However most data gaps, jitters and other inaccuracies need to be manually' 'cleaned' or fixed before the data is ready to be exported to the next part of the pipeline.

04. Holes in the data

Inaccuracies in data are often caused by occlusion of the physical marker. A motion capture system has a 360 degree view of a markered performer. Each camera tracks a 2D view, and is set up and aimed to a position where a marker is being tracked by a minimum of three cameras at any time, which enables the reconstruction of 3D data.

This means that the more cameras with vision, the better quality the data you'll capture. However when an actor moves, does a particular stunt, or interacts with other actors, the maker's visibility can be lost to some or all cameras, preventing the software from getting enough information to allow the creation of the 3D marker.

05. Filling the holes

In the reconstruction process, when there isn't enough data, the software will produce a gap where no data exists. When there is enough data to produce a marker, but not enough for a reliable reconstruction, this will result in data with sporadic keyframes.

Swaps in data are often caused by the use of an automated labelling tool, or when the markers come into close proximity of each other, such as inner knees touching or hands touching together, resulting in the sofware misidentifying them.

The end result of a 'tracked' file will be accurately labelled trajectories with minimum noise and no gaps. This will allow a solve of a skeleton, based on the positions and movements of the makers, to be created.

A solve is reliant on the marker's accuracy. A swap or noise can affect the movement of markers, and therefore affect the authenticity of the performance when it is later retargeted onto a CGI character.

06. No shortcuts

Motion capture data tracking requires an eye for detail, patience for reaching perfection and fast turnaround, but in turn allows you to understand the movements and limitations of the human body.

Performance capture allows you to take those limitations and apply them to characters, which traditionally could not be achieved without this technology.

Like any pipeline, results are reliant on consistent quality. Badly tracked data will most likely result in a lower quality performance, while keen attention to detail and a thorough approach is vital to attain the high quality and much-loved performances given to computer generated characters seen in movies like The Hobbit and Avatar.

Free talk! The Imaginarium's Andy Morley on real-time performance capture

This Monday, 14 October, Andy Morley and the Imaginarium team will be talking through all aspects of the process of performance capture, from the actual first script stage to the final delivery of performance capture data.

The event takes place at ZED, HP's 'Pop-Up Studio for the Creative Community' in London, Soho. Details of the event and how to register are here (hurry - tickets are free, but limited!).