BY CAMERON NEILSON
FROM ESC ENTERTAINMENT

The Burly Brawl was originally conceived by Larry & Andy Wachowski, but it would take a huge effort to create the finished piece from a rough concept. The sequence was first roughed out as a series of storyboards that illustrated and blocked out the action. With these conceptual drawings as a starting point, the process began to bring the Brawl to reality.

ESC Entertainment (who did the Visual Effects for the Burly Brawl sequence) realized early on that the task of creating multiple Smiths would require many shots to be created entirely in the computer using synthetic humans and virtual backgrounds. While much of the sequence would be created digitally, there were still many live-action shots, so matching the look of the two was imperative. To accomplish this, ESC worked closely with Production Designer Owen Paterson, Director of Photography Bill Pope, and Visual Effects Supervisor John Gaeta (among others) to ensure that the virtual environment would match the live set as closely as possible.

A lidar scan of the entire set was created as a reference model for recreating the geometry in the computer. Photographs of the lit set were taken both as reference and for use as virtual background textures. A technique known as Image-Based Lighting (IBL) was used where chrome balls are photographed in order to capture and replicate the on-set lighting.

With both the practical and virtual sets complete, work could turn toward the actual choreography of the Burly Brawl. The first step was for fight choreographer Yuen Wo Ping, the Wachowski Brothers and John Gaeta to plan out the action of Neo fighting the many Smith copies. The complex choreography was then input into the computer using a motion capture setup. Since the scene involved dozens of people, it had to be broken down and captured a few people at a time, and then rebuilt afterwards. The mocap data was then mapped to the corresponding Neo and Smith models and the animation imported into Maya.

Working from this initial mocap animation, John Gaeta and the Wachowskis then worked with ESC to refine the action. Using low resolution models and virtual background textures, the animation team cleaned up and modified the imported animation by hand. At the same time, this pre-vis setup was used to adjust the camera moves to best frame the actors in each shot.

Now that the animation was complete, the process of recreating Smith and Neo began. Each actor’s likeness, facial expressions, skin tone, and clothing all had to be recreated and simulated in the computer in photo-realistic quality. A process called Universal Capture (UCAP) was developed to capture and recreate each actor’s head and facial characteristics. Five high-definition cameras and a small army of computers were set up to record the various facial expressions of each actor. The amount of data collected was so immense that each day’s work was backed up every night by a robot tape system. Combined with a cyberscan cast of each actor’s head, the Universal Capture information allowed ESC to create an exact moving replica of each actor’s head.

To simulate the appearance of cloth and its interaction with light, the wardrobes for both Neo and Smith were scanned using a Bidirectional Reflectometer, which captures the light reflectance (Bidirectional Reflectance Distribution Function or BRDF) values for all kinds of cloth. Using this extremely powerful machine allowed ESC to scan the various types of cloth involved in the actors’ wardrobes, which could then be recreated and simulated in the computer.

Combining the Universal Capture data, facial casts/scans, BRDF information, and light characteristics through IBL, a virtual human is created. Side by side tabletop comparisons of the real Smith and the virtual one yielded practically indistinguishable results. The facial performances from the UCAP sessions were then assigned to the heads of Neo and the many Smiths; and for the final touch animation was added to the actor’s virtual clothing to simulate realistic movement.

All pieces complete, the final synthetic human animation is composited with the finished virtual background to create the final shot. Before rendering out a given frame, each shot exists in the computer as lines of code, millions of assets compressed into one final shot.