When Neo and Agent Smith walk into the courtyard, we see the real actors Keanu Reeves and Hugo Weaving. But once the fighting starts, everyone and everything on the screen is computer-generated–including the perspective of the camera itself, steering through arcs at speeds that would tear any physical camera apart. Perhaps the most impressive aspect of the Burly Brawl is that it doesn’t look virtual at all. The digital faces of Reeves and Weaving are flawless, the buildings look dreary and lived-in.

The standard way of creating a computer generated world is to build it from the inside out, building objects out of polygons and applying textures and lighting. For Reloaded, the special effects team took a radically different approach known as image-based rendering, loading as much of the real world as possible into the computer first, essentially building from the outside in. While the topography of the human face is difficult to simulate digitally, it turns out to be easy to map photogrammetrically, having few shadows and occlusions. To replace the faces of the stuntmen with that of Agent Smith, Gaeta and his team built a system for sampling the real at a higher resolution than had ever before been attempted, dubbing this process “universal capture”. Once a scene is captured, filmmakers can now fly the virtual camera through thousands of “takes” of the original performance, from any angle, zooming in and out, or launching into the sky. Virtual cinematography.

But the ability to create photorealistic virtual human beings, and to cut-and-paste them into any landscape, raises unsettling questions. Questions that troubled Gaeta so much that, several years ago, he wrote a letter alerting President Clinton to the fact that such technology could be used for purposes of mass deception. The letter was never answered. It turns out that DARPA (Defense Advanced Research Projects Agency) is very interested in using image-based rendering and lighting for use in immersive battle simulations. In 1999, the US Army launched the Institute for Creative Technologies at the University of Southern California.

The paradox is not lossed on Gaeta. “You have these paranoid films about the Matrix depicting how people are put in a mental prison by misusing this technology, and you have the military constructing something like the actual Matrix. Or maybe our technology will become the actual Matrix, and we have inadvertently spilled the vial of green shit out onto the planet.”