Digital cameras make it easy for filmmakers to record hours of video for each minute that appears on screen.
But in between the set and the movie theater, film editors must sift through all that raw footage to put together the first rough cut of what will become the final production.
Now, a team of computer scientists at Stanford have built an editing tool that can help film editors make a rough cut in minutes rather than hours, so they can spend more time on the creative work of designing narratives.
The team, spearheaded by computer science graduate student Mackenzie Leake, has described its autoediting system in a recent scientific paper and will demonstrate the technology at SIGGRAPH, the annual scientific conference on computer graphics.
Team leader Maneesh Agrawala, a professor of computer science at Stanford, said the current software is designed to help human film editors make rough cuts based on scenes involving two or more characters engaged in dialogue.
Today, human film editors usually have to watch and rewatch the raw film multiple times to find the most appropriate camera framing and performance for each line of dialogue. Eliminating the drudgery of these steps is one of the big time savers of the software, the researchers say.
The human film editor working with the tool would feed the software raw video of scenes along with a text script of the dialogue for the scene. The software automatically processes the raw footage into a series of short clips that it labels in different ways using a variety of image and language processing technologies.
For instance, face-tracking software enables the program to identify who is speaking. The same technology also allows the software to identify whether the scene is a close-up, a medium shot or a wide-angle view. The software also uses language processing techniques to attach positive or negative emotional intensity scores to each line of dialogue.
The software then uses these labels to string together the clips into a rough cut that conforms to the common idioms of film editing. For instance, one common idiom might be for a film editor to start a rough cut with a wide shot to establish whether the scene occurs in a bedroom or a board room, then perhaps move to medium shots or close-ups as the narrative continues.
The challenge was turning these commonly understood idioms, which are all matters of human judgment, into a series of mathematical constraints that software could implement. Given a set of such idioms, the software considers all possible sequences of clips and uses artificial intelligence techniques to find a path through the clips that lies within the constraints imposed by the idioms. The current software offers human film editors a palette of 13 idioms they can use to autoedit the rough cut. For instance, start wide, then make sure to show the face of each speaker while they talk, and use close-ups for particularly emotional lines of dialogue.
Agrawala said it typically takes a professional editor about 90 to 180 minutes to build a rough cut for 1–2-minute scenes, but with this tool, an editor could make many rough cuts in a few minutes.
“It will allow editors and storytellers to focus on the important parts of the job – figuring out what sequence of cuts will tell the story the best,” he said.
The current version of the tool only works with dialogue scenes. The researchers hope to develop it to work with action sequences, how-to videos and other visual story types. But first they have to figure out a way to “teach” the program the film editing idioms for constructing such narrative.
Eventually, Agrawala envisions using this editing tool during the actual filming of a scene. For instance, film editors could quickly process the raw footage and then whip up a few rough cuts to see whether extra takes are necessary. Filmmakers typically capture lots of takes to ensure that they have good coverage of the scene. If the rough cut looks good, it might reduce the need to capture such extra takes.
“It’s still going to require a professional editor to produce a professional-quality edit,” said Agrawala. “But it should make their job much easier and allow them to be more creative and explore the design space of possible edits much more easily.”