The race to create a reliable AI video generator. A group of researchers, including interns at Netflix, have developed of Go-with-the-Flow, a new model that aims to provide an easy way to control motion patterns in video diffusion models.
Appearing hot on the heels of Tencent's HunyuanVideo, the model lets users decide how objects in a scene will move simply by cutting and dragging. And it allows camera control and even motion transfer from one video to another.
Go-with-the-Flow was developed by researchers from Netflix, Netflix Eyeline Studios, Stony Brook University, University of Maryland and Stanford University. It allows users to animate an object simply by selecting an object and defining a trajectory by dragging like you might do with a mask in video editing software.
First, a crude animation is created as a cut out, then warped noise is added in a diffusion script (this part requires GPU) to turn this into smooth(ish) animation. Like most AI video, the results look janky for now, particularly with people and animals since limbs sometimes get mixed up and appear to trip over each other, but it's still a notable advance.
The researchers say Go-with-the-Flow simply fine-tunes a base model, requiring no changes to the original pipeline or architecture, except the use of warped noise instead of pure IID Gaussian noise. The model also allows motion transfer, using the original video as the motion signal and a new target prompt to guide the output. And it can create turntable animations using a 3D-rendered turntable camera motion as a guide.
The code is available at GitHub and Hugging Face. You can see a tutorial on the AI video model below.
Netflix, whose occasional use of AI art has already earned criticism, was keen to point out that Netflix has not created an AI image generator of its own. It says this project was part of "continued research to develop ways for artists to have creative control and leverage video diffusion models. This technology is currently in the research phase and is not being applied to any production use at this time."
The streaming platform has previously announced that it's working on using generative AI to make games. Meanwhile, Lionsgate, the company behind John Wick is developing a bespoke AI video model with Runway and Showrunner AI still thinks users will want to use AI to make their own shows.
28/1: Updated to clarify that Netflix has not created an AI video generator for production use.
