Toward More Controllable AI Video Editing: An Early Research Exploration at Netflix

Summary

Generative video editing tools often regenerate entire video clips, which inadvertently modifies untouched scene details or violates physical continuity during object deletion. To provide artists with precise control, two research prototypes were developed: Vera and VOID. Vera uses a layered video diffusion framework with a Mixture-of-Transformers architecture to jointly generate separate edit and alpha matte layers, compositing them with original footage while leaving unchanged pixels untouched. VOID addresses object deletion by conditioning on interaction-aware quadmasks and using a second-pass appearance refiner to reconstruct scenes with plausible physical continuity. In a user study evaluated across 75 real-world scenarios, reviewers selected VOID 64.8% of the time over six baselines.

Takeaways

1. 1

   Vera relies on a Mixture-of-Transformers design with three separate DiTs that maintain independent QKV and FFN weights while sharing joint self-attention across output tokens.

2. 2

   VOID incorporates quadmask conditioning and a second-pass appearance refiner to remove objects while maintaining physically plausible downstream scene dynamics.

3. 3

   In a user study against six baseline models across 75 real-world scenarios, VOID was selected as the most realistic output 64.8% of the time.