Loading…
Toward More Controllable AI Video Editing: An Early Research Exploration at Netflix
NetflixNetflix Technology Blog
Summary
Generative video editing tools often regenerate entire video clips, which inadvertently modifies untouched scene details or violates physical continuity during object deletion. To provide artists with precise control, two research prototypes were developed: Vera and VOID. Vera uses a layered video diffusion framework with a Mixture-of-Transformers architecture to jointly generate separate edit and alpha matte layers, compositing them with original footage while leaving unchanged pixels untouched. VOID addresses object deletion by conditioning on interaction-aware quadmasks and using a second-pass appearance refiner to reconstruct scenes with plausible physical continuity. In a user study evaluated across 75 real-world scenarios, reviewers selected VOID 64.8% of the time over six baselines.
Context
Generative video editing models frequently regenerate every pixel, inadvertently altering preserved identities, performances, and scene details, or ignoring physical continuity when erasing objects.
Approach / What changed
Netflix introduced two models: Vera, a layered video diffusion model using a Mixture-of-Transformers architecture with three separate DiTs for edit, alpha, and composite outputs; and VOID, a video inpainting model fine-tuned on the CogVideoX-Fun-V1.5-5b-InP backbone using interaction-aware quadmask conditioning and a second-pass video appearance refiner.
Takeaways
- Vera relies on a Mixture-of-Transformers design with three separate DiTs that maintain independent QKV and FFN weights while sharing joint self-attention across output tokens.
- VOID incorporates quadmask conditioning and a second-pass appearance refiner to remove objects while maintaining physically plausible downstream scene dynamics.
- In a user study against six baseline models across 75 real-world scenarios, VOID was selected as the most realistic output 64.8% of the time.
Related reading
Netflix ·
GenPage: Towards End-to-End Generative Homepage Construction at Netflix
Netflix traditionally constructs its structured two-dimensional homepage through a complex, multi-stage recommender pipeline that separates candidate generation and ranking across rows and entities. To simplify this architecture and optimize directly for whole-page user satisfaction, Netflix developed GenPage, an end-to-end generative transformer model that autoregressively builds the entire homepage from raw tokenized context. The system relies on a domain-specific tokenizer to compress engagement history, context injection for cold start, hybrid row decoding to minimize decoding steps, and reinforcement learning post-training. In online A/B testing against the production baseline, GenPage delivered statistically significant gains in core user engagement metrics while cutting end-to-end serving latency by 20 percent. Offline evaluations further showed that enriching context prompts improved recommendation quality more effectively than increasing model capacity in the current operating regime.
Netflix Technology BlogDropbox ·
Beyond code generation: rethinking engineering productivity in the age of AI agents
Dropbox shares how widespread AI code generation shifts software development bottlenecks downstream into code review, CI infrastructure, and validation pipelines. To adapt, they built Nova, an internal coding agent platform that safely automates scoped tasks such as migrations and flaky test remediation. They also evolved their developer productivity framework to measure end-to-end customer impact and code quality rather than simple pull request throughput.
Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki OkumuraNetflix ·
In-House LLM Serving at Netflix
Netflix established an in-house serving platform to run large language model inference directly inside existing production environments alongside traditional machine learning models. Built upon NVIDIA Triton Inference Server and vLLM, the unified architecture handles member-scale routing, candidate generation, feature fetching, and model execution over gRPC and OpenAI-compatible HTTP endpoints. The team shifted their primary engine from TensorRT-LLM to vLLM to support custom architectures, simpler debugging, and non-trivial constraint logic. Addressing production obstacles required patching Triton's frontend for guided decoding, pinning dependent library versions, rewriting logits processors in C++ for vLLM V1, and handling state machine resets during engine preemptions. The resulting platform unifies deployment flows while preserving operational stability across evolving model schemas.
Netflix Technology BlogDropbox ·
How our universal content processing platform Riviera evolved for AI and beyond
Dropbox developed Riviera, a centralized content processing platform that evolved from an internal file preview generator into a shared infrastructure supporting over 300 file formats. By decomposing complex conversion tasks into modular, reusable transformations, the system now powers search indexing, media transcoding, and AI context extraction across Dropbox products.
Ilya Yakovlev,Andrew Cheung,Binoy Dash