How Databricks is turning video into searchable, actionable intelligence

Summary

Databricks presents video analysis as a data engineering problem for organizations with terabytes of footage that is difficult and expensive to review manually. An app accepts a video and a natural-language prompt, then triggers a Lakeflow job on Serverless GPU Compute to run Meta’s SAM3 segmentation model frame by frame and retain matching moments. Those clips preserve original timestamps and are sent through the Databricks Foundation Model API for summaries that can be written to tables or passed into downstream workflows. In one example, 26 minutes of traffic footage became one minute and 55 seconds of relevant video. The model-agnostic pipeline uses MLflow signatures to support interchangeable or custom models, while event-driven execution and independent serverless GPUs allow concurrent processing without cluster management or idle GPU costs.

Context

Large volumes of unstructured video are generated for infrastructure inspection, public safety, traffic analysis, and other uses, but manually reviewing the footage is time-consuming and expensive. Scaling computer vision and vision language model inference also creates logistical challenges involving unstructured data, GPU capacity, and pipeline orchestration.

Approach / What changed

A Databricks app accepts uploaded or stored video and a natural-language search prompt, then triggers a Lakeflow job on Serverless GPU Compute. Meta’s SAM3 segments matching objects frame by frame, produces timestamp-preserving truncated clips, and sends them to the Databricks Foundation Model API for text summaries. MLflow signatures keep the pipeline model agnostic, while event-driven jobs and independent serverless GPUs support concurrent processing.

Takeaways

• A 26-minute traffic-camera video was reduced to one minute and 55 seconds of relevant footage, with original timestamps preserved for returning to the source.
• The pipeline can use SAM3, YOLO, transformer-based vision models, or domain-specific models, and supports custom or fine-tuned models through MLflow model signatures.
• Videos arriving in a Databricks Volume can automatically trigger processing; concurrent jobs obtain independent serverless GPUs and release resources after completion.

Related reading