Run, debug, and scale Databricks workloads from your local IDE

Summary

Databricks introduces updated IDE and CLI integrations that connect VS Code, Cursor, or a terminal directly to Databricks compute, workspace, and Unity Catalog. Existing Databricks Connect supported local Spark development, but remote non-Spark workloads and dependency synchronization remained pain points. The new SSH tunnel supports interactive execution and debugging of Python and SQL workloads on Serverless, AI Runtime, and dedicated clusters, while keeping files and dependencies aligned with Databricks Runtime and the workspace. CLI flags can select serverless, GPU-backed AI Runtime, or a dedicated cluster, and can start the tunnel in VS Code or Cursor; additional options configure base environments and usage policies, while the IDE can browse Unity Catalog assets.

Context

Users may prefer local IDEs and CLI tools, including coding agents, for complex, large-scale data pipelines and machine learning models. Before these updates, local Spark development was supported, but running non-Spark workloads remotely and keeping dependencies synchronized with Databricks Runtime remained common pain points.

Approach / What changed

The updates add an SSH tunnel connecting VS Code, Cursor, or the CLI to Serverless, AI Runtime, and dedicated Databricks clusters. CLI and IDE options support interactive Python and SQL execution, debugging, base environments with pre-installed dependencies, usage policies, and Unity Catalog browsing.

Takeaways

• The SSH tunnel supports interactive execution and debugging of workspace files and notebooks from VS Code, Cursor, or the CLI.
• The databricks ssh connect command can target serverless compute, AI Runtime with GPU_1xA10 or GPU_8xH100, or a dedicated cluster.
• The --base-environment and --usage-policy-id flags support dependency setup and usage-cost tracking, respectively.

Related reading