Loading post…
DatabricksTanishq Maheshwari, Matt JonesAnnouncement
Run, debug, and scale Databricks workloads from your local IDE
Summary
Databricks introduces updated IDE and CLI integrations that connect VS Code, Cursor, or a terminal directly to Databricks compute, workspace, and Unity Catalog. Existing Databricks Connect supported local Spark development, but remote non-Spark workloads and dependency synchronization remained pain points. The new SSH tunnel supports interactive execution and debugging of Python and SQL workloads on Serverless, AI Runtime, and dedicated clusters, while keeping files and dependencies aligned with Databricks Runtime and the workspace. CLI flags can select serverless, GPU-backed AI Runtime, or a dedicated cluster, and can start the tunnel in VS Code or Cursor; additional options configure base environments and usage policies, while the IDE can browse Unity Catalog assets.
Context
Users may prefer local IDEs and CLI tools, including coding agents, for complex, large-scale data pipelines and machine learning models. Before these updates, local Spark development was supported, but running non-Spark workloads remotely and keeping dependencies synchronized with Databricks Runtime remained common pain points.
Approach / What changed
The updates add an SSH tunnel connecting VS Code, Cursor, or the CLI to Serverless, AI Runtime, and dedicated Databricks clusters. CLI and IDE options support interactive Python and SQL execution, debugging, base environments with pre-installed dependencies, usage policies, and Unity Catalog browsing.
Takeaways
The SSH tunnel supports interactive execution and debugging of workspace files and notebooks from VS Code, Cursor, or the CLI.
The databricks ssh connect command can target serverless compute, AI Runtime with GPU_1xA10 or GPU_8xH100, or a dedicated cluster.
The --base-environment and --usage-policy-id flags support dependency setup and usage-cost tracking, respectively.
Source and freshness
- Source
- databricks.com
- Published
- Added to Yomu
- Last updated
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.