DatabricksAngel De LeonProblem & solution
Security reviews at Databricks relied on automation but still consumed expert time on predictable work, leaving less room for genuinely novel, high-risk architectures. The author built an agent-based layer on Databricks using Unity Catalog for governed tables containing standards, request data, and decisions; Databricks-hosted foundation models (Claude Haiku, Sonnet, Opus) for classification and reasoning; and Lakeflow Jobs for orchestrating notebook-based workflows on serverless compute. A conversational intake app built with Databricks Apps turns plain-language descriptions into structured requests and provides preliminary risk indications before a formal request is created. Seven focused agents each have a bounded responsibility—intake, risk assessment, requirements, specialized reviews, validation, workflow, and learning—ensuring behavior remains inspectable and testable. Routine requests that once waited days in the queue can now be completed in minutes, and security engineers spend more time on designs requiring genuine expertise.
A multi-agent system on Databricks using Unity Catalog for governed tables of standards, request data, and decisions; Databricks-hosted foundation models (Claude Haiku, Sonnet, Opus) for classification and reasoning; Lakeflow Jobs for orchestrating seven focused notebook-based agents on serverless compute; and Databricks Apps for the conversational intake experience and an executive dashboard.
Separate process from judgment: automate repeatable steps only when criteria are explicit, evidence is required, and uncertainty is escalated to a human.
Seven focused agents with bounded responsibilities (intake, risk assessment, requirements, specialized reviews, validation, workflow, learning) keep the system inspectable and testable.
Building on a unified platform (Unity Catalog, Databricks models, Lakeflow Jobs, Databricks Apps) allowed a working system in under two hours instead of weeks.
The system used Unity Catalog for governed tables, Databricks-hosted foundation models (Claude Haiku, Sonnet, Opus) for classification and reasoning, Lakeflow Jobs for orchestration on serverless compute, and Databricks Apps for the intake experience and dashboard.
When information is missing or contradictory, the system asks for clarification or routes the request to a human reviewer. The system does not infer its way to approval; it escalates when uncertain.
The executive dashboard tracks request volume, risk distribution, automated completion, human escalation, cycle time, and estimated time saved by reviewers, all derived from operational records in Unity Catalog.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
Boards of directors turn ownership into a governance control loop: shareholders elect directors, directors set strategy and guardrails, committees verify independently, and management executes inside those limits.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
