Loading…
Lakeflow
11 posts about Lakeflow. Every summary links to the original.
Databricks for Good and Virtue Foundation: Partnering to Connect Medical Volunteers to Critical Health Services in 72 Countries
Virtue Foundation’s VF Match platform connects medical professionals with volunteer opportunities across 72 low and low-middle income countries, but scaling its initial LLM proof of concept into production required a robust data pipeline. The Databricks-based Foundational Data Refresh combines Overture Maps and Bright Data, processes more than 25 million web pages with targeted GPT extraction steps, and uses Spark, Photon, and Lakeflow Jobs to orchestrate checkpointed, modular workloads. Splink resolves duplicate facilities and nonprofits across messy sources; Photon reduced the slowest entity-resolution partition from 30 minutes to approximately two minutes, a 15x improvement. A prototype VF Agent adds natural-language access through LangGraph, Databricks Model Serving, AI Search, and Genie, while the resulting system provides an up-to-date global view of healthcare infrastructure and volunteer needs.
Priyanka Mehta, Shaunak SenAutomate Data & KPI Monitoring with SQL Alerts
Databricks SQL Alerts is generally available as an automated alternative to manually checking dashboards and rerunning monitoring queries for business KPIs, pipeline health, and data quality. Teams define a SQL query, evaluation condition, schedule, and notification destinations, then receive alerts when results cross configured guardrails; standalone alerts can also run within Lakeflow Jobs. GA supports authoring in the SQL editor, notifications through email, Slack, PagerDuty, Microsoft Teams, or webhooks, and production management through Git, Declarative Automation Bundles, APIs, Terraform, and SDKs. The post’s revenue example compares yesterday’s result with a seven-day average and triggers when revenue_pct_change is below -5%, while the SQL Alert task can expose OK, TRIGGERED, or ERROR to downstream pipeline logic.
Srilekha DornadulaHow CFOs in consulting can recover margin with Databricks
Consulting firms are facing margin pressure despite growing deal pipelines, with 2024 EBITDA margins at 9.8%, utilization at 68.9%, and revenue growth at 4.6%. The post attributes this to fragmented finance workflows spanning Salesforce, SAP, Workday, Concur, and spreadsheets, plus inconsistent metrics and manual reconciliation. It proposes using Databricks to create a governed data layer with Lakeflow, Lakehouse, and Unity Catalog, then adding Genie for natural-language analysis and finance apps that trigger workflows and write approvals back in real time. A global consulting firm reportedly cut cash forecasting cycles by 3–5 days and reduced reporting-cycle FTE hours by 80%, while Databricks’ own close fell from 15 days to 8. The proposed rollout spans 10–12 weeks and begins with one tightly scoped workflow such as revenue close, unbilled WIP, T&E leakage, or practice-margin analysis.
Rajaram Suresh, Kateryna Savchyn, Pavithra Rao, Mark Hahn, Kim Hatton, Sharmey Shah, Lisa FoxAgentic data engineering with Genie Code and Lakeflow
Genie Code is presented as a natural-language assistant for developing, orchestrating, deploying, and debugging data pipelines and jobs. It uses pipeline and job context, including code, configuration, run results, Unity Catalog metadata, lineage, popularity, and code samples, to help engineers discover datasets and understand data flows. Engineers can describe pipelines or jobs, and Genie Code can generate Spark Declarative Pipelines with Bronze, Silver, and Gold layers, sources, transformations, data quality expectations, and outputs, then configure orchestration, schedules, dependencies, Auto Loader, AutoCDC flows, and Declarative Automation Bundles. It also analyzes failures and unexpected row-count or schema changes, proposes cross-file updates with reviewable diffs, and supports extensions through custom instructions, agent skills, and MCP servers. The stated result is faster development and guided debugging while workflows remain aligned with Unity Catalog governance, performance, and data quality standards; future plans include background failure response and cluster right-sizing.
Gal Oshri, Camiel Steenstra, Lennart Kats, Joanna ZouhourModel risk management in 2026: A banker's guide to the revised interagency guidance
The post presents the April 17, 2026 revision to model risk management guidance as a shift toward risk-based, principles-driven oversight, replacing several earlier issuances from the Federal Reserve, FDIC, and OCC. It identifies five operational changes: materiality tiering, end-to-end lifecycle governance, reproducible effective challenge, continuous drift and performance monitoring, and application of MRM principles to GenAI and agentic systems. Its proposed response is a Databricks reference architecture built on Unity Catalog, Delta Lake, Lakeflow Declarative Pipelines, Feature Store, MLflow, Model Registry, Model Serving, and assurance tools. The design makes tiering metadata-driven and turns lineage, validation, approvals, monitoring, documentation, and retirement records into evidence generated during normal model work. The stated conclusion is that a unified substrate can reduce integration and manual evidence work, allowing regulatory changes to become configuration exercises rather than multi-quarter programs.
Pavithra Rao, Jennifer Miller, Chaitanya Varanasi, Kim HattonOpenAI GPT-5.5 + Codex, now available and fully governed on Databricks
Databricks announces native support for OpenAI’s GPT-5.5, making it available for coding workflows with Codex, enterprise agents, document pipelines, and data-driven employee workflows. Access is governed through Unity AI Gateway, which provides permissions and rate limits, configurable guardrails for PII, prompt injection, and content safety, MCP tool-call auditing, failover, and request-level observability for model and Codex interactions. The announcement describes GPT-5.5 use with Genie for natural-language analytics, Agent Bricks Custom Agents for multi-step workflows, and Lakeflow Spark Declarative Pipelines for document-focused GenAI ETL involving summarization, extraction, and classification. GPT-5.5 is available on Databricks across AWS, Azure, and GCP, with entry points including AI Playground, governed Codex endpoints, Agent Bricks, and the Foundation Model API.
Patrick Wendell, Hanlin Tang, Ahmed Bilal, Arnav Singhvi, Ivan Zhou, Harish GaurAnnouncing the Public Preview of Lakeflow Designer
Databricks announces the Public Preview of Lakeflow Designer, a visual, no-code, AI-native experience for data preparation and analytics built directly into the Databricks platform. It targets analysts, domain experts, and other less technical users with drag-and-drop workflows and natural-language authoring, while representing each transformation as an operator with previews. Genie Code can use Unity Catalog metadata, descriptions, lineage, popularity, and example queries to generate or modify workflows, inspect transformation results, and iterate when needed. Every visual transformation produces production-ready Python code that can be reviewed, versioned in Git, and scheduled through Lakeflow Jobs, while users pay for compute rather than per-user licenses. The preview is available in all workspaces, although an administrator may need to enable Visual data prep in the preview portal.
Jason Messer, Emanuel Zgraggen, V Maharajh, Matt Jones, Tracy YangHow to transform document activation workflows with Genie and Agent Bricks
Enterprise document workflows often require people to read PDFs, copy fields into spreadsheets, and re-enter data into business systems, creating delays, error risk, and weak auditability. The proposed alternative combines AI/BI Genie for natural-language queries over Delta tables, Agent Bricks for production agents, and Unity Catalog for permissions, lineage, and traceability. A five-phase workflow extracts fields into Lakeflow Bronze, Silver, and Gold layers, then supports structured queries, clause-level RAG answers, multi-agent routing, and actions through MCP servers. Examples include syncing validated rights data to ERP or CRM systems, updating billing entitlements, and triggering workflows from detected obligations. The approach targets media, advertising, ad tech, and telecommunications, where customers see improvements including faster month-end close, recovered revenue, reduced leakage, and lower operational risk.
Elena TesserGet hands-on with agents, vibe coding and more at Data+ AI Summit
Data + AI Summit returns to San Francisco from June 14 through June 18, 2026, with Databricks Training and Certification offering more than 20 hands-on courses across AI agents, vibe coding, data engineering, and Lakebase. Training-exclusive days on Sunday and Monday are designed to provide dedicated learning time, while sessions continue during the conference week. New courses cover agentic development with Genie Code, Claude Code, Cursor, MCP servers, and structured prompting; production agent deployment with MLflow’s ResponsesAgent framework and tracing; Lakeflow Spark Declarative Pipelines; and AI/BI dashboards and Genie Spaces. Onsite certification exams cost $100 plus taxes, a 50% discount, and seven listed exams are available. The Learning Hub adds labs, skills assessments, customized learning plans, and community access.
Pratyarth RaoWhy your agents can't read enterprise documents — and how to fix it
Enterprise agents often reason well over clean text but struggle with scanned PDFs, inconsistent layouts, nested tables, images, and handwriting, making document processing an accuracy ceiling for business workflows. The post announces Document Intelligence, a pipeline of chainable AI Functions: ai_parse_document converts scans into layout-enriched structured text, while ai_classify and ai_extract support routing and entity extraction. On OfficeQA treasury-bond documents, preprocessing with ai_parse_document produced a 16% average performance gain across tested agent frameworks without changing their reasoning harness. Across invoice, contract, medical-note, and financial-filing extraction tasks, it reportedly delivered the highest accuracy at 5–7x lower cost than comparable pipelines, with serverless batch infrastructure scaling the same SQL call from 100 to 100,000. Databricks integrates ingestion, orchestration, governance, and agent building through Lakeflow Connect, Lakeflow Jobs or Spark Declarative Pipelines, Unity Catalog, and Agent Bricks, aiming to replace fragmented services with reusable workflows.
Archika Dogra, Sergei Tsarev, Erich ElsenBuilding with Databricks Document Intelligence and Lakeflow
The post presents a production-oriented approach to Intelligent Document Processing (IDP) that brings PDFs, images, and office documents into Databricks’ governed data lifecycle. Lakeflow Connect ingests files from sources such as SharePoint and Google Drive into Unity Catalog Volumes and tables, applying access control, lineage, and auditing while supporting incremental reads and writes. Databricks’ Document Intelligence then uses ai_parse_document to preserve structure in VARIANT output, with ai_extract, ai_classify, and ai_prep_search available for enrichment, routing, and retrieval preparation. Lakeflow Jobs productionizes the workflow with orchestration, control flow, retries, triggers, serverless scaling, and observability across ingestion, parsing, and serving tasks. The post also recommends joining extracted data to master data, using service principals, and monitoring drift and extraction accuracy, presenting the combined platform as a way to create trusted, queryable datasets from unstructured content.
Giselle Goicochea, Joanna Zouhour
