--- title: "Lakeflow" description: "11 posts about Lakeflow, summarised, each linking to the original." --- # Lakeflow > 11 posts about Lakeflow, summarised, each linking to the original. ## Articles ### [Databricks for Good and Virtue Foundation: Partnering to Connect Medical Volunteers to Critical Health Services in 72 Countries](https://yomu.fyi/post/databricks-for-good-and-virtue-foundation-partnering-to-connect-medica.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Priyanka Mehta, Shaunak Sen - Published: May 20, 2026 Virtue Foundation’s VF Match platform connects medical professionals with volunteer opportunities across 72 low and low-middle income countries, but scaling its initial LLM proof of concept into production required a robust data pipeline. The Databricks-based Foundational Data Refresh combines Overture Maps and Bright Data, processes more than 25 million web pages with targeted GPT extraction steps, and uses Spark, Photon, and Lakeflow Jobs to orchestrate checkpointed, modular workloads. Splink resolves duplicate facilities and nonprofits across messy sources; Photon reduced the slowest entity-resolution partition from 30 minutes to approximately two minutes, a 15x improvement. A prototype VF Agent adds natural-language access through LangGraph, Databricks Model Serving, AI Search, and Genie, while the resulting system provides an up-to-date global view of healthcare infrastructure and volunteer needs. ### [Automate Data & KPI Monitoring with SQL Alerts](https://yomu.fyi/post/automate-data-kpi-monitoring-with-sql-alerts.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Srilekha Dornadula - Published: May 19, 2026 Databricks SQL Alerts is generally available as an automated alternative to manually checking dashboards and rerunning monitoring queries for business KPIs, pipeline health, and data quality. Teams define a SQL query, evaluation condition, schedule, and notification destinations, then receive alerts when results cross configured guardrails; standalone alerts can also run within Lakeflow Jobs. GA supports authoring in the SQL editor, notifications through email, Slack, PagerDuty, Microsoft Teams, or webhooks, and production management through Git, Declarative Automation Bundles, APIs, Terraform, and SDKs. The post’s revenue example compares yesterday’s result with a seven-day average and triggers when revenue\_pct\_change is below -5%, while the SQL Alert task can expose OK, TRIGGERED, or ERROR to downstream pipeline logic. ### [How CFOs in consulting can recover margin with Databricks](https://yomu.fyi/post/how-cfos-in-consulting-can-recover-margin-with-databricks.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Rajaram Suresh, Kateryna Savchyn, Pavithra Rao, Mark Hahn, Kim Hatton, Sharmey Shah, Lisa Fox - Published: May 12, 2026 Consulting firms are facing margin pressure despite growing deal pipelines, with 2024 EBITDA margins at 9.8%, utilization at 68.9%, and revenue growth at 4.6%. The post attributes this to fragmented finance workflows spanning Salesforce, SAP, Workday, Concur, and spreadsheets, plus inconsistent metrics and manual reconciliation. It proposes using Databricks to create a governed data layer with Lakeflow, Lakehouse, and Unity Catalog, then adding Genie for natural-language analysis and finance apps that trigger workflows and write approvals back in real time. A global consulting firm reportedly cut cash forecasting cycles by 3–5 days and reduced reporting-cycle FTE hours by 80%, while Databricks’ own close fell from 15 days to 8. The proposed rollout spans 10–12 weeks and begins with one tightly scoped workflow such as revenue close, unbilled WIP, T&E leakage, or practice-margin analysis. ### [Agentic data engineering with Genie Code and Lakeflow](https://yomu.fyi/post/agentic-data-engineering-with-genie-code-and-lakeflow.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Gal Oshri, Camiel Steenstra, Lennart Kats, Joanna Zouhour - Published: Apr 28, 2026 Genie Code is presented as a natural-language assistant for developing, orchestrating, deploying, and debugging data pipelines and jobs. It uses pipeline and job context, including code, configuration, run results, Unity Catalog metadata, lineage, popularity, and code samples, to help engineers discover datasets and understand data flows. Engineers can describe pipelines or jobs, and Genie Code can generate Spark Declarative Pipelines with Bronze, Silver, and Gold layers, sources, transformations, data quality expectations, and outputs, then configure orchestration, schedules, dependencies, Auto Loader, AutoCDC flows, and Declarative Automation Bundles. It also analyzes failures and unexpected row-count or schema changes, proposes cross-file updates with reviewable diffs, and supports extensions through custom instructions, agent skills, and MCP servers. The stated result is faster development and guided debugging while workflows remain aligned with Unity Catalog governance, performance, and data quality standards; future plans include background failure response and cluster right-sizing. ### [Model risk management in 2026: A banker's guide to the revised interagency guidance](https://yomu.fyi/post/model-risk-management-in-2026-a-banker-s-guide-to-the-revised-interage.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Pavithra Rao, Jennifer Miller, Chaitanya Varanasi, Kim Hatton - Published: Apr 25, 2026 The post presents the April 17, 2026 revision to model risk management guidance as a shift toward risk-based, principles-driven oversight, replacing several earlier issuances from the Federal Reserve, FDIC, and OCC. It identifies five operational changes: materiality tiering, end-to-end lifecycle governance, reproducible effective challenge, continuous drift and performance monitoring, and application of MRM principles to GenAI and agentic systems. Its proposed response is a Databricks reference architecture built on Unity Catalog, Delta Lake, Lakeflow Declarative Pipelines, Feature Store, MLflow, Model Registry, Model Serving, and assurance tools. The design makes tiering metadata-driven and turns lineage, validation, approvals, monitoring, documentation, and retirement records into evidence generated during normal model work. The stated conclusion is that a unified substrate can reduce integration and manual evidence work, allowing regulatory changes to become configuration exercises rather than multi-quarter programs. ### [OpenAI GPT-5.5 + Codex, now available and fully governed on Databricks](https://yomu.fyi/post/openai-gpt-5-5-codex-now-available-and-fully-governed-on-databricks.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Patrick Wendell, Hanlin Tang, Ahmed Bilal, Arnav Singhvi, Ivan Zhou, Harish Gaur - Published: Apr 24, 2026 Databricks announces native support for OpenAI’s GPT-5.5, making it available for coding workflows with Codex, enterprise agents, document pipelines, and data-driven employee workflows. Access is governed through Unity AI Gateway, which provides permissions and rate limits, configurable guardrails for PII, prompt injection, and content safety, MCP tool-call auditing, failover, and request-level observability for model and Codex interactions. The announcement describes GPT-5.5 use with Genie for natural-language analytics, Agent Bricks Custom Agents for multi-step workflows, and Lakeflow Spark Declarative Pipelines for document-focused GenAI ETL involving summarization, extraction, and classification. GPT-5.5 is available on Databricks across AWS, Azure, and GCP, with entry points including AI Playground, governed Codex endpoints, Agent Bricks, and the Foundation Model API. ### [Announcing the Public Preview of Lakeflow Designer](https://yomu.fyi/post/announcing-the-public-preview-of-lakeflow-designer.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Jason Messer, Emanuel Zgraggen, V Maharajh, Matt Jones, Tracy Yang - Published: Apr 23, 2026 Databricks announces the Public Preview of Lakeflow Designer, a visual, no-code, AI-native experience for data preparation and analytics built directly into the Databricks platform. It targets analysts, domain experts, and other less technical users with drag-and-drop workflows and natural-language authoring, while representing each transformation as an operator with previews. Genie Code can use Unity Catalog metadata, descriptions, lineage, popularity, and example queries to generate or modify workflows, inspect transformation results, and iterate when needed. Every visual transformation produces production-ready Python code that can be reviewed, versioned in Git, and scheduled through Lakeflow Jobs, while users pay for compute rather than per-user licenses. The preview is available in all workspaces, although an administrator may need to enable Visual data prep in the preview portal. ### [How to transform document activation workflows with Genie and Agent Bricks](https://yomu.fyi/post/how-to-transform-document-activation-workflows-with-genie-and-agent-br.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Elena Tesser - Published: Apr 22, 2026 Enterprise document workflows often require people to read PDFs, copy fields into spreadsheets, and re-enter data into business systems, creating delays, error risk, and weak auditability. The proposed alternative combines AI/BI Genie for natural-language queries over Delta tables, Agent Bricks for production agents, and Unity Catalog for permissions, lineage, and traceability. A five-phase workflow extracts fields into Lakeflow Bronze, Silver, and Gold layers, then supports structured queries, clause-level RAG answers, multi-agent routing, and actions through MCP servers. Examples include syncing validated rights data to ERP or CRM systems, updating billing entitlements, and triggering workflows from detected obligations. The approach targets media, advertising, ad tech, and telecommunications, where customers see improvements including faster month-end close, recovered revenue, reduced leakage, and lower operational risk. ### [Get hands-on with agents, vibe coding and more at Data+ AI Summit](https://yomu.fyi/post/get-hands-on-with-agents-vibe-coding-and-more-at-data-ai-summit.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Pratyarth Rao - Published: Apr 20, 2026 Data + AI Summit returns to San Francisco from June 14 through June 18, 2026, with Databricks Training and Certification offering more than 20 hands-on courses across AI agents, vibe coding, data engineering, and Lakebase. Training-exclusive days on Sunday and Monday are designed to provide dedicated learning time, while sessions continue during the conference week. New courses cover agentic development with Genie Code, Claude Code, Cursor, MCP servers, and structured prompting; production agent deployment with MLflow’s ResponsesAgent framework and tracing; Lakeflow Spark Declarative Pipelines; and AI/BI dashboards and Genie Spaces. Onsite certification exams cost $100 plus taxes, a 50% discount, and seven listed exams are available. The Learning Hub adds labs, skills assessments, customized learning plans, and community access. ### [Why your agents can't read enterprise documents — and how to fix it](https://yomu.fyi/post/why-your-agents-can-t-read-enterprise-documents-and-how-to-fix-it.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Archika Dogra, Sergei Tsarev, Erich Elsen - Published: Apr 16, 2026 Enterprise agents often reason well over clean text but struggle with scanned PDFs, inconsistent layouts, nested tables, images, and handwriting, making document processing an accuracy ceiling for business workflows. The post announces Document Intelligence, a pipeline of chainable AI Functions: ai\_parse\_document converts scans into layout-enriched structured text, while ai\_classify and ai\_extract support routing and entity extraction. On OfficeQA treasury-bond documents, preprocessing with ai\_parse\_document produced a 16% average performance gain across tested agent frameworks without changing their reasoning harness. Across invoice, contract, medical-note, and financial-filing extraction tasks, it reportedly delivered the highest accuracy at 5–7x lower cost than comparable pipelines, with serverless batch infrastructure scaling the same SQL call from 100 to 100,000. Databricks integrates ingestion, orchestration, governance, and agent building through Lakeflow Connect, Lakeflow Jobs or Spark Declarative Pipelines, Unity Catalog, and Agent Bricks, aiming to replace fragmented services with reusable workflows. ### [Building with Databricks Document Intelligence and Lakeflow](https://yomu.fyi/post/building-with-databricks-document-intelligence-and-lakeflow.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Giselle Goicochea, Joanna Zouhour - Published: Apr 16, 2026 The post presents a production-oriented approach to Intelligent Document Processing (IDP) that brings PDFs, images, and office documents into Databricks’ governed data lifecycle. Lakeflow Connect ingests files from sources such as SharePoint and Google Drive into Unity Catalog Volumes and tables, applying access control, lineage, and auditing while supporting incremental reads and writes. Databricks’ Document Intelligence then uses ai\_parse\_document to preserve structure in VARIANT output, with ai\_extract, ai\_classify, and ai\_prep\_search available for enrichment, routing, and retrieval preparation. Lakeflow Jobs productionizes the workflow with orchestration, control flow, retries, triggers, serverless scaling, and observability across ingestion, parsing, and serving tasks. The post also recommends joining extracted data to master data, using service principals, and monitoring drift and extraction accuracy, presenting the combined platform as a way to create trusted, queryable datasets from unstructured content.