How Apache DataFusion Runs SQL In-Process
Apache DataFusion is an embeddable query engine: SQL goes in, runs as columnar Arrow batches inside your own process, and results come out without a separate service.
DatabricksJason Messer, Emanuel Zgraggen, V Maharajh, Kyrylo Bakumenko, Xiao Yan, Matt JonesAnnouncement
Lakeflow Designer, a visual no-code data preparation tool in Databricks, added three new operators for cleaning, deduplication, and quality checks, addressing customer requests for more no-code capabilities. The Prepare operator standardizes data by changing case, converting types, replacing values, and applying formulas; Unique removes duplicates across selected columns with ordering control; Guardrails enforces quality rules with configurable failure handling. Existing operators gained multi-table joins with separate outputs for matched, left unmatched, and right unmatched rows, branching in Filter, and dynamic column selection in Select. Output options expanded to materialized views in Unity Catalog that refresh automatically, merge into existing tables, and direct file writes to CSV, Excel, or JSON with dynamic worksheets. Canvas improvements include a redesigned interface with better contrast and larger connection targets for large workflows, a searchable table of contents, visual version history, and live cursor presence.
Databricks Lakeflow Designer is a visual no-code data preparation tool that launched to GA in June 2026. This post details updates from the following four months, including new operators, operator enhancements, expanded outputs, and canvas improvements.
The updates include three new no-code operators (Prepare, Unique, Guardrails), enhancements to Join, Filter, and Select operators, expanded output options (materialized views, merge, file writes), and canvas improvements (redesign, table of contents, version history, live cursor presence).
Lakeflow Designer now includes no-code operators for data cleaning (Prepare), deduplication (Unique), and data quality checks (Guardrails), reducing the need for custom code.
The Join operator supports multi-table joins and provides separate outputs for matched, left unmatched, and right unmatched rows, enabling branching workflows.
Output options have been expanded to materialized views in Unity Catalog, merge into existing tables, and direct file writes to CSV, Excel, or JSON, with dynamic worksheet support for Excel.
Yes, select Visual Data Prep in the left navigation to get started in Free Edition.
There are no per-user licenses or separate licensing fees; you only pay for the compute you use.
Yes, place Lakeflow Designer files in a Databricks Git folder to track changes and sync with your Git provider; each file contains both visual layout and underlying Python code.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
Apache DataFusion is an embeddable query engine: SQL goes in, runs as columnar Arrow batches inside your own process, and results come out without a separate service.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.