--- title: "SQL" description: "14 posts about SQL, summarised, each linking to the original." --- # SQL > 14 posts about SQL, summarised, each linking to the original. ## Articles ### [Convert proprietary code to open ANSI SQL with Genie Code](https://yomu.fyi/post/convert-proprietary-code-to-open-ansi-sql-with-genie-code.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Jonathan Brito - Published: Jul 30, 2026 Databricks introduces an agentic converter in Genie Code to translate proprietary warehouse dialects into open ANSI SQL, initially supporting T-SQL, Snowflake, Redshift, Oracle, BigQuery, and Teradata. Migration projects provide a workspace hub for source files, conversion status, complexity scoring, dependency lineage, and collaboration, helping teams prioritize work and identify scripts that can move independently. When launched, swarms of subagents convert files in parallel, iteratively fixing errors and validating syntax and semantic intent; in the proof-of-concept, six of eight files converted successfully. Two files required review because stored procedures needed three-part Unity Catalog names, which could be fixed manually or encoded as a custom Genie Code skill for reuse across the codebase. The post also notes that Databricks supports multi-statement transactions, temporary tables, and stored procedures, while planned additions include legacy ETL sources, new target dialects, data migration, and reconciliation. ### [Introducing Apache Spark 4.2](https://yomu.fyi/post/introducing-apache-spark-4-2.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Wenchen Fan, Andreas Neumann, Serge Rielau, Szehon Ho, Gengliang Wang, Linhong Liu, Hyukjin Kwon, Jerry Peng, DB Tsai, Xiao Li, Reynold Xin - Published: Jul 16, 2026 Apache Spark 4.2 extends the engine’s role in modern data and AI workloads with governed metrics, vector and top-K primitives, Arrow-first Python execution, native change data capture, and stronger streaming foundations. Metric views provide shared business definitions, while Spark Connect uses gRPC and Arrow to let remote clients submit logical plans without a full Spark runtime. Spark SQL adds vector similarity functions, NEAREST BY, geospatial types, sketches, and time-series features; Python interoperability includes Arrow UDFs and can move Spark DataFrames into supported Arrow-native tools without copying or serializing underlying data. Spark Declarative Pipelines adds Auto CDC for SCD Type 1 targets, while Data Source V2 standardizes change streams through CHANGES and expands row-level operations and schema evolution. The release also includes operational updates such as Web UI modernization, Kubernetes improvements, JDK 25 support, and dependency upgrades. ### [Barracuda makes security logs conversational with Genie](https://yomu.fyi/post/barracuda-makes-security-logs-conversational-with-genie.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Barracuda XDR Engineering Team - Published: Jul 6, 2026 Barracuda Managed XDR needed to make security-log investigations accessible beyond analysts who knew each vendor’s schema and could write SQL. Its AI-Powered Log Search integrates Databricks Genie into the XDR dashboard: Genie translates natural-language questions into SQL, runs them against customer logs, and returns tabular results with the generated query. Security-specific metadata in Unity Catalog helps map requests across network, server, cloud, email, and endpoint sources, while multi-turn context supports iterative filtering. Secure views enforce organization-level filtering before execution; scoped service principals and SQL validation add further controls against cross-tenant access. Barracuda reports reducing routine investigation time from hours to minutes and estimates that self-service could free 83 to 100 SOC analyst hours monthly, or 1,000 to 1,200 annually. ### [Granular Usage Attribution for dbt Pipelines with Query Tags](https://yomu.fyi/post/granular-usage-attribution-for-dbt-pipelines-with-query-tags.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Heeren Sharma, Lennart Reschke, JooHo Yeo - Published: Jul 1, 2026 Databricks Query Tags give dbt pipelines granular usage attribution when identical query-history rows obscure model ownership, cost, and performance. The dbt-databricks adapter version 1.11+ automatically injects model, materialization, dbt-core, and adapter-version metadata, while profile-level and model-level configurations add custom dimensions such as team, cost center, project, and environment. These tags are stored as a MAP in system.query.history, where SQL can extract them for cost attribution, performance debugging, monitoring, and dashboards. In the reference project, four mart tables accounted for 92% of compute time, whereas staging views and metric views were near-instantaneous. The accompanying open-source project demonstrates Genie exploration, a self-monitoring AI/BI dashboard, metric-view tagging, and deployment with Databricks bundles, while recommending consistent profile hierarchies, environment tags, and limited custom metadata. ### [Building a SQL ETL Pipeline: The Complete Guide for Data Engineers](https://yomu.fyi/post/building-a-sql-etl-pipeline-the-complete-guide-for-data-engineers.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Databricks Staff - Published: Jun 18, 2026 SQL ETL pipelines are presented as repeatable workflows that extract data from sources, transform it, and load it into warehouses, lakes, or lakehouses for analysis and machine-learning use. The guide addresses source connectivity, extraction patterns, transformation logic, loading targets, governance, performance, testing, and operational design, while contrasting ETL with ELT and broader data pipelines. It explains that SQL can serve as the primary implementation language for transformations and load operations, with techniques including JOIN and GROUP BY, window functions, MERGE upserts, and deduplication with ROW\_NUMBER() or DISTINCT. It also covers full versus incremental extraction, batch and streaming needs, schema-on-write versus schema-on-read, and layered validation using row counts, checksums, business rules, and schema-drift monitoring. ### [Geospatial Unbounded: Spatial SQL GA with AI/BI Maps, Delta Sharing, and Iceberg v3](https://yomu.fyi/post/geospatial-unbounded-spatial-sql-ga-with-ai-bi-maps-delta-sharing-and.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Kent Marten - Published: Jun 11, 2026 Databricks announces Spatial SQL as Generally Available, positioning its platform for geospatial analysis without separate spatial databases, warehouses, and mapping tools. It supports native GEOMETRY columns in Delta or Iceberg, more than 90 OGC-compliant ST\_\* functions, spatial joins, and boolean set operations. AI/BI dashboards can render Geometry and Geography columns as maps, while Genie can generate spatial queries and dashboards and respect Unity Catalog row filters. Geo columns are supported by Delta Sharing, and Databricks can read and write managed Iceberg tables or read externally written Iceberg tables with geospatial types in Iceberg v3. Benchmarks show eight of twelve SpatialBench queries improved since Public Preview, with gains from 20% to 15X, while areal boolean operations are twice as fast on average versus prior versions. ### [Query Tags: The Context Your Warehouse Queries Have Been Missing](https://yomu.fyi/post/query-tags-the-context-your-warehouse-queries-have-been-missing.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: JooHo Yeo, Jiabin Hu - Published: Jun 2, 2026 Databricks SQL Query Tags address a visibility gap in warehouse workloads: built-in logs identify users, warehouses, and tools, but not dashboards, projects, teams, or cost centers. Query Tags, now in Public Preview, attach multiple custom key-value pairs to each SQL execution, carry them into the Query History System Table, and expose them in the Query Profile UI. Partner integrations can automatically tag dbt models, while Power BI and Tableau support connection-level tags, and APIs and connectors support connection- or statement-level metadata. Users can also issue SET QUERY\_TAGS in SQL Editor, notebooks, dashboards, or alerts to label subsequent session statements. These tags enable SQL or Genie queries for cost allocation, regression analysis, workload filtering, and environment comparisons; future plans include broader connector and workload support plus Query History search. ### [Automate Data & KPI Monitoring with SQL Alerts](https://yomu.fyi/post/automate-data-kpi-monitoring-with-sql-alerts.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Srilekha Dornadula - Published: May 19, 2026 Databricks SQL Alerts is generally available as an automated alternative to manually checking dashboards and rerunning monitoring queries for business KPIs, pipeline health, and data quality. Teams define a SQL query, evaluation condition, schedule, and notification destinations, then receive alerts when results cross configured guardrails; standalone alerts can also run within Lakeflow Jobs. GA supports authoring in the SQL editor, notifications through email, Slack, PagerDuty, Microsoft Teams, or webhooks, and production management through Git, Declarative Automation Bundles, APIs, Terraform, and SDKs. The post’s revenue example compares yesterday’s result with a seven-day average and triggers when revenue\_pct\_change is below -5%, while the SQL Alert task can expose OK, TRIGGERED, or ERROR to downstream pipeline logic. ### [Announcing the Databricks analytics engineer learning pathway](https://yomu.fyi/post/announcing-the-databricks-analytics-engineer-learning-pathway.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Maroua Lazzarou, Pratyarth Rao - Published: May 18, 2026 Databricks is launching the Analytics Engineer Learning Pathway, a curriculum for SQL practitioners that covers data modeling, pipelines, metrics, and Genie spaces on the lakehouse. The pathway teaches learners to transform raw data into governed, AI-ready semantic models and metric views, the foundation for analytics, dashboards, and AI agents. Courses cover Analytics Fundamentals, production SQL ETL with Materialized Views, Streaming Tables, AUTO CDC, and Lakeflow Jobs, plus data modeling with Delta Lake and Unity Catalog. The curriculum also addresses metric views, Genie spaces, governance with Unity Catalog permissions and ABAC policies, and Spark Declarative Pipelines with expectations, event logs, and metrics. Courses are available in self-paced and instructor-led formats through Databricks Academy, with the full pathway included in active learning subscriptions. ### [Rethinking SQL ETL for modern data platforms](https://yomu.fyi/post/rethinking-sql-etl-for-modern-data-platforms.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Matt Jones, Shanelle Roman - Published: Apr 29, 2026 SQL ETL is often distributed across a warehouse, transformation framework, orchestrator, monitoring, lineage, and data-quality systems, creating operational fragmentation as data teams scale. The post proposes a unified platform model that combines execution, orchestration, observability, governance, and optimization while supporting analytics engineers, warehouse engineers, and analysts through dbt, scripts and stored procedures, Materialized Views, declarative pipelines, and no-code tools. Databricks implements this model with shared execution, governance, and observability, using serverless infrastructure and AI-driven optimization alongside open table formats and ANSI SQL. The stated benefits include reduced operational overhead, support for batch and real-time workloads, and incremental modernization rather than disruptive rewrites. Reported examples include HP’s 32% cloud savings and 36% lower combined job runtime after moving to serverless compute, and Adobe reducing query execution from eight minutes to three seconds with Materialized Views. ### [Powering KPMG UK Audit's AI future with Databricks](https://yomu.fyi/post/powering-kpmg-uk-audit-s-ai-future-with-databricks.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Mark Wallington, Greta Nasai - Published: Apr 27, 2026 KPMG UK is evolving its Audit data platform with Databricks to support AI-enabled analytics while preserving governance, rigour, and professional standards. The program converges structured data, advanced analytics, and AI on a unified cloud-native Lakehouse, with Databricks SQL replacing core SQL Server workloads and Delta underpinning the platform. Lakebridge assessed migration complexity, while Databricks-hosted large language models including Claude Sonnet and Genie Code helped convert T-SQL, refactor stored procedures, modularize queries, and suggest Delta-based optimizations under engineer review. This reduced refactoring time by around 60% and enabled modernization of more than 400 scripts and stored procedures in roughly three months. Databricks SQL Serverless provides elastic compute for spiky workloads, while Genie offers traceable, version-controlled SQL and Delta Sharing supports governed data exchange. ### [Streamlining RiskOps with the SOP agent framework](https://yomu.fyi/post/streamlining-riskops-with-the-sop-agent-framework.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Fujiao Liu - Published: May 8, 2025 Risk Operations teams investigating Account Takeover (ATO) cases must gather data across systems, run SQL queries, and make decisions under pressure, creating delays and error risk. The SOP-driven LLM agent framework represents the workflow as a natural-language, indentation-based tree whose steps specify queries, external functions, and decision criteria. During execution, the SOP planner navigates the tree and produces JSON-formatted instructions, while a Worker Agent invokes APIs or SQL queries and stores results in a feedback loop. The process ends by synthesising investigation data into a report and final decision. In reported results, handling time fell from 22 to 3 minutes per ticket, 87% of ATO cases were automated since launch, and the error rate was zero. ### [Building Ramp's MCP server](https://yomu.fyi/post/building-ramp-s-mcp-server.md) - Company: [Ramp](https://yomu.fyi/company/ramp.md) - Author: Junho Choi - Published: Mar 25, 2025 Ramp built an open-source MCP server that exposes its developer API to LLMs through tools and a SQL interface, enabling natural-language access to business spend data. Using FastMCP in Python and Claude Desktop, the prototype could run analyses, generate visualizations, and issue cards on demand, but pagination struggled beyond a few hundred transactions because of miscalculations, context limits, input sizes, and token use. The team replaced bulk responses with an in-memory ETL that flattens API JSON into SQLite, then exposes load, transformation, and query tools; an OLAP-powered API was added for reporting workloads that caused timeouts. This let Claude analyze tens of thousands of spend events with aggregate and window functions while reducing token usage, although API latency, occasional reliability problems, tool-selection errors, and security risks remain; audit logs, OAuth scopes, and constrained tools provide mitigations, while write actions await a safety framework. ### [7 Ways to Make Your SQL Workshop Beginner-friendly](https://yomu.fyi/post/7-ways-to-make-your-sql-workshop-beginner-friendly.md) - Company: [Shopify](https://yomu.fyi/company/shopify.md) - Author: 2023-10-18 - Published: May 21, 2020 Shopify’s 90-minute SQL workshop serves new RnD hires with varied technical backgrounds, including participants who may have no experience with databases or programming. It separates beginners from intermediate learners and aims to help beginners use data responsibly to discover insights. The beginner format spends its first 30 minutes on data warehouse architecture, data preparation, and privacy, then uses real production data and tools such as Mode Analytics with an accessible dataset. The curriculum identifies core SQL objectives, translates them into practical questions, starts with SELECT \* exploration, and builds queries toward a larger analytical question. It also provides beginner, intermediate, and advanced learning resources covering topics from foundational clauses and aggregations to joins, window functions, CTEs, and Presto-specific techniques.