--- title: "SQL" description: "42 posts about SQL, summarised, each linking to the original." --- # SQL > 42 posts about SQL, summarised, each linking to the original. ## Articles ### [Pipelined SQL in ClickHouse 26.8](https://yomu.fyi/post/pipelined-sql-in-clickhouse-26-8.md) - Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md) - Published: Sep 4, 2026 - Type: Explainer ClickHouse 26.8 introduces pipelined SQL, extending the earlier ability to place FROM before SELECT with a |> operator for sequencing query transformations. Using the UK property prices dataset, the post compares conventional, FROM-first, and pipelined forms for filtering London sales, aggregating median prices by district, ordering results, and limiting output. Each pipeline stage acts as a complete-query checkpoint, while EXPLAIN SYNTAX shows that ClickHouse translates the syntax into nested standard SQL and optimizes the complete query without materializing every intermediate result. The syntax also supports EXTEND, reuse of aggregation aliases in later stages, views, and INSERT ... SELECT statements, though stage order affects which rows feed subsequent aggregations in these examples; the conclusion is that pipelines complement rather than replace conventional SQL. ### [Convert proprietary code to open ANSI SQL with Genie Code](https://yomu.fyi/post/convert-proprietary-code-to-open-ansi-sql-with-genie-code.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Jonathan Brito - Published: Jul 30, 2026 - Type: Announcement Databricks introduces an agentic converter in Genie Code to translate proprietary warehouse dialects into open ANSI SQL, initially supporting T-SQL, Snowflake, Redshift, Oracle, BigQuery, and Teradata. Migration projects provide a workspace hub for source files, conversion status, complexity scoring, dependency lineage, and collaboration, helping teams prioritize work and identify scripts that can move independently. When launched, swarms of subagents convert files in parallel, iteratively fixing errors and validating syntax and semantic intent; in the proof-of-concept, six of eight files converted successfully. Two files required review because stored procedures needed three-part Unity Catalog names, which could be fixed manually or encoded as a custom Genie Code skill for reuse across the codebase. The post also notes that Databricks supports multi-statement transactions, temporary tables, and stored procedures, while planned additions include legacy ETL sources, new target dialects, data migration, and reconciliation. ### [Introducing Apache Spark 4.2](https://yomu.fyi/post/introducing-apache-spark-4-2.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Wenchen Fan, Andreas Neumann, Serge Rielau, Szehon Ho, Gengliang Wang, Linhong Liu, Hyukjin Kwon, Jerry Peng, DB Tsai, Xiao Li, Reynold Xin - Published: Jul 16, 2026 - Type: Announcement Apache Spark 4.2 extends the engine’s role in modern data and AI workloads with governed metrics, vector and top-K primitives, Arrow-first Python execution, native change data capture, and stronger streaming foundations. Metric views provide shared business definitions, while Spark Connect uses gRPC and Arrow to let remote clients submit logical plans without a full Spark runtime. Spark SQL adds vector similarity functions, NEAREST BY, geospatial types, sketches, and time-series features; Python interoperability includes Arrow UDFs and can move Spark DataFrames into supported Arrow-native tools without copying or serializing underlying data. Spark Declarative Pipelines adds Auto CDC for SCD Type 1 targets, while Data Source V2 standardizes change streams through CHANGES and expands row-level operations and schema evolution. The release also includes operational updates such as Web UI modernization, Kubernetes improvements, JDK 25 support, and dependency upgrades. ### [Barracuda makes security logs conversational with Genie](https://yomu.fyi/post/barracuda-makes-security-logs-conversational-with-genie.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Barracuda XDR Engineering Team - Published: Jul 6, 2026 - Type: Problem & solution Barracuda Managed XDR needed to make security-log investigations accessible beyond analysts who knew each vendor’s schema and could write SQL. Its AI-Powered Log Search integrates Databricks Genie into the XDR dashboard: Genie translates natural-language questions into SQL, runs them against customer logs, and returns tabular results with the generated query. Security-specific metadata in Unity Catalog helps map requests across network, server, cloud, email, and endpoint sources, while multi-turn context supports iterative filtering. Secure views enforce organization-level filtering before execution; scoped service principals and SQL validation add further controls against cross-tenant access. Barracuda reports reducing routine investigation time from hours to minutes and estimates that self-service could free 83 to 100 SOC analyst hours monthly, or 1,000 to 1,200 annually. ### [Presto® on Apache Kafka® At Uber Scale](https://yomu.fyi/post/presto-on-apache-kafka-at-uber-scale.md) - Company: [Uber](https://yomu.fyi/company/uber.md) - Published: Jul 2, 2026 - Type: Problem & solution Uber connected Presto and Apache Kafka to provide lightweight, interactive SQL queries over Kafka topics for ad-hoc analysis of real-time data. The work addresses operational requests such as checking whether an order UUID appears in a service’s input or output streams, where streaming processors are poorly suited to historical point lookups and real-time OLAP requires onboarding and dedicated resources. The enhanced connector discovers Kafka clusters, topics, and schemas on demand, supports multiple Kafka clusters, caches metadata, enforces \_timestamp or \_partition\_offset filters, and uses a shared Kafka client ID with broker quotas to control consumption. After rollout, example lookups returned within a few seconds, daily usage reached 6,000 queries, and the teams reported improved productivity, while concurrent queries may take longer under quota control. ### [Granular Usage Attribution for dbt Pipelines with Query Tags](https://yomu.fyi/post/granular-usage-attribution-for-dbt-pipelines-with-query-tags.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Heeren Sharma, Lennart Reschke, JooHo Yeo - Published: Jul 1, 2026 - Type: Tutorial Databricks Query Tags give dbt pipelines granular usage attribution when identical query-history rows obscure model ownership, cost, and performance. The dbt-databricks adapter version 1.11+ automatically injects model, materialization, dbt-core, and adapter-version metadata, while profile-level and model-level configurations add custom dimensions such as team, cost center, project, and environment. These tags are stored as a MAP in system.query.history, where SQL can extract them for cost attribution, performance debugging, monitoring, and dashboards. In the reference project, four mart tables accounted for 92% of compute time, whereas staging views and metric views were near-instantaneous. The accompanying open-source project demonstrates Genie exploration, a self-monitoring AI/BI dashboard, metric-view tagging, and deployment with Databricks bundles, while recommending consistent profile hierarchies, environment tags, and limited custom metadata. ### [Building a SQL ETL Pipeline: The Complete Guide for Data Engineers](https://yomu.fyi/post/building-a-sql-etl-pipeline-the-complete-guide-for-data-engineers.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Databricks Staff - Published: Jun 18, 2026 - Type: Tutorial SQL ETL pipelines are presented as repeatable workflows that extract data from sources, transform it, and load it into warehouses, lakes, or lakehouses for analysis and machine-learning use. The guide addresses source connectivity, extraction patterns, transformation logic, loading targets, governance, performance, testing, and operational design, while contrasting ETL with ELT and broader data pipelines. It explains that SQL can serve as the primary implementation language for transformations and load operations, with techniques including JOIN and GROUP BY, window functions, MERGE upserts, and deduplication with ROW\_NUMBER() or DISTINCT. It also covers full versus incremental extraction, batch and streaming needs, schema-on-write versus schema-on-read, and layered validation using row counts, checksums, business rules, and schema-drift monitoring. ### [Geospatial Unbounded: Spatial SQL GA with AI/BI Maps, Delta Sharing, and Iceberg v3](https://yomu.fyi/post/geospatial-unbounded-spatial-sql-ga-with-ai-bi-maps-delta-sharing-and.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Kent Marten - Published: Jun 11, 2026 - Type: Announcement Databricks announces Spatial SQL as Generally Available, positioning its platform for geospatial analysis without separate spatial databases, warehouses, and mapping tools. It supports native GEOMETRY columns in Delta or Iceberg, more than 90 OGC-compliant ST\_\* functions, spatial joins, and boolean set operations. AI/BI dashboards can render Geometry and Geography columns as maps, while Genie can generate spatial queries and dashboards and respect Unity Catalog row filters. Geo columns are supported by Delta Sharing, and Databricks can read and write managed Iceberg tables or read externally written Iceberg tables with geospatial types in Iceberg v3. Benchmarks show eight of twelve SpatialBench queries improved since Public Preview, with gains from 20% to 15X, while areal boolean operations are twice as fast on average versus prior versions. ### [Query Tags: The Context Your Warehouse Queries Have Been Missing](https://yomu.fyi/post/query-tags-the-context-your-warehouse-queries-have-been-missing.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: JooHo Yeo, Jiabin Hu - Published: Jun 2, 2026 - Type: Announcement Databricks SQL Query Tags address a visibility gap in warehouse workloads: built-in logs identify users, warehouses, and tools, but not dashboards, projects, teams, or cost centers. Query Tags, now in Public Preview, attach multiple custom key-value pairs to each SQL execution, carry them into the Query History System Table, and expose them in the Query Profile UI. Partner integrations can automatically tag dbt models, while Power BI and Tableau support connection-level tags, and APIs and connectors support connection- or statement-level metadata. Users can also issue SET QUERY\_TAGS in SQL Editor, notebooks, dashboards, or alerts to label subsequent session statements. These tags enable SQL or Genie queries for cost allocation, regression analysis, workload filtering, and environment comparisons; future plans include broader connector and workload support plus Query History search. ### [Automate Data & KPI Monitoring with SQL Alerts](https://yomu.fyi/post/automate-data-kpi-monitoring-with-sql-alerts.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Srilekha Dornadula - Published: May 19, 2026 - Type: Announcement Databricks SQL Alerts is generally available as an automated alternative to manually checking dashboards and rerunning monitoring queries for business KPIs, pipeline health, and data quality. Teams define a SQL query, evaluation condition, schedule, and notification destinations, then receive alerts when results cross configured guardrails; standalone alerts can also run within Lakeflow Jobs. GA supports authoring in the SQL editor, notifications through email, Slack, PagerDuty, Microsoft Teams, or webhooks, and production management through Git, Declarative Automation Bundles, APIs, Terraform, and SDKs. The post’s revenue example compares yesterday’s result with a seven-day average and triggers when revenue\_pct\_change is below -5%, while the SQL Alert task can expose OK, TRIGGERED, or ERROR to downstream pipeline logic. ### [Announcing the Databricks analytics engineer learning pathway](https://yomu.fyi/post/announcing-the-databricks-analytics-engineer-learning-pathway.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Maroua Lazzarou, Pratyarth Rao - Published: May 18, 2026 - Type: Announcement Databricks is launching the Analytics Engineer Learning Pathway, a curriculum for SQL practitioners that covers data modeling, pipelines, metrics, and Genie spaces on the lakehouse. The pathway teaches learners to transform raw data into governed, AI-ready semantic models and metric views, the foundation for analytics, dashboards, and AI agents. Courses cover Analytics Fundamentals, production SQL ETL with Materialized Views, Streaming Tables, AUTO CDC, and Lakeflow Jobs, plus data modeling with Delta Lake and Unity Catalog. The curriculum also addresses metric views, Genie spaces, governance with Unity Catalog permissions and ABAC policies, and Spark Declarative Pipelines with expectations, event logs, and metrics. Courses are available in self-paced and instructor-led formats through Databricks Academy, with the full pathway included in active learning subscriptions. ### [Rethinking SQL ETL for modern data platforms](https://yomu.fyi/post/rethinking-sql-etl-for-modern-data-platforms.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Matt Jones, Shanelle Roman - Published: Apr 29, 2026 - Type: Problem & solution SQL ETL is often distributed across a warehouse, transformation framework, orchestrator, monitoring, lineage, and data-quality systems, creating operational fragmentation as data teams scale. The post proposes a unified platform model that combines execution, orchestration, observability, governance, and optimization while supporting analytics engineers, warehouse engineers, and analysts through dbt, scripts and stored procedures, Materialized Views, declarative pipelines, and no-code tools. Databricks implements this model with shared execution, governance, and observability, using serverless infrastructure and AI-driven optimization alongside open table formats and ANSI SQL. The stated benefits include reduced operational overhead, support for batch and real-time workloads, and incremental modernization rather than disruptive rewrites. Reported examples include HP’s 32% cloud savings and 36% lower combined job runtime after moving to serverless compute, and Adobe reducing query execution from eight minutes to three seconds with Materialized Views. ### [Powering KPMG UK Audit's AI future with Databricks](https://yomu.fyi/post/powering-kpmg-uk-audit-s-ai-future-with-databricks.md) - Company: [Databricks](https://yomu.fyi/company/databricks.md) - Author: Mark Wallington, Greta Nasai - Published: Apr 27, 2026 - Type: Problem & solution KPMG UK is evolving its Audit data platform with Databricks to support AI-enabled analytics while preserving governance, rigour, and professional standards. The program converges structured data, advanced analytics, and AI on a unified cloud-native Lakehouse, with Databricks SQL replacing core SQL Server workloads and Delta underpinning the platform. Lakebridge assessed migration complexity, while Databricks-hosted large language models including Claude Sonnet and Genie Code helped convert T-SQL, refactor stored procedures, modularize queries, and suggest Delta-based optimizations under engineer review. This reduced refactoring time by around 60% and enabled modernization of more than 400 scripts and stored procedures in roughly three months. Databricks SQL Serverless provides elastic compute for spiky workloads, while Genie offers traceable, version-controlled SQL and Delta Sharing supports governed data exchange. ### [Unlocking Faster Insights with Experimenter-Defined Segmentations](https://yomu.fyi/post/unlocking-faster-insights-with-experimenter-defined-segmentations.md) - Company: [Etsy](https://yomu.fyi/company/etsy.md) - Author: Athena Chang - Published: Oct 8, 2025 - Type: Problem & solution At Etsy, Catapult supported only 13 hard-coded segmentations, forcing experimenters who needed other cuts to rely on slow, ad-hoc SQL analysis outside the platform. They replaced that fixed model with a Segmentation Library and creation tool where experimenters define reusable segmentations as SQL queries plus join columns connecting segment data to experimental units. Each definition is validated before production, then processed through a configuration-driven pipeline that detects new segmentations and generates independent tasks, improving failure isolation and allowing downstream work to wait only for required results. In the first three months, teams added 34 segmentations, increasing support from 13 to 47, while 78% of experimenters reported saving 2–5 hours monthly through custom analysis in Catapult. ### [Streamlining RiskOps with the SOP agent framework](https://yomu.fyi/post/streamlining-riskops-with-the-sop-agent-framework.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Fujiao Liu - Published: May 8, 2025 - Type: Problem & solution Risk Operations teams investigating Account Takeover (ATO) cases must gather data across systems, run SQL queries, and make decisions under pressure, creating delays and error risk. The SOP-driven LLM agent framework represents the workflow as a natural-language, indentation-based tree whose steps specify queries, external functions, and decision criteria. During execution, the SOP planner navigates the tree and produces JSON-formatted instructions, while a Worker Agent invokes APIs or SQL queries and stores results in a feedback loop. The process ends by synthesising investigation data into a report and final decision. In reported results, handling time fell from 22 to 3 minutes per ticket, 87% of ATO cases were automated since launch, and the error rate was zero. ### [Building Ramp's MCP server](https://yomu.fyi/post/building-ramp-s-mcp-server.md) - Company: [Ramp](https://yomu.fyi/company/ramp.md) - Author: Junho Choi - Published: Mar 25, 2025 - Type: Explainer Ramp built an open-source MCP server that exposes its developer API to LLMs through tools and a SQL interface, enabling natural-language access to business spend data. Using FastMCP in Python and Claude Desktop, the prototype could run analyses, generate visualizations, and issue cards on demand, but pagination struggled beyond a few hundred transactions because of miscalculations, context limits, input sizes, and token use. The team replaced bulk responses with an in-memory ETL that flattens API JSON into SQLite, then exposes load, transformation, and query tools; an OLAP-powered API was added for reporting workloads that caused timeouts. This let Claude analyze tens of thousands of spend events with aggregate and window functions while reducing token usage, although API latency, occasional reliability problems, tool-selection errors, and security risks remain; audit logs, OAuth scopes, and constrained tools provide mitigations, while write actions await a safety framework. ### [Skip the API, Ship Your Database](https://yomu.fyi/post/skip-the-api-ship-your-database.md) - Company: [Fly.io](https://yomu.fyi/company/fly-io.md) - Published: Sep 13, 2023 - Type: Explainer The post describes an unexpected use for LiteFS, a distributed SQLite database: replacing service-to-service APIs with read-only database replicas. Typical APIs require anticipating access patterns; shipping the full database lets clients use SQL joins, aggregation, and windowing locally, while one-way LiteFS replication avoids chatty gossip and is simple to configure with an upstream primary hostname. Moving queries to client hardware removes shared service compute contention and the need for tenant rate limiting and timeouts, but replicas cannot handle mutations and raw schema changes create looser contracts, mitigated by backward-compatible changes and views. Data access is harder to restrict in multi-tenant databases, so per-tenant SQLite files are a suggested workaround. The approach is positioned for specific read-heavy use cases, potentially including local copies of Stripe or GitHub data, with event buses such as Kafka generating per-tenant databases. ### [How the SQLite Virtual Machine Works](https://yomu.fyi/post/how-the-sqlite-virtual-machine-works.md) - Company: [Fly.io](https://yomu.fyi/company/fly-io.md) - Published: Sep 7, 2022 - Type: Explainer SQLite processes SQL through a parse-optimize-execute pipeline that turns query text into executable virtual-machine instructions. It tokenizes characters, parses them into an AST such as a Select struct, and uses table statistics to choose between a full table scan and index-based access. The source describes sqlite\_stat1's coarse index and duplicate-value counts alongside sqlite\_stat4's distributed samples, then traces bytecode that opens the persons table, compares favorite\_color with "blue", and returns matching columns. Registers hold intermediate values, cursors iterate through tables, and sqlite3\_step() resumes execution after each returned row; the VM has 186 opcodes as of writing, while bind parameters allow prepared statements to skip repeated parsing and optimization. ### [7 Ways to Make Your SQL Workshop Beginner-friendly](https://yomu.fyi/post/7-ways-to-make-your-sql-workshop-beginner-friendly.md) - Company: [Shopify](https://yomu.fyi/company/shopify.md) - Author: 2023-10-18 - Published: May 21, 2020 - Type: Tutorial Shopify’s 90-minute SQL workshop serves new RnD hires with varied technical backgrounds, including participants who may have no experience with databases or programming. It separates beginners from intermediate learners and aims to help beginners use data responsibly to discover insights. The beginner format spends its first 30 minutes on data warehouse architecture, data preparation, and privacy, then uses real production data and tools such as Mode Analytics with an accessible dataset. The curriculum identifies core SQL objectives, translates them into practical questions, starts with SELECT \* exploration, and builds queries toward a larger analytical question. It also provides beginner, intermediate, and advanced learning resources covering topics from foundational clauses and aggregations to joins, window functions, CTEs, and Presto-specific techniques. ### [Slow Database Queries? Are Database Types to Blame?](https://yomu.fyi/post/slow-database-queries-are-database-types-to-blame.md) - Company: [Heroku](https://yomu.fyi/company/heroku.md) - Author: Ben Fritsch - Published: Dec 19, 2019 - Type: Problem & solution Slow database queries can stem from schema choices as well as inefficient SQL. The example uses PostgreSQL with a json column holding large payloads, then selects a single url value for an app\_uuid and orders the results. EXPLAIN ANALYZE with BUFFERS showed 7,106 shared buffer hits, representing about 60MB of JSON data loaded and discarded, while execution reached 1,141.296 milliseconds in the reported run. The solution split the JSON into separate text fields for data and url, bringing query time down from about 1,200 milliseconds to 10 milliseconds. The broader conclusion is that schemas should be revisited as application usage becomes better understood, but changes are not necessary when the design isn't causing a problem. [Newer posts](https://yomu.fyi/topic/sql.md) · [Older posts](https://yomu.fyi/topic/sql/page/3.md)