Loading…
Topic
SQL
35 posts about SQL. Every summary links to the original.
- 35 posts
- 13 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Geospatial Unbounded: Spatial SQL GA with AI/BI Maps, Delta Sharing, and Iceberg v3
Databricks announces Spatial SQL as Generally Available, positioning its platform for geospatial analysis without separate spatial databases, warehouses, and mapping tools. It supports native GEOMETRY columns in Delta or Iceberg, more than 90 OGC-compliant ST_* functions, spatial joins, and boolean set operations. AI/BI dashboards can render Geometry and Geography columns as maps, while Genie can generate spatial queries and dashboards and respect Unity Catalog row filters. Geo columns are supported by Delta Sharing, and Databricks can read and write managed Iceberg tables or read externally written Iceberg tables with geospatial types in Iceberg v3. Benchmarks show eight of twelve SpatialBench queries improved since Public Preview, with gains from 20% to 15X, while areal boolean operations are twice as fast on average versus prior versions.
Query Tags: The Context Your Warehouse Queries Have Been Missing
Databricks SQL Query Tags address a visibility gap in warehouse workloads: built-in logs identify users, warehouses, and tools, but not dashboards, projects, teams, or cost centers. Query Tags, now in Public Preview, attach multiple custom key-value pairs to each SQL execution, carry them into the Query History System Table, and expose them in the Query Profile UI. Partner integrations can automatically tag dbt models, while Power BI and Tableau support connection-level tags, and APIs and connectors support connection- or statement-level metadata. Users can also issue SET QUERY_TAGS in SQL Editor, notebooks, dashboards, or alerts to label subsequent session statements. These tags enable SQL or Genie queries for cost allocation, regression analysis, workload filtering, and environment comparisons; future plans include broader connector and workload support plus Query History search.
Automate Data & KPI Monitoring with SQL Alerts
Databricks SQL Alerts is generally available as an automated alternative to manually checking dashboards and rerunning monitoring queries for business KPIs, pipeline health, and data quality. Teams define a SQL query, evaluation condition, schedule, and notification destinations, then receive alerts when results cross configured guardrails; standalone alerts can also run within Lakeflow Jobs. GA supports authoring in the SQL editor, notifications through email, Slack, PagerDuty, Microsoft Teams, or webhooks, and production management through Git, Declarative Automation Bundles, APIs, Terraform, and SDKs. The post’s revenue example compares yesterday’s result with a seven-day average and triggers when revenue_pct_change is below -5%, while the SQL Alert task can expose OK, TRIGGERED, or ERROR to downstream pipeline logic.
Announcing the Databricks analytics engineer learning pathway
Databricks is launching the Analytics Engineer Learning Pathway, a curriculum for SQL practitioners that covers data modeling, pipelines, metrics, and Genie spaces on the lakehouse. The pathway teaches learners to transform raw data into governed, AI-ready semantic models and metric views, the foundation for analytics, dashboards, and AI agents. Courses cover Analytics Fundamentals, production SQL ETL with Materialized Views, Streaming Tables, AUTO CDC, and Lakeflow Jobs, plus data modeling with Delta Lake and Unity Catalog. The curriculum also addresses metric views, Genie spaces, governance with Unity Catalog permissions and ABAC policies, and Spark Declarative Pipelines with expectations, event logs, and metrics. Courses are available in self-paced and instructor-led formats through Databricks Academy, with the full pathway included in active learning subscriptions.
Rethinking SQL ETL for modern data platforms
SQL ETL is often distributed across a warehouse, transformation framework, orchestrator, monitoring, lineage, and data-quality systems, creating operational fragmentation as data teams scale. The post proposes a unified platform model that combines execution, orchestration, observability, governance, and optimization while supporting analytics engineers, warehouse engineers, and analysts through dbt, scripts and stored procedures, Materialized Views, declarative pipelines, and no-code tools. Databricks implements this model with shared execution, governance, and observability, using serverless infrastructure and AI-driven optimization alongside open table formats and ANSI SQL. The stated benefits include reduced operational overhead, support for batch and real-time workloads, and incremental modernization rather than disruptive rewrites. Reported examples include HP’s 32% cloud savings and 36% lower combined job runtime after moving to serverless compute, and Adobe reducing query execution from eight minutes to three seconds with Materialized Views.
Powering KPMG UK Audit's AI future with Databricks
KPMG UK is evolving its Audit data platform with Databricks to support AI-enabled analytics while preserving governance, rigour, and professional standards. The program converges structured data, advanced analytics, and AI on a unified cloud-native Lakehouse, with Databricks SQL replacing core SQL Server workloads and Delta underpinning the platform. Lakebridge assessed migration complexity, while Databricks-hosted large language models including Claude Sonnet and Genie Code helped convert T-SQL, refactor stored procedures, modularize queries, and suggest Delta-based optimizations under engineer review. This reduced refactoring time by around 60% and enabled modernization of more than 400 scripts and stored procedures in roughly three months. Databricks SQL Serverless provides elastic compute for spiky workloads, while Genie offers traceable, version-controlled SQL and Delta Sharing supports governed data exchange.
Unlocking Faster Insights with Experimenter-Defined Segmentations
At Etsy, Catapult supported only 13 hard-coded segmentations, forcing experimenters who needed other cuts to rely on slow, ad-hoc SQL analysis outside the platform. They replaced that fixed model with a Segmentation Library and creation tool where experimenters define reusable segmentations as SQL queries plus join columns connecting segment data to experimental units. Each definition is validated before production, then processed through a configuration-driven pipeline that detects new segmentations and generates independent tasks, improving failure isolation and allowing downstream work to wait only for required results. In the first three months, teams added 34 segmentations, increasing support from 13 to 47, while 78% of experimenters reported saving 2–5 hours monthly through custom analysis in Catapult.
Streamlining RiskOps with the SOP agent framework
Risk Operations teams investigating Account Takeover (ATO) cases must gather data across systems, run SQL queries, and make decisions under pressure, creating delays and error risk. The SOP-driven LLM agent framework represents the workflow as a natural-language, indentation-based tree whose steps specify queries, external functions, and decision criteria. During execution, the SOP planner navigates the tree and produces JSON-formatted instructions, while a Worker Agent invokes APIs or SQL queries and stores results in a feedback loop. The process ends by synthesising investigation data into a report and final decision. In reported results, handling time fell from 22 to 3 minutes per ticket, 87% of ATO cases were automated since launch, and the error rate was zero.
Building Ramp's MCP server
Ramp built an open-source MCP server that exposes its developer API to LLMs through tools and a SQL interface, enabling natural-language access to business spend data. Using FastMCP in Python and Claude Desktop, the prototype could run analyses, generate visualizations, and issue cards on demand, but pagination struggled beyond a few hundred transactions because of miscalculations, context limits, input sizes, and token use. The team replaced bulk responses with an in-memory ETL that flattens API JSON into SQLite, then exposes load, transformation, and query tools; an OLAP-powered API was added for reporting workloads that caused timeouts. This let Claude analyze tens of thousands of spend events with aggregate and window functions while reducing token usage, although API latency, occasional reliability problems, tool-selection errors, and security risks remain; audit logs, OAuth scopes, and constrained tools provide mitigations, while write actions await a safety framework.
Skip the API, Ship Your Database
The post describes an unexpected use for LiteFS, a distributed SQLite database: replacing service-to-service APIs with read-only database replicas. Typical APIs require anticipating access patterns; shipping the full database lets clients use SQL joins, aggregation, and windowing locally, while one-way LiteFS replication avoids chatty gossip and is simple to configure with an upstream primary hostname. Moving queries to client hardware removes shared service compute contention and the need for tenant rate limiting and timeouts, but replicas cannot handle mutations and raw schema changes create looser contracts, mitigated by backward-compatible changes and views. Data access is harder to restrict in multi-tenant databases, so per-tenant SQLite files are a suggested workaround. The approach is positioned for specific read-heavy use cases, potentially including local copies of Stripe or GitHub data, with event buses such as Kafka generating per-tenant databases.
How the SQLite Virtual Machine Works
SQLite processes SQL through a parse-optimize-execute pipeline that turns query text into executable virtual-machine instructions. It tokenizes characters, parses them into an AST such as a Select struct, and uses table statistics to choose between a full table scan and index-based access. The source describes sqlite_stat1's coarse index and duplicate-value counts alongside sqlite_stat4's distributed samples, then traces bytecode that opens the persons table, compares favorite_color with "blue", and returns matching columns. Registers hold intermediate values, cursors iterate through tables, and sqlite3_step() resumes execution after each returned row; the VM has 186 opcodes as of writing, while bind parameters allow prepared statements to skip repeated parsing and optimization.
7 Ways to Make Your SQL Workshop Beginner-friendly
Shopify’s 90-minute SQL workshop serves new RnD hires with varied technical backgrounds, including participants who may have no experience with databases or programming. It separates beginners from intermediate learners and aims to help beginners use data responsibly to discover insights. The beginner format spends its first 30 minutes on data warehouse architecture, data preparation, and privacy, then uses real production data and tools such as Mode Analytics with an accessible dataset. The curriculum identifies core SQL objectives, translates them into practical questions, starts with SELECT * exploration, and builds queries toward a larger analytical question. It also provides beginner, intermediate, and advanced learning resources covering topics from foundational clauses and aggregations to joins, window functions, CTEs, and Presto-specific techniques.
Slow Database Queries? Are Database Types to Blame?
Slow database queries can stem from schema choices as well as inefficient SQL. The example uses PostgreSQL with a json column holding large payloads, then selects a single url value for an app_uuid and orders the results. EXPLAIN ANALYZE with BUFFERS showed 7,106 shared buffer hits, representing about 60MB of JSON data loaded and discarded, while execution reached 1,141.296 milliseconds in the reported run. The solution split the JSON into separate text fields for data and url, bringing query time down from about 1,200 milliseconds to 10 milliseconds. The broader conclusion is that schemas should be revisited as application usage becomes better understood, but changes are not necessary when the design isn't causing a problem.
Dataclips Power Insights at Heroku
Heroku Dataclips let employees create SQL queries through a web GUI against Heroku Postgres databases, then share results through unique URLs or HTTP endpoints returning CSV or JSON. At Heroku, Dataclips reduce dependence on engineers or DBAs by letting departments retrieve fresh production metrics, build dashboards, track product usage, identify deprecated-feature users, and investigate suspicious signup patterns. To make querying safer, the organization runs dataclips against a read-only Postgres follower rather than the leader; writes propagate from leader to followers in close to real-time. Business Operations also teaches SQL and the internal schema through Office Hours, aiming to replace recurring one-off requests with self-service query skills.
Manage Real-time Event Streams and SQL Analytics with Apache Kafka on Heroku
The post presents a Heroku-based system for handling high-volume product-usage events, combining near-real-time visualization with deeper SQL analytics. A Node.js producer sends simulated e-commerce events to an Apache Kafka topic, while a visualization consumer forwards selected data over WebSockets to a D3 stacked stream chart. A second consumer uses a Kafka consumer group, batches events, writes them to Amazon Redshift, and commits offsets after successful processing; Metabase then provides dashboards and ad-hoc SQL queries. The example uses Kafka, Redshift, and Metabase on Heroku, with Terraform described for provisioning Redshift and private connectivity, and notes that the setup incurs AWS and Heroku costs and requires Heroku Enterprise for that configuration. The source says the system can be provisioned on Heroku in 15 minutes.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.