Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,671 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Shipping faster isn't learning faster
Product organizations can ship features in days while taking weeks to understand their behavioral impact, because fragmented analytics stacks depend on analysts, BI expertise, or SQL skills. The post argues that this architectural bottleneck slows the insight-to-ship cycle, causing roadmap decisions to rely on instinct, anecdotes, and lagging indicators. It presents Databricks AI/BI Genie as a conversational interface to event-level behavioral data, with experiment integration, cohort analysis, and product-specific growth-metric definitions. According to the post, product leaders can ask questions in plain language without filing analyst requests, while governed data access supports faster follow-up and feature-impact decisions. It reports that Genie users across 3,300+ Databricks customers cited a 49% productivity gain, 41% faster speed to market, and 5x faster ad-hoc analysis, though these are reported customer results.
Why your OEE dashboard is lying to you
Manufacturing OEE dashboards can look healthy while omitting recurring downtime that supervisors recognize, because relevant facts are split across SCADA, MES logs, maintenance tickets, and shift reports. OEE combines Availability, Performance, and Quality, but extracting those inputs often requires SQL or an analyst, delaying root-cause analysis after throughput drops. Databricks Genie is presented as a conversational AI layer over a unified data platform that leaves MES and SCADA in place while allowing business leaders to ask plain-language questions such as OEE by line against maintenance windows. Its semantic-layer awareness maps terms to actual fields, while governed access and logged, source-cited answers support different visibility levels and traceability. The proposed shift is from static reporting to faster operational questioning about production status, forecast risk, line performance, and quality signals.
The turbine that tried to tell you it was failing
Gas turbines produce millions of daily readings—vibration, temperature, pressure, flow rates, and electrical output—but maintenance teams may learn about warning signals only when an unplanned outage occurs. The post argues that predictive maintenance has struggled less because of model capability than because decision-makers lack fluid access to model findings and operational context. Databricks Genie provides a conversational interface to sensor, maintenance, dispatch, cost, and regulatory data, allowing questions about elevated vibration against maintenance baselines and timing maintenance around outage cycles. It is presented as decision support, not automation, with time-series analysis, maintenance history, integrated costs, and compliance context helping asset managers investigate fleets and act with more confidence.
Predicting readmissions isn't enough. Acting in time is.
Readmission risk models can identify patients likely to return within 30 days, but prediction alone does not ensure timely intervention. In large health systems, risk information may remain in population-health dashboards or EHR flags without reaching the care coordinator with enough context to create an effective post-discharge plan. The post presents Databricks Genie as a natural-language interface for governed clinical and outcomes data, allowing leaders to query readmission rates and compare periods while combining EHR, operational, financial, risk-score, intervention, and outcome information. Genie operates within Unity Catalog governance, with access controls, audit logging, and de-identification policies, and is described as supporting EHR integration and clinical taxonomy awareness; the intended result is to shorten the path from prediction to action.
Clinical trials run longer than they have to. That's a patient problem.
Clinical trial operations can lose weeks because site-level performance data is reviewed with a 2–4 week lag: enrollment velocity is monthly, while protocol deviation rates are assessed quarterly. The post presents Databricks Genie as a natural-language interface to unified trial data, allowing clinical operations leaders to query enrollment, screen-failure, protocol-deviation, query-response, and data-entry metrics across sites. Its stated capabilities include automatic site comparison, integration across CTMS, EDC, safety databases, and site-performance data, plus protocol-aware reasoning and traceability to source records for GCP documentation. The example query identifies Phase II oncology sites with screen-failure rates above 40% over 60 days and compares enrollment pace with activation targets, positioning earlier detection as a way to reduce timeline impact and speed treatment access.
Network quality is a revenue problem, not a technical one
Telecommunications network operations centers monitor extensive performance telemetry, but the source describes a gap between technical network events and the customers, contracts, and revenue they affect. A degraded tower serving 12,000 postpaid customers averaging $85 per month presents a different business problem from one affecting 12,000 prepaid customers with high churn propensity, while enterprise SLA exposure can remain hidden from the NOC. Databricks Genie connects network elements, serving areas, customer records, contract terms, SLA thresholds, and churn-risk scores so leaders can query commercial impact in seconds. It also supports proactive alerts based on defined commercial thresholds, enabling restoration and investment decisions to account for customer impact rather than technical severity alone.
Shelf availability starts with better demand visibility
Retail out-of-stock rates in grocery and general merchandise typically run between 7% and 10%, leaving roughly one in ten sought-after items unavailable at a given moment. The revenue impact is real, but repeated shelf gaps can cause customers to build shopping habits elsewhere. Modern retail replenishment requires real-time synthesis of POS velocity by store and SKU, distribution-center inventory, on-order quantities and delivery windows, supplier fill-rate histories, promotional calendars, and other demand signals. Databricks Genie is presented as a conversational interface to the full inventory and demand environment, allowing leaders to ask which high-velocity SKUs are projected to stock out within 72 hours and receive current on-order positions in seconds rather than hours. Its stated differentiators include store-SKU granularity, shared promotional, event, and weather data, supplier performance context, and cross-distribution-center rebalancing; the post says Genie is available today.
When predicting the next hit requires more than intuition
Content investment decisions in entertainment commit substantial funds despite limited information, because leaders often combine executive instinct, competitive benchmarks, and historical performance data. Although companies hold viewing and subscriber signals such as episode completion, skip patterns, genre performance by demographic, acquisition, and retention, those insights are frequently buried in dashboards and spreadsheets. The post presents Databricks Genie as a natural-language interface that lets content leaders query governed performance data directly, including comparisons linking content types with 90-day retention, subscriber demographics, lifetime value, acquisition source, and historical greenlights. It argues that current, conversational access can complement creative judgment by reducing dependence on delayed analyst recommendations, while external market data can add competitive context.
Data Mesh at Grab (Part II): The foundational tools behind certification
Grab’s Signals Marketplace uses three foundational platforms to make data certification operational across its data mesh: Hubble, Genchi, and the Data Contract Registry. Hubble, built on DataHub, catalogs assets, ownership, documentation, lineage, contracts, and certification status, while its event-driven engine recalculates certification as metadata and upstream health change. Genchi provides self-service observability and tests for freshness, volume, schema, completeness, and business checks; the registry manages explicit expectations between producers and consumers. Together, these signals let Hubble classify assets as Uncertified, Certified, CertifiedPlus, or Revoked, giving consumers a workflow centered on certified data rather than hearsay. The post reports that P80 datasets, the tables accounting for 80% of queries, fell by over 58% in one year after the Signals Marketplace campaign began in 2024, and positions the foundation as support for AI agents and automated analytics.
Welcoming Diel Duarte, our new Open source Engineer
Resend introduces Diel Duarte as its new open source engineer, bringing more than 13 years of experience building web applications and SDKs. His focus is developer experience, especially API design with clean ergonomics that lets developers concentrate on building rather than wrestling with interfaces. Diel describes a path from studying programming and computer fundamentals at Cotemig to following Resend as a user and joining after an interview process he found unusually human; he cites Resend’s DX and craft, along with Zeno’s open source work, as major references. Outside work, he draws inspiration from deliberately curated feeds, new languages and APIs, visual-design communities, and creator friends, while keeping music and interior design as interests; he names Warp Terminal as his favorite tool and Japan as a memorable destination.
Approximate answers, exact decisions: New sketch functions for analytics
Databricks now supports four sketch function families, built on Apache DataSketches, for replacing expensive exact percentiles, distinct counts, set operations, top-K rankings, and metric aggregations with bounded-memory approximations. KLL sketches answer quantiles, Theta sketches support unions, intersections, and differences, approximate top-K sketches track frequent items, and Tuple sketches combine distinct counting with aggregated metrics such as customer revenue. The sketches can be built during ETL, stored as columns in Delta tables, and merged on read, allowing dashboards and streaming pipelines to avoid rescanning raw data; SQL, DataFrame, and Structured Streaming pipelines are supported. The stated trade-off is configurable 1–2% relative error, with the source citing a 1000x speedup for applicable workloads. Exact methods remain appropriate for financial auditing, compliance reporting, and other precision-required uses.
Companies winning with AI built the data layer first
Trinity Industries’ experience is presented as a case for treating the data layer, rather than models or dashboards, as the foundation of enterprise AI. The railcar manufacturer migrated 95% of its enterprise data to a single Databricks lakehouse, adopted Medallion architecture, moved transformations upstream, and retired legacy dashboards containing nearly 600 measures. That consolidation supports streaming ETA predictions, procurement agents, and Genie conversational analytics: the ETA model is reported as 50% more accurate than industry ETAs, agents helped increase on-time material delivery by 15%, and Genie handles more than 1,000 questions monthly. The migration took close to a year, followed by six to eight months of additional work, but the account argues that trusted, standardized data enables experimentation, automation, and broader employee access to analysis.
Massachusetts Driver Benefits
In June 2024, Uber reached an agreement with Massachusetts Attorney General Andrea Campbell that adds protections and benefits for independent Massachusetts Uber Rides drivers. For Rides trips beginning in Massachusetts, drivers are guaranteed $34.48 per hour of active time, calculated over two-week periods, with supplemental payments when trip fares and included incentives fall short. Beginning November 1, 2024, drivers can accrue up to 40 hours of paid sick leave annually at $21.22 per hour, subject to the stated accrual and claiming rules. The post also describes occupational accident insurance, Paid Family Medical Leave payments, a portable health fund with quarterly eligibility, and account-deactivation appeals. It explains how active time, sick leave, guaranteed earnings, and health-fund participation are viewed or claimed through Uber’s systems and related administrators.
Rethinking SQL ETL for modern data platforms
SQL ETL is often distributed across a warehouse, transformation framework, orchestrator, monitoring, lineage, and data-quality systems, creating operational fragmentation as data teams scale. The post proposes a unified platform model that combines execution, orchestration, observability, governance, and optimization while supporting analytics engineers, warehouse engineers, and analysts through dbt, scripts and stored procedures, Materialized Views, declarative pipelines, and no-code tools. Databricks implements this model with shared execution, governance, and observability, using serverless infrastructure and AI-driven optimization alongside open table formats and ANSI SQL. The stated benefits include reduced operational overhead, support for batch and real-time workloads, and incremental modernization rather than disruptive rewrites. Reported examples include HP’s 32% cloud savings and 36% lower combined job runtime after moving to serverless compute, and Adobe reducing query execution from eight minutes to three seconds with Materialized Views.
Stripe data now available on Databricks via Databricks Marketplace
Stripe Data Pipeline is now available on Databricks Marketplace through Delta Sharing, allowing payment and business data to be analyzed in a Databricks workspace without ETL. Stripe data—including transaction records, customer histories, subscriptions, refunds, and payouts—remains in Stripe’s infrastructure and is queried directly through Unity Catalog, where it can be joined with other tables. The approach replaces polling integrations and custom jobs, avoiding per-call charges, connector licensing fees, data duplication, and maintenance code while providing row- and column-level access controls, audit trails, and compliance features. The shared data is positioned for AI-native uses such as continuous transaction monitoring, churn scoring, LLM-powered retention workflows, Genie queries, and finance analytics apps. Getting started involves visiting the Stripe Data Pipeline listing on Databricks Marketplace.
Databricks and Stripe projects: Infrastructure built for agents
The post announces Databricks as a launch partner for Stripe Projects, an agent-first CLI that lets AI agents discover, provision, and pay for Neon databases without human interaction. It addresses the manual work still required to provision services, navigate interfaces, configure accounts, and enter payment details, which can slow autonomous app development. With Lakebase architecture, Neon can provide production-ready Postgres in under 350ms, while decoupled compute and storage support rapid creation and teardown of OLTP databases. Serverless scaling adjusts compute to traffic and can scale to zero, while zero-copy branching creates isolated production-data branches for testing, migrations, and prompt experiments without risking the primary environment. The post also notes Databricks’ Stripe Data Pipeline is available in the Databricks Marketplace for analyzing Stripe data without ETL.
Granite 4.1 LLMs: How They’re Built
Granite 4.1 is a family of dense decoder-only language models in 3B, 8B, and 30B sizes, trained on approximately 15T tokens. Its five-phase pre-training pipeline moves from broad web-scale data to progressively curated mixtures, increases math and code emphasis, and extends context from 4K to 512K tokens through staged long-context training. Supervised fine-tuning uses about 4.1M curated samples evaluated with LLM-as-Judge and rule-based checks, while reinforcement learning uses on-policy GRPO with DAPO loss. The 8B instruct model matches or surpasses the earlier Granite 4.0-H-Small 32B-A9B MoE model despite its simpler dense architecture and fewer parameters, and the Granite 4.1 models are released under Apache 2.0.
Agents are ready, but your architecture probably isn't
Enterprise AI initiatives often produce activity rather than value because organizations start with technology instead of a defined outcome and overlook data architecture, governance, and semantic context. Agentic systems add risk when they can send messages, update records, place orders, or delete records, making permissions and situational controls essential. The discussion argues that dashboards and batch pipelines are poorly matched to low-latency, high-scale agent workloads, which require transactional infrastructure alongside existing analytics. Lakebase is presented as that transactional foundation, while AgentBricks, Databricks Apps, and Genie provide agent development and monitoring, application delivery, and conversational data access. The recommended path is to define success first, isolate a focused pilot, learn what works, and redesign underlying processes rather than merely add AI to them.
Interoperability between Unity Catalog and Google BigQuery via catalog federation
Databricks and Google Cloud announce bidirectional catalog federation that lets customers access the same data from Unity Catalog or BigQuery without duplication. The interoperability uses Apache Iceberg REST catalog APIs, enabling both platforms to read and write the same tables through a common open interface. Google Cloud's catalog federation is in preview for reading Unity Catalog-managed tables from BigQuery, while Google Cloud's Lakehouse federation is in private preview for Databricks users to govern and read foreign Iceberg tables. Unity Catalog's policies, fine-grained access controls, and lineage tracking apply to federated data, and users can query it with Genie using natural language. The announcement positions open standards as a path to interoperability across data platforms and previews future catalog metadata exchange through Iceberg REST catalog APIs.
Build programmatic agents with the Cursor SDK
Cursor’s SDK is now in public beta, letting developers build programmatic coding agents with the runtime, harness, and models used by Cursor’s desktop app, CLI, and web app through TypeScript. Agents can run locally, on Cursor’s cloud in dedicated VMs, or on self-hosted workers; cloud sessions include a repository clone, configured development environment, streaming, reconnection, and options to push branches or open pull requests. The SDK also provides codebase indexing, semantic search, grep, MCP servers, repository skills, hooks, and named subagents, while supporting every model available in Cursor, including Composer 2.5. Described uses include CI/CD automations, custom internal platforms, and customer-facing agent experiences, with starter projects for quickstarts, prototyping, kanban workflows, and a CLI; the SDK uses standard token-based consumption pricing.
Top topics
- Architecture834
- Performance816
- LLMs660
- Machine Learning622
- Open Source563
- AI Agents553
- AI470
- Developer Experience448
Weekly digest
The best engineering reads, once a week.