---
title: "Latest reads"
description: "The engineering internet, summarised so you can actually read it."
---

# Latest reads
> The engineering internet, summarised so you can actually read it.

## Articles

### [How lakebase architecture delivers 5x faster Postgres writes](https://yomu.fyi/post/how-lakebase-architecture-delivers-5x-faster-postgres-writes.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: David Wein, Vlad Lazar
- Published: May 7, 2026
- Type: Benchmark

Lakebase’s separation of Postgres compute and distributed storage removes the local-disk torn-page risk that necessitates full page writes (FPW) in traditional deployments. The change disables FPW on compute and moves full page image generation into the pageserver, which creates reset points after a configured number of WAL deltas while preserving bounded read reconstruction. In HammerDB TPROC-C tests, 32-vCPU throughput rose from 95,686 to 439,300 new orders per minute, while average WAL per transaction fell from 58KB to under 4KB, a 94% reduction. Production validation reported WAL reductions from 30MB/s to 1MB/s on a 56-vCPU project, p99 read latency improvements of 30% to 50%, and up to threefold regional storage p99 improvement. The rollout reached all Lakebase Serverless and Neon databases without restarts or customer interruption, using the existing XLOG\_FPW\_CHANGE mechanism.


### [Why talent transformation is the missing focus of enterprise AI](https://yomu.fyi/post/why-talent-transformation-is-the-missing-focus-of-enterprise-ai.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Christy Seto, Pratyarth Rao
- Published: May 7, 2026
- Type: Announcement

Enterprise AI adoption is presented as a talent problem as much as a technology or strategy problem: insufficient worker skills are identified as the biggest barrier to integrating AI into work, while hiring alone cannot fill demand. The proposed response is continuous upskilling for both technical practitioners and line-of-business users, replacing one-time enablement with learning tied to current work and changing platform capabilities. Databricks Academy Pro is an annual, per-seat subscription combining self-paced courses, weekly live reviews, unlimited public instructor-led classes, hands-on labs in hosted Databricks environments, and one certification exam voucher per user each year. Starter, Growth, and Enterprise tiers add private online community access, a custom Academy portal, and a designated Talent Transformation program manager as seats increase. The offering is available today, alongside private instructor-led training and standalone certification vouchers.


### [Public health intelligence shouldn't require a data scientist](https://yomu.fyi/post/public-health-intelligence-shouldn-t-require-a-data-scientist.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Kacey Hertan
- Published: May 7, 2026
- Type: Problem & solution

State, tribal, local, and territorial (STLT) health agencies manage data across surveillance, vital records, Medicaid, WIC, and emergency preparedness systems, but those systems are fragmented and difficult to query together. That limits rapid public health intelligence: questions linking emergency-department visits with pharmacy dispensing, school absenteeism, vaccination, demographic, or geographic data can require epidemiologists to assemble manual queries over weeks, even when decisions require answers within hours. The post presents Databricks Genie as a natural-language interface for querying this environment, backed by a Databricks engine that handles petabyte-scale datasets across real-time streams and historical records. It describes cross-program synthesis, Unity Catalog row- and column-level access controls, HIPAA-compliant governance, traceability to the underlying query, and validation controlled by health experts. Examples include county-level influenza-like illness trends overlaid with vaccination coverage and identifying counties with high opioid overdose rates and low treatment utilization; Genie is described as available today.


### [Mean time to detect is a data access problem](https://yomu.fyi/post/mean-time-to-detect-is-a-data-access-problem.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Taylor Kain
- Published: May 7, 2026
- Type: Problem & solution

Security operations centers measure MTTD, MTTR, false-positive rates, and analyst utilization, yet investigations often stall because analysts assemble evidence across fragmented systems. A single alert may require separate queries for logs, identity records, asset information, prior alerts, and cross-source timelines, making the analyst the integration layer and creating an MTTI bottleneck. The post presents Lakewatch with Databricks Genie as an agentic interface powered by Anthropic Claude models: analysts ask natural-language questions while autonomous agents hunt, summarize, correlate, and reconstruct timelines across security, IT, and business data. It argues that this architecture can reduce investigation work from manual, multi-system workflows to answers in seconds, while retaining analyst-level access controls and governed data access as exploit time has shrunk to 1.3 days.


### [Enhancing Flink deployment with shadow testing](https://yomu.fyi/post/enhancing-flink-deployment-with-shadow-testing.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Tee Long Lang
- Published: May 7, 2026
- Type: Problem & solution

Grab’s Coban data streaming team addresses production-only Apache Flink deployment failures, which can leave an application unavailable for about 10 minutes while the system observes it and rolls it back. Its Shadow Testing stage runs a new version in parallel with the current Main version, replaying production data without affecting Main outputs and allowing behavior and regressions to be assessed before promotion. The Shadow deployment uses a shadow Kubernetes namespace, a shadow-prefixed application name, isShadow=true, a distinct Job ID, isolated sinks, and connector-specific separation such as suffixed Kafka consumer groups and shifted Debezium Server ID ranges. By default, Shadow runs for one hour while its job-manager status is monitored; users then continue to Main deployment, which receives a 10-minute health check and automatic rollback if unhealthy. The framework isolates Kafka and S3 outputs, prefixes metrics and logs for observability, and plans more connector support.


### [Fixing JavaScript observability, one library at a time](https://yomu.fyi/post/fixing-javascript-observability-one-library-at-a-time.md)
- Company: [Sentry](https://yomu.fyi/company/sentry.md)
- Author: Abdelrahman Awad
- Published: May 7, 2026
- Type: Problem & solution

JavaScript application performance monitoring tools rely on monkey-patching via require-in-the-middle and import-in-the-middle, which breaks with ECMAScript Modules, bundlers, and non-Node runtimes. To solve this, an initiative is replacing monkey-patching with Node's built-in diagnostics\_channel TracingChannel API across server-side JavaScript libraries. Under this pattern, libraries publish structured events while monitoring vendors subscribe to them with zero overhead when unlistened. The author leveraged Claude Code workflows alongside direct maintainer communication to draft proposals, implement code, and manage reviews across 44 target packages. Ten libraries including mysql2, node-redis, ioredis, and unjs modules have merged support, while active efforts continue on shared OpenTelemetry mapper registries and remaining ecosystem packages.


### [vLLM V0 to V1: Correctness Before Corrections in RL](https://yomu.fyi/post/vllm-v0-to-v1-correctness-before-corrections-in-rl.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Rafael Pardinas, Ehsan Kamalloo
- Published: May 6, 2026
- Type: Problem & solution

PipelineRL migrated its rollout inference engine from vLLM V0 (0.8.5) to V1 (0.18.1), but trainer-side logprobs, policy ratios, KL, entropy, clip rate, and reward diverged from the V0 reference. The investigation separated semantic mismatch, inference-path mismatch, and objective mismatch, then prioritized backend parity before changing the RL objective. Fixes included processed rollout logprobs, explicit V1 runtime settings disabling prefix caching and async scheduling, a V0-like inflight weight-update path, and an fp32 lm\_head for the final projection. With those changes, the final V1 run tracked the V0 trajectory across the reported metrics, while processed\_logprobs alone did not resolve the training gap.


### [Rethinking Distributed Systems for Serverless Performance and Reliability](https://yomu.fyi/post/rethinking-distributed-systems-for-serverless-performance-and-reliabil.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Aaron Davidson, Roland Fäustlin, Zach Williams
- Published: May 6, 2026
- Type: Explainer

Building serverless compute for Apache Spark requires more than warm machine pools or basic autoscaling because traditional deployments couple user applications to infrastructure, allowing contention and inefficiencies to undermine performance and reliability. The proposed architecture separates these concerns through Spark Connect’s client-server model over gRPC, a gateway that routes workloads using query size, cluster utilization, and latency profile, and an adaptive autoscaler that adjusts capacity horizontally and vertically. Spark Connect isolates user applications from drivers, while the gateway continually re-evaluates placement to reduce interference between workloads. The autoscaler offers Standard and Performance-Optimized modes and can respond to out-of-memory errors by restarting tasks on larger VMs without manual intervention. Reported outcomes include a 99.998% upgrade success rate across more than 4.5 billion workloads, 2–5x faster Unilever pipelines, and operational-cost reductions of 25%.


### [Navigating uncertainty in Amazon's middle-mile network](https://yomu.fyi/post/navigating-uncertainty-in-amazon-apos-s-middle-mile-network.md)
- Company: amazon.science
- Author: Ruth Misener, Hana Ku, Georgios Paschos
- Published: May 6, 2026
- Type: Problem & solution

Amazon's middle-mile logistics network coordinates tens of millions of packages across hundreds of facilities while managing subtle demand variations and major operational disruptions. To solve this mixed-integer optimization challenge without combinatorial explosion, Amazon models routes at coarse time resolutions augmented by precomputed 15-minute timing bounds. The system uses Monte Carlo methods alongside a graph attention network featuring interconnected site graphs and origin-destination graphs to capture complex spatial correlations and flow interdependencies. By generating synthetic flow permutations, planners stress-test candidate network designs against hundreds of plausible scenarios to prioritize routing optionality. This risk-aware modeling approach preserves network flexibility, absorbs severe shocks, and yields potential cost savings while safeguarding delivery promises.


### [Responsible AI Governance: A Practical Framework for Business Leaders](https://yomu.fyi/post/responsible-ai-governance-a-practical-framework-for-business-leaders.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: May 6, 2026
- Type: Explainer

Responsible AI governance is presented as an operational framework for leaders overseeing systems that can produce biased outputs, expose sensitive data, and create regulatory, financial, or reputational harm. It draws on the NIST AI RMF and OECD AI principles, maps to EU AI Act requirements, and uses human dignity, fairness, privacy, accountability, transparency, and security as governance values. The program starts with a living inventory recording purpose, ownership, training-data sources, affected populations, review dates, model lineage, and third-party status, followed by risk classification and assessments based on potential impact. It calls for lifecycle controls including bias mitigation, security testing, human review, drift monitoring, audits, incident exercises, and confidential concern reporting. The roadmap recommends piloting governance on a highest-risk product line, scaling controls across business units, and reviewing the framework annually or after major incidents, regulatory updates, or portfolio changes.


### [Machine Learning Use Cases: Practical Industry Applications](https://yomu.fyi/post/machine-learning-use-cases-practical-industry-applications.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: May 6, 2026
- Type: Explainer

This guide surveys practical machine learning applications across industries, from medical imaging and fraud detection to demand forecasting, document processing, and customer-service automation. It defines supervised, unsupervised, semi-supervised, and reinforcement learning, then relates technique selection to the business question, data type, and availability of labels. Technical coverage includes convolutional neural networks for image analysis, transformers and large language models for generative AI, and time-series workflows using cleaning, lag features, and forecasting. The operational guidance addresses feature stores, experiment tracking, CI/CD, drift monitoring, retraining, cost optimization, fairness, privacy, explainability, and model risk management. Templates and checklists structure projects around business problems, data sources, metrics, architecture, measured outcomes, validation, governance, and escalation, while cited Databricks case studies and tools provide implementation examples.


### [Improved debugging for Expo apps with the React Native SDK](https://yomu.fyi/post/improved-debugging-for-expo-apps-with-the-react-native-sdk.md)
- Company: [Sentry](https://yomu.fyi/company/sentry.md)
- Author: Aleksandr Pantiukhov
- Published: May 6, 2026
- Type: Announcement

Expo applications generate approximately 75% of the total event volume received from React Native applications in Sentry. In response, Sentry updated its React Native SDK starting in version 8.10 to improve Expo debugging and performance monitoring workflows. The update automatically enriches events with Expo constants and over-the-air update context, including channels, runtime versions, and emergency launch warnings when an OTA update fails. Developers can track remote EAS Build lifecycle failures and successes using dedicated hooks configured in package.json. Performance instrumentation now supports span generation for Expo Router route prefetching as well as asset and image loading through wrapper functions around expo-image and expo-asset.


### [Introducing @supabase/server](https://yomu.fyi/post/introducing-supabase-server.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: tomas\_pozo, kalleby\_santos, katerina\_skroumpelou, matt\_johnston
- Published: May 6, 2026
- Type: Announcement

Supabase is releasing @supabase/server in public beta, a package for authentication verification, client setup, request context, and common server-side boilerplate across backend runtimes. Supabase says an anonymous analysis of 25,000 deployed Edge Functions found developers repeatedly rebuilding setup for clients, JWT verification, claims parsing, CORS, auth context, and shared files. The package's withSupabase wrapper or createSupabaseContext function supplies a SupabaseContext containing a user-scoped client, service-role admin client, verified identity, JWT claims, auth metadata, and auth mode, while declarative modes cover user, none, secret, publishable, or combined access. It returns standard Request-to-Response handlers for Edge Functions, Vercel Functions, Cloudflare Workers, Hono, Bun, and Deno, and includes primitives for custom middleware and adapters. @supabase/server does not replace @supabase/ssr, which remains focused on cookie-based sessions, and the release invites feedback on its public-beta API and adapters.


### [Bootstrapping Composer with autoinstall](https://yomu.fyi/post/bootstrapping-composer-with-autoinstall.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Shomil Jain, Joshua Warner & Andrew Zhai
- Published: May 6, 2026
- Type: Explainer

RL training depends on runnable environments, because broken setup can consume tokens and compute without producing a reward signal or can make a task unsolvable. Composer autoinstall addresses this by using earlier Composer models to turn unconfigured repository checkouts into working mock environments. A Cursor agent first proposes 10 commands and expected outputs, while a separate Composer agent configures the checkout, mocks dependencies, and is evaluated against three selected commands. The process can create placeholder files, images, database tables, MinIO configurations, Docker containers, and startup scripts, restarting up to five times before discarding an environment. In a Celo monorepo experiment, the agent installed dependencies including Foundry and eventually mocked a user to run a minimal application; Composer 2 later scored 61.7% on Terminal-Bench versus 47.9% for Composer 1.5.


### [Adding Benchmaxxer Repellant to the Open ASR Leaderboard](https://yomu.fyi/post/adding-benchmaxxer-repellant-to-the-open-asr-leaderboard.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Eric Bezzam, Steven Zheng, Eustache Le Bihan, Sergio Bruccoleri, Jeanine Sinanan-Singh, Casey Ford, Guanbo Wang, Yukai Huang, Ke Li, Yufeng Hao, Liao Xiaoling
- Published: May 6, 2026
- Type: Announcement

The Open ASR Leaderboard is adding private English speech-recognition datasets from Appen Inc. and DataoceanAI to evaluate scripted and conversational speech across several accents. To reduce benchmark-specific optimization (“benchmaxxing”) and test-set contamination, the datasets remain private, while the default Average WER continues to use public datasets only. A toggle exposes the private data, with macroaverages for overall, scripted, conversational, American, and non-American subsets; individual split scores are withheld to discourage provider- or accent-specific optimization. The leaderboard uses a shared Hub dataset, normalized transcripts and outputs, and open-source evaluation code, while private evaluation is run after public-set verification when models are submitted.


### [Agentic analytics starts with query-ready data: the write-side cost of Snowflake vs. ClickHouse](https://yomu.fyi/post/agentic-analytics-starts-with-query-ready-data-the-write-side-cost-of.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Tom Schreiber, Lionel Palacin
- Published: May 6, 2026
- Type: Benchmark

Agentic analytics workflows introduce bursts of concurrent, low-latency analytical queries over continuously ingested data. Analytical query efficiency relies on columnar storage and chunk-level pruning, which require contiguous, sorted data to avoid full-table scans. Rather than evaluating warehouses solely at query runtime, this benchmark compares the write-side cost of maintaining query-ready data in ClickHouse and Snowflake during continuous ingestion at approximately one million rows per second. Snowflake relies on post-write clustering to organize data, whereas ClickHouse constructs sorted layouts on the write path and refines them via background merges. Across 100B, 200B, and 300B row checkpoints on the ClickBench dataset, ClickHouse achieved a query-ready layout at 22 times lower cost and delivered 28 times better write-side cost-performance.


### [The AI scaling gap hiding in digital native companies](https://yomu.fyi/post/the-ai-scaling-gap-hiding-in-digital-native-companies.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Madelyn Mullen
- Published: May 5, 2026
- Type: Explainer

An Economist survey of 1,220+ executives across eight industries finds that digital native companies lead in ambition and breadth of AI deployment, but not in full operational maturity. At 18%, they are the most likely group to prioritize embedding AI across core processes at scale, and nearly 92% report AI ROI ahead of plan. Yet they lead fully embedded AI—defined as use by 100+ users, SLAs, and performance and impact monitoring—in only one of eight functions, R&D/product development; they rank seventh in finance and sixth in operations and supply chain. Telecom, media and entertainment, manufacturing, and energy outperform them in selected functions despite lower stated scaling priority. The proposed response is to build shared, production-grade foundations for data, governance, workloads, models, agents, and applications, with security, lineage, monitoring, and performance measurement treated as reusable capabilities.


### [10 trillion samples a day: Scaling beyond traditional monitoring infra at Databricks](https://yomu.fyi/post/10-trillion-samples-a-day-scaling-beyond-traditional-monitoring-infra.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: David Yuan, Yi Jin, Karan Bavishi, HC Zhu, Joey Beyda
- Published: May 5, 2026
- Type: Problem & solution

Databricks’ monitoring infrastructure now tracks 5 billion active timeseries in real time and ingests more than 10 trillion samples daily, exposing scalability, reliability, cost, and operability limits in its older stack. The replacement, Pantheon, is a fork of CNCF Thanos deployed across more than 160 instances in about 70 regions and three cloud providers; tiered storage, differentiated memory retention, isolated replicated Receive groups, multitenancy, and a custom control plane support automated scaling and recovery. Pantheon’s largest instance holds about 300 million in-memory timeseries and handles nearly 1,000 PromQL queries per second, while migration reduced annual cloud costs by millions and monitoring downtime by roughly five times. For high-cardinality troubleshooting, Hydra preserves raw metrics in Delta tables, exposes them through Grafana and SQL, and unifies metric semantics across aggregated and raw paths, with freshness improvements planned.


### [AI success starts with clean data, not just better models](https://yomu.fyi/post/ai-success-starts-with-clean-data-not-just-better-models.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Aly McGue
- Published: May 5, 2026
- Type: Explainer

Kraken’s data transformation work argues that successful AI depends on clean, unified, accessible data rather than model quality alone. Serving more than 90 million customer accounts across 27 countries, the platform uses Databricks to distribute data securely and at scale, while clients need documentation, join logic and business context to make it useful. Unification reduces the analyst bottleneck, builds trust in shared numbers and enables self-service analytics, including conversational querying through Databricks Genie. The discussion also describes metadata as a live model input: Unity Catalog and Delta Sharing let Kraken share context alongside data instead of relegating it to PDFs or separate web pages. Reported client examples include call-center dashboards updated every few hours with predictive models and faster tariff experimentation, while organizations with stronger data skills and culture are positioned to adopt agentic AI more quickly.


### [How nOps rebuilt their cloud optimization platform on Databricks Lakebase, and why other ISVs should too](https://yomu.fyi/post/how-nops-rebuilt-their-cloud-optimization-platform-on-databricks-lakeb.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Bryan Smith
- Published: May 5, 2026
- Type: Explainer

nOps migrated its production cloud optimization platform to Databricks Lakebase, a fully managed PostgreSQL database integrated with the Databricks Lakehouse. Its previous setup separated customer-facing operational data from analytics, requiring scheduled jobs, cron-based change-detection logic, and synchronization work that introduced lag and operational overhead, especially as workloads expanded to GCP and Azure. In the new architecture, Lakebase is the central Postgres database and single source of truth for the front end and AI infrastructure, while the Lakehouse continuously consumes its data for analytics and metric computation. Databricks Metric Views are automatically surfaced in the application, and data flows one way without direct write-back. The reported results are faster data pipelines, lower operational overhead, and a better customer experience, with auto-scaling, auto-stop, restore, OAuth, and compatibility features supporting adoption.


[Newer posts](https://yomu.fyi/posts/page/132.md) · [Older posts](https://yomu.fyi/posts/page/134.md)