Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,668 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Enhancing Flink deployment with shadow testing
Grab’s Coban data streaming team addresses production-only Apache Flink deployment failures, which can leave an application unavailable for about 10 minutes while the system observes it and rolls it back. Its Shadow Testing stage runs a new version in parallel with the current Main version, replaying production data without affecting Main outputs and allowing behavior and regressions to be assessed before promotion. The Shadow deployment uses a shadow Kubernetes namespace, a shadow-prefixed application name, isShadow=true, a distinct Job ID, isolated sinks, and connector-specific separation such as suffixed Kafka consumer groups and shifted Debezium Server ID ranges. By default, Shadow runs for one hour while its job-manager status is monitored; users then continue to Main deployment, which receives a 10-minute health check and automatic rollback if unhealthy. The framework isolates Kafka and S3 outputs, prefixes metrics and logs for observability, and plans more connector support.
Fixing JavaScript observability, one library at a time
JavaScript application performance monitoring tools rely on monkey-patching via require-in-the-middle and import-in-the-middle, which breaks with ECMAScript Modules, bundlers, and non-Node runtimes. To solve this, an initiative is replacing monkey-patching with Node's built-in diagnostics_channel TracingChannel API across server-side JavaScript libraries. Under this pattern, libraries publish structured events while monitoring vendors subscribe to them with zero overhead when unlistened. The author leveraged Claude Code workflows alongside direct maintainer communication to draft proposals, implement code, and manage reviews across 44 target packages. Ten libraries including mysql2, node-redis, ioredis, and unjs modules have merged support, while active efforts continue on shared OpenTelemetry mapper registries and remaining ecosystem packages.
vLLM V0 to V1: Correctness Before Corrections in RL
PipelineRL migrated its rollout inference engine from vLLM V0 (0.8.5) to V1 (0.18.1), but trainer-side logprobs, policy ratios, KL, entropy, clip rate, and reward diverged from the V0 reference. The investigation separated semantic mismatch, inference-path mismatch, and objective mismatch, then prioritized backend parity before changing the RL objective. Fixes included processed rollout logprobs, explicit V1 runtime settings disabling prefix caching and async scheduling, a V0-like inflight weight-update path, and an fp32 lm_head for the final projection. With those changes, the final V1 run tracked the V0 trajectory across the reported metrics, while processed_logprobs alone did not resolve the training gap.
Rethinking Distributed Systems for Serverless Performance and Reliability
Building serverless compute for Apache Spark requires more than warm machine pools or basic autoscaling because traditional deployments couple user applications to infrastructure, allowing contention and inefficiencies to undermine performance and reliability. The proposed architecture separates these concerns through Spark Connect’s client-server model over gRPC, a gateway that routes workloads using query size, cluster utilization, and latency profile, and an adaptive autoscaler that adjusts capacity horizontally and vertically. Spark Connect isolates user applications from drivers, while the gateway continually re-evaluates placement to reduce interference between workloads. The autoscaler offers Standard and Performance-Optimized modes and can respond to out-of-memory errors by restarting tasks on larger VMs without manual intervention. Reported outcomes include a 99.998% upgrade success rate across more than 4.5 billion workloads, 2–5x faster Unilever pipelines, and operational-cost reductions of 25%.
amazon.science
Navigating uncertainty in Amazon's middle-mile network
Amazon's middle-mile logistics network coordinates tens of millions of packages across hundreds of facilities while managing subtle demand variations and major operational disruptions. To solve this mixed-integer optimization challenge without combinatorial explosion, Amazon models routes at coarse time resolutions augmented by precomputed 15-minute timing bounds. The system uses Monte Carlo methods alongside a graph attention network featuring interconnected site graphs and origin-destination graphs to capture complex spatial correlations and flow interdependencies. By generating synthetic flow permutations, planners stress-test candidate network designs against hundreds of plausible scenarios to prioritize routing optionality. This risk-aware modeling approach preserves network flexibility, absorbs severe shocks, and yields potential cost savings while safeguarding delivery promises.
Responsible AI Governance: A Practical Framework for Business Leaders
Responsible AI governance is presented as an operational framework for leaders overseeing systems that can produce biased outputs, expose sensitive data, and create regulatory, financial, or reputational harm. It draws on the NIST AI RMF and OECD AI principles, maps to EU AI Act requirements, and uses human dignity, fairness, privacy, accountability, transparency, and security as governance values. The program starts with a living inventory recording purpose, ownership, training-data sources, affected populations, review dates, model lineage, and third-party status, followed by risk classification and assessments based on potential impact. It calls for lifecycle controls including bias mitigation, security testing, human review, drift monitoring, audits, incident exercises, and confidential concern reporting. The roadmap recommends piloting governance on a highest-risk product line, scaling controls across business units, and reviewing the framework annually or after major incidents, regulatory updates, or portfolio changes.
Machine Learning Use Cases: Practical Industry Applications
This guide surveys practical machine learning applications across industries, from medical imaging and fraud detection to demand forecasting, document processing, and customer-service automation. It defines supervised, unsupervised, semi-supervised, and reinforcement learning, then relates technique selection to the business question, data type, and availability of labels. Technical coverage includes convolutional neural networks for image analysis, transformers and large language models for generative AI, and time-series workflows using cleaning, lag features, and forecasting. The operational guidance addresses feature stores, experiment tracking, CI/CD, drift monitoring, retraining, cost optimization, fairness, privacy, explainability, and model risk management. Templates and checklists structure projects around business problems, data sources, metrics, architecture, measured outcomes, validation, governance, and escalation, while cited Databricks case studies and tools provide implementation examples.
Improved debugging for Expo apps with the React Native SDK
Expo applications generate approximately 75% of the total event volume received from React Native applications in Sentry. In response, Sentry updated its React Native SDK starting in version 8.10 to improve Expo debugging and performance monitoring workflows. The update automatically enriches events with Expo constants and over-the-air update context, including channels, runtime versions, and emergency launch warnings when an OTA update fails. Developers can track remote EAS Build lifecycle failures and successes using dedicated hooks configured in package.json. Performance instrumentation now supports span generation for Expo Router route prefetching as well as asset and image loading through wrapper functions around expo-image and expo-asset.
Introducing @supabase/server
Supabase is releasing @supabase/server in public beta, a package for authentication verification, client setup, request context, and common server-side boilerplate across backend runtimes. Supabase says an anonymous analysis of 25,000 deployed Edge Functions found developers repeatedly rebuilding setup for clients, JWT verification, claims parsing, CORS, auth context, and shared files. The package's withSupabase wrapper or createSupabaseContext function supplies a SupabaseContext containing a user-scoped client, service-role admin client, verified identity, JWT claims, auth metadata, and auth mode, while declarative modes cover user, none, secret, publishable, or combined access. It returns standard Request-to-Response handlers for Edge Functions, Vercel Functions, Cloudflare Workers, Hono, Bun, and Deno, and includes primitives for custom middleware and adapters. @supabase/server does not replace @supabase/ssr, which remains focused on cookie-based sessions, and the release invites feedback on its public-beta API and adapters.
Bootstrapping Composer with autoinstall
RL training depends on runnable environments, because broken setup can consume tokens and compute without producing a reward signal or can make a task unsolvable. Composer autoinstall addresses this by using earlier Composer models to turn unconfigured repository checkouts into working mock environments. A Cursor agent first proposes 10 commands and expected outputs, while a separate Composer agent configures the checkout, mocks dependencies, and is evaluated against three selected commands. The process can create placeholder files, images, database tables, MinIO configurations, Docker containers, and startup scripts, restarting up to five times before discarding an environment. In a Celo monorepo experiment, the agent installed dependencies including Foundry and eventually mocked a user to run a minimal application; Composer 2 later scored 61.7% on Terminal-Bench versus 47.9% for Composer 1.5.
Adding Benchmaxxer Repellant to the Open ASR Leaderboard
The Open ASR Leaderboard is adding private English speech-recognition datasets from Appen Inc. and DataoceanAI to evaluate scripted and conversational speech across several accents. To reduce benchmark-specific optimization (“benchmaxxing”) and test-set contamination, the datasets remain private, while the default Average WER continues to use public datasets only. A toggle exposes the private data, with macroaverages for overall, scripted, conversational, American, and non-American subsets; individual split scores are withheld to discourage provider- or accent-specific optimization. The leaderboard uses a shared Hub dataset, normalized transcripts and outputs, and open-source evaluation code, while private evaluation is run after public-set verification when models are submitted.
Agentic analytics starts with query-ready data: the write-side cost of Snowflake vs. ClickHouse
Agentic analytics workflows introduce bursts of concurrent, low-latency analytical queries over continuously ingested data. Analytical query efficiency relies on columnar storage and chunk-level pruning, which require contiguous, sorted data to avoid full-table scans. Rather than evaluating warehouses solely at query runtime, this benchmark compares the write-side cost of maintaining query-ready data in ClickHouse and Snowflake during continuous ingestion at approximately one million rows per second. Snowflake relies on post-write clustering to organize data, whereas ClickHouse constructs sorted layouts on the write path and refines them via background merges. Across 100B, 200B, and 300B row checkpoints on the ClickBench dataset, ClickHouse achieved a query-ready layout at 22 times lower cost and delivered 28 times better write-side cost-performance.
The AI scaling gap hiding in digital native companies
An Economist survey of 1,220+ executives across eight industries finds that digital native companies lead in ambition and breadth of AI deployment, but not in full operational maturity. At 18%, they are the most likely group to prioritize embedding AI across core processes at scale, and nearly 92% report AI ROI ahead of plan. Yet they lead fully embedded AI—defined as use by 100+ users, SLAs, and performance and impact monitoring—in only one of eight functions, R&D/product development; they rank seventh in finance and sixth in operations and supply chain. Telecom, media and entertainment, manufacturing, and energy outperform them in selected functions despite lower stated scaling priority. The proposed response is to build shared, production-grade foundations for data, governance, workloads, models, agents, and applications, with security, lineage, monitoring, and performance measurement treated as reusable capabilities.
10 trillion samples a day: Scaling beyond traditional monitoring infra at Databricks
Databricks’ monitoring infrastructure now tracks 5 billion active timeseries in real time and ingests more than 10 trillion samples daily, exposing scalability, reliability, cost, and operability limits in its older stack. The replacement, Pantheon, is a fork of CNCF Thanos deployed across more than 160 instances in about 70 regions and three cloud providers; tiered storage, differentiated memory retention, isolated replicated Receive groups, multitenancy, and a custom control plane support automated scaling and recovery. Pantheon’s largest instance holds about 300 million in-memory timeseries and handles nearly 1,000 PromQL queries per second, while migration reduced annual cloud costs by millions and monitoring downtime by roughly five times. For high-cardinality troubleshooting, Hydra preserves raw metrics in Delta tables, exposes them through Grafana and SQL, and unifies metric semantics across aggregated and raw paths, with freshness improvements planned.
AI success starts with clean data, not just better models
Kraken’s data transformation work argues that successful AI depends on clean, unified, accessible data rather than model quality alone. Serving more than 90 million customer accounts across 27 countries, the platform uses Databricks to distribute data securely and at scale, while clients need documentation, join logic and business context to make it useful. Unification reduces the analyst bottleneck, builds trust in shared numbers and enables self-service analytics, including conversational querying through Databricks Genie. The discussion also describes metadata as a live model input: Unity Catalog and Delta Sharing let Kraken share context alongside data instead of relegating it to PDFs or separate web pages. Reported client examples include call-center dashboards updated every few hours with predictive models and faster tariff experimentation, while organizations with stronger data skills and culture are positioned to adopt agentic AI more quickly.
How nOps rebuilt their cloud optimization platform on Databricks Lakebase, and why other ISVs should too
nOps migrated its production cloud optimization platform to Databricks Lakebase, a fully managed PostgreSQL database integrated with the Databricks Lakehouse. Its previous setup separated customer-facing operational data from analytics, requiring scheduled jobs, cron-based change-detection logic, and synchronization work that introduced lag and operational overhead, especially as workloads expanded to GCP and Azure. In the new architecture, Lakebase is the central Postgres database and single source of truth for the front end and AI infrastructure, while the Lakehouse continuously consumes its data for analytics and metric computation. Databricks Metric Views are automatically surfaced in the application, and data flows one way without direct write-back. The reported results are faster data pipelines, lower operational overhead, and a better customer experience, with auto-scaling, auto-stop, restore, OAuth, and compatibility features supporting adoption.
From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines
Slack's data platform accumulated over 700 SSH-based Airflow operators executing jobs directly on AWS Elastic MapReduce clusters, creating broad security surfaces and operational instability. Stateful SSH connections frequently failed when Kubernetes pods restarted, leaving orphaned zombie processes and preventing infrastructure modernization such as migrating EMR clusters to child accounts. To eliminate SSH access, Slack adopted a REST-based architecture using Quarry, a gateway providing unified REST APIs and token authentication across compute engines. For non-Hadoop CLI workloads, Slack leveraged YARN Distributed Shell to execute arbitrary shell scripts stored in Amazon S3 within managed, isolated YARN containers. The phased migration achieved zero downtime across eight data regions, unlocking reliable monitoring, simplified onboarding, and future initiatives like Spark on Kubernetes.
amazon.science
How mechanism design theory helps optimize Amazon-vendor collaboration
Amazon and its vendors face an asymmetric information problem when coordinating purchase orders across fulfillment centers because neither party shares proprietary cost, capacity, or demand data. To solve this coordination gap without exposing sensitive information, Amazon's Supply Chain Optimization Technologies organization designed Flo Pro by integrating the Vickrey-Clarke-Groves mechanism with the Consensus Planning Protocol. The distributed optimization framework uses the alternating-direction method of multipliers to coordinate iterative agent responses, calculating cost-benefit transfers across rolling six-week planning horizons. In lower-dimensional settings, Amazon also evaluated a transparent menu-of-contracts approach where vendors select optimal plans priced at Amazon's incurred costs. A nine-week pilot with a consumer-product manufacturer confirmed that Flo Pro successfully aligns incentives and achieves measurable cost savings.
Peril predicts: Precision payouts for a volatile world
Parametric insurance pays automatically when objective thresholds—such as wind speed, rainfall, or earthquake magnitude—are met, replacing lengthy loss assessments with event-based payouts. Modern catastrophe modeling combines geospatial data, weather observations, engineering insights, and historical loss records to estimate extreme-event probability and impact and define reliable triggers. Operationalizing these programs requires near-real-time processing of satellite imagery, weather feeds, exposure datasets, and model outputs. Databricks’ Geospatial Lakehouse unifies those sources on Delta Lake while Spark runs spatial joins and catastrophe modeling pipelines. When thresholds are crossed, the system identifies eligible policies, calculates tiered payouts, and surfaces results through dashboards, Lakehouse Apps, and Genie; aerial imagery and multimodal AI can support damage validation and fraud detection, while Unity Catalog governs access and Delta Sharing supports controlled data exchange.
Realtime or Pipelines? How to choose the right tool
Supabase Realtime and Supabase Pipelines both read PostgreSQL changes through logical replication, but they target different jobs. Realtime delivers live updates and communication to browsers and apps through Broadcast, Presence, and Postgres Changes; its Postgres Changes path checks Row Level Security for subscribed users, uses temporary replication slots, and offers best-effort delivery without replay after disconnects. Pipelines is a managed change-data-capture system that performs an initial sync, then replicates selected publication data in batches to supported analytical destinations through a persistent replication slot, retaining progress and retrying transient failures. The choice depends on the destination and delivery needs: use Realtime for low-latency user experiences that can tolerate gaps, Pipelines for durable analytical copies with at-least-once processing and idempotent destination writes, and both when an application needs both capabilities.
Top topics
- Architecture834
- Performance815
- LLMs659
- Machine Learning621
- Open Source563
- AI Agents553
- AI469
- Developer Experience448
Weekly digest
The best engineering reads, once a week.