---
title: "Latest reads"
description: "The engineering internet, summarised so you can actually read it."
---

# Latest reads
> The engineering internet, summarised so you can actually read it.

## Articles

### [From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines](https://yomu.fyi/post/from-ssh-to-rest-a-security-driven-modernization-of-slack-s-emr-data-p.md)
- Company: [Slack](https://yomu.fyi/company/slack.md)
- Author: Mahendran Vasagam
- Published: May 5, 2026
- Type: Problem & solution

Slack's data platform accumulated over 700 SSH-based Airflow operators executing jobs directly on AWS Elastic MapReduce clusters, creating broad security surfaces and operational instability. Stateful SSH connections frequently failed when Kubernetes pods restarted, leaving orphaned zombie processes and preventing infrastructure modernization such as migrating EMR clusters to child accounts. To eliminate SSH access, Slack adopted a REST-based architecture using Quarry, a gateway providing unified REST APIs and token authentication across compute engines. For non-Hadoop CLI workloads, Slack leveraged YARN Distributed Shell to execute arbitrary shell scripts stored in Amazon S3 within managed, isolated YARN containers. The phased migration achieved zero downtime across eight data regions, unlocking reliable monitoring, simplified onboarding, and future initiatives like Spark on Kubernetes.


### [How mechanism design theory helps optimize Amazon-vendor collaboration](https://yomu.fyi/post/how-mechanism-design-theory-helps-optimize-amazon-vendor-collaboration.md)
- Company: amazon.science
- Author: Dirk Bergemann
- Published: May 5, 2026
- Type: Problem & solution

Amazon and its vendors face an asymmetric information problem when coordinating purchase orders across fulfillment centers because neither party shares proprietary cost, capacity, or demand data. To solve this coordination gap without exposing sensitive information, Amazon's Supply Chain Optimization Technologies organization designed Flo Pro by integrating the Vickrey-Clarke-Groves mechanism with the Consensus Planning Protocol. The distributed optimization framework uses the alternating-direction method of multipliers to coordinate iterative agent responses, calculating cost-benefit transfers across rolling six-week planning horizons. In lower-dimensional settings, Amazon also evaluated a transparent menu-of-contracts approach where vendors select optimal plans priced at Amazon's incurred costs. A nine-week pilot with a consumer-product manufacturer confirmed that Flo Pro successfully aligns incentives and achieves measurable cost savings.


### [Peril predicts: Precision payouts for a volatile world](https://yomu.fyi/post/peril-predicts-precision-payouts-for-a-volatile-world.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Anindita Mahapatra, Timo Roest, Justin Monaldo
- Published: May 5, 2026
- Type: Explainer

Parametric insurance pays automatically when objective thresholds—such as wind speed, rainfall, or earthquake magnitude—are met, replacing lengthy loss assessments with event-based payouts. Modern catastrophe modeling combines geospatial data, weather observations, engineering insights, and historical loss records to estimate extreme-event probability and impact and define reliable triggers. Operationalizing these programs requires near-real-time processing of satellite imagery, weather feeds, exposure datasets, and model outputs. Databricks’ Geospatial Lakehouse unifies those sources on Delta Lake while Spark runs spatial joins and catastrophe modeling pipelines. When thresholds are crossed, the system identifies eligible policies, calculates tiered payouts, and surfaces results through dashboards, Lakehouse Apps, and Genie; aerial imagery and multimodal AI can support damage validation and fraud detection, while Unity Catalog governs access and Delta Sharing supports controlled data exchange.


### [Realtime or Pipelines? How to choose the right tool](https://yomu.fyi/post/realtime-or-pipelines-how-to-choose-the-right-tool.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: eduardo\_gurgel, riccardo\_busetti
- Published: May 5, 2026
- Type: Explainer

Supabase Realtime and Supabase Pipelines both read PostgreSQL changes through logical replication, but they target different jobs. Realtime delivers live updates and communication to browsers and apps through Broadcast, Presence, and Postgres Changes; its Postgres Changes path checks Row Level Security for subscribed users, uses temporary replication slots, and offers best-effort delivery without replay after disconnects. Pipelines is a managed change-data-capture system that performs an initial sync, then replicates selected publication data in batches to supported analytical destinations through a persistent replication slot, retaining progress and retrying transient failures. The choice depends on the destination and delivery needs: use Realtime for low-latency user experiences that can tolerate gaps, Pipelines for durable analytical copies with at-least-once processing and idempotent destination writes, and both when an application needs both capabilities.


### [Stop guessing in production: Full fidelity tracing at scale with ClickHouse and Odigos](https://yomu.fyi/post/stop-guessing-in-production-full-fidelity-tracing-at-scale-with-clickh.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Will Searle
- Published: May 5, 2026
- Type: Problem & solution

Distributed tracing frequently suffers from visibility gaps, broken context across asynchronous boundaries like Kafka, and high operational overhead from manually configuring OpenTelemetry SDKs and collectors. To resolve these issues, Odigos provides zero-code instrumentation via eBPF probes that dynamically attach to running workloads without modifying application code. An intelligent control plane within Odigos dynamically adjusts trace sampling and attribute enrichment based on real-time latency and error rates. The resulting telemetry exports over OTLP into ClickStack, an all-in-one backend featuring an OpenTelemetry Collector, ClickHouse storage, and the HyperDX query interface. This unified pipeline preserves end-to-end trace continuity, captures deep application and database context, and enables fast SQL-based trace analysis.


### [Introducing Application Metrics: Track the signal, see the spike, jump to the trace](https://yomu.fyi/post/introducing-application-metrics-track-the-signal-see-the-spike-jump-to.md)
- Company: [Sentry](https://yomu.fyi/company/sentry.md)
- Author: Ben Coe
- Published: May 5, 2026
- Type: Announcement

Sentry launched Application Metrics to capture application signals without pre-aggregating counters or discarding high-cardinality attributes like user IDs, project IDs, and regions. The system supports counters, distributions, and gauges within the existing SDK while linking each metric event directly to trace IDs for integrated debugging. Engineers demonstrated the feature by troubleshooting a Session Replay failure that triggered silently when sessions exceeded 1,000 video segments. By instrumenting distributions for replay event counts with project and replay attributes, the team isolated seven affected occurrences, navigated to connected traces and log lines, and reproduced the issue. Application Metrics comes enabled in recent SDK versions with 5GB included per plan.


### [The 5 Layers Every Cloud Commitment Depends On](https://yomu.fyi/post/the-5-layers-every-cloud-commitment-depends-on.md)
- Company: [Uber](https://yomu.fyi/company/uber.md)
- Published: May 4, 2026
- Type: Explainer

Long-term hyperscale cloud commitments are presented as distributed-systems problems with expensive consequences when physical assumptions are wrong. The post proposes a dependency sequence across regional and availability-zone topology, power architecture, fit-for-purpose compute, SKU qualification, and network design. It compares application-observable latency with bare-metal RTT, uses power consumption as a migration check, and argues that workload characteristics should drive memory-to-core ratios, silicon generations, instance families, and routing choices rather than provider defaults. It also explains why zone placement, inter-zone and inter-region transfer, and egress pricing affect quorum latency, reliability, and economics, while noting that cross-region traffic may require explicit justification. The conclusion is that these constraints compound over the commitment unless teams manage architectural drift, with software and contracting layers addressed separately.


### [The foundation of AI scalability: One team, one platform, one operating model](https://yomu.fyi/post/the-foundation-of-ai-scalability-one-team-one-platform-one-operating-m.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Aly McGue
- Published: May 4, 2026
- Type: Explainer

Albertsons Companies describes a centralized AI strategy for scaling decisions across merchandising, labor, supply chain, and customer experience across approximately 2,300 stores. The model combines one central AI core, the Databricks Data + AI Platform, and a shared operating model spanning data engineering, ML, governance, and analytics. Reusable ingestion pipelines, templates, feature-store patterns, model monitoring, performance observability, and governance wrappers support local execution, while a company-wide governance committee sets shared standards. Albertsons reports accepting 1.38 million lines of AI-generated code in nine months, with more than 90% of engineers using AI tools, and it provides low-code dashboards, prompt libraries, and conversational agent generation for nontechnical teams. Success is measured through reuse rates, time to deployment, responsible AI compliance, and business outcomes linked to AI uplift, with initiatives required to demonstrate impact before scaling.


### [Agentic identity: modeling agents to keep users in control](https://yomu.fyi/post/agentic-identity-modeling-agents-to-keep-users-in-control.md)
- Company: [Ramp](https://yomu.fyi/company/ramp.md)
- Author: Steve Venzerul, Rachel Kaplan
- Published: May 4, 2026
- Type: Problem & solution

Ramp for Agents adds CLI and MCP interfaces through which AI agents can manage cards, bills, expenses, and approvals, creating a need to represent agents as accountable actors instead of handing them users’ session tokens or API keys. The initial identity model links a named Agent Key to a human sponsor and business, bounds scopes by the sponsor’s role, records agent attribution, and adds expiration and revocation. Ramp extends RBAC and its DenormalizedActor audit model with AgentContext, so logs, approvals, and webhooks can distinguish “Sarah (via Codex)” while reusing existing permissioning and rendering paths. OAuth2-PKCE exchanges the Agent Key for short-lived access and refresh tokens, with re-validation during refresh to limit leaked-credential exposure and preserve session traceability. The first release uses on-behalf-of-user agents because they fit current demand and keep a human accountable, while admins can oversee keys or disable agent access for roles and groups.


### [Next-gen transit: Votran’s innovative model for multimodal partnership](https://yomu.fyi/post/next-gen-transit-votran-s-innovative-model-for-multimodal-partnership.md)
- Company: [Uber](https://yomu.fyi/company/uber.md)
- Published: May 4, 2026
- Type: Explainer

Volusia County Transit Services (Votran) has partnered with RATP Dev, Spare, and Uber Transit to unify demand-responsive transportation for more than half a million Volusia County residents. The VoTran On Demand app, powered by Spare, brings together the Transportation Disadvantaged program, VoRide microtransit, and other options alongside traditional fixed-route and paratransit services. Votran uses a mixed fleet of dedicated vehicles and Uber’s non-dedicated driver network to improve on-time performance, enable same-day trips, and reduce the cost of serving low-density areas. In zones such as Ponce Inlet, the app can request an Uber ride for local travel or connect riders to a suitable fixed bus route for destinations outside the zone, offering a model for balancing coverage and efficiency across a 1,100-plus-square-mile county.


### [Building trust into AI](https://yomu.fyi/post/building-trust-into-ai.md)
- Company: amazon.science
- Author: Staff writer
- Published: May 4, 2026
- Type: Explainer

Amazon integrates responsible AI practices across four core model development phases: pretraining, post-training, evaluation, and third-party monitoring. During pretraining, researchers augment training corpuses with specialized safety datasets, multimodality alignments, and learning exercises to teach foundational safety concepts instead of purely filtering harmful text. In post-training, reinforcement learning from human feedback uses human preference rankings, auxiliary-reward models, and independent LLM judges to ensure model outputs adhere to safety policies. For specialized use cases requiring access to sensitive domains like security testing, researchers apply low-rank adaptors to alter model behaviors surgically without retraining base weights. Cross-functional policy teams guide these technical stages by mapping risks against eight responsible AI dimensions and continuously updating behavioral boundaries to reflect evolving regulatory frameworks.


### [What we learned during the Custom Agents beta](https://yomu.fyi/post/what-we-learned-during-the-custom-agents-beta.md)
- Company: [Notion](https://yomu.fyi/company/notion.md)
- Author: Akshay Kothari
- Published: May 4, 2026
- Type: Announcement

Notion’s Custom Agents launched in beta in February, and more than one million agents were created in two months before the feature moved out of beta. The post reports that teams scaled fastest when they treated agents as internal products, iterated with workflow experts, and shared operational knowledge rather than relying on engineers alone. Teams also redesigned end-to-end processes, such as Braintrust’s agent that scans competitor updates and Gong transcripts, updates a Notion sales one-pager, and flags relevant launches. Usage costs fell by up to 50%, partly through lighter models including Haiku 4.5 and GPT-5.4 Mini, which use up to 10× fewer credits, while new limits, dashboards, alerts, and pause controls support administration. From May 4, usage-based pricing applies through Notion credits at $10 per 1,000 credits for Business and Enterprise plans.


### [Branching Without Git Is Now The Default](https://yomu.fyi/post/branching-without-git-is-now-the-default.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: joshenlim, qiao, saxonf
- Published: May 4, 2026
- Type: Announcement

Supabase has made dashboard-based database branching the default for all projects, adding a GitHub-free alternative to its existing git-based workflow. Developers can create a branch from the dashboard, work in the SQL Editor or Table Editor against a separate Postgres instance, review a schema diff, and merge changes into production without Git configuration. The workflow uses pg-delta, a new schema-diffing engine in pg-toolbelt that covers tables, columns, RLS policies, functions, triggers, indexes, and extensions, replacing migra as dashboard branching’s default engine; pg-delta is alpha software and is also available in the CLI behind a flag. Git-based branching remains fully supported, and the two approaches can be switched between or used together, while branches created through the Supabase MCP server use dashboard branching automatically.


### [Boosting multimodal inference performance by >10% with a single Python dictionary](https://yomu.fyi/post/boosting-multimodal-inference-performance-by-10-with-a-single-python-d.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: May 4, 2026
- Type: Problem & solution

Multimodal vision-language models pose new challenges for inference engines like SGLang. Profiling with py-spy revealed that the scheduler’s process\_input\_requests consumed about 13% of CPU time, with a quarter of that spent on repeated calls to torch.UntypedStorage.\_new\_shared\_cuda for CUDA IPC handle book-keeping. The solution was a simple Python dictionary cache for pool handles, eliminating redundant opens. On a Qwen2.5-VL-3B-Instruct benchmark with a single H100, throughput increased 16.2% from 22.2 to 25.7 requests per second, mean time-to-first-token dropped 13.2%, and mean time-per-output-token dropped 17.2%. The improvement is merged in SGLang v0.5.10 and applies to any multimodal model using SGLang’s CUDA IPC transport.


### [Building a new enterprise AI services company](https://yomu.fyi/post/building-a-new-enterprise-ai-services-company.md)
- Company: [Anthropic](https://yomu.fyi/company/anthropic.md)
- Published: May 4, 2026
- Type: Announcement

Anthropic, Blackstone, Hellman & Friedman, and Goldman Sachs announced a new AI services company that will help mid-sized organizations bring Claude into core operations. The firm will combine its engineering team with Anthropic Applied AI engineers to identify high-impact opportunities, build custom Claude-powered systems, and support customers over the long term; it is backed by additional alternative asset managers including General Atlantic, Leonard Green, Apollo Global Management, GIC, and Sequoia Capital. Engagements will start with a small team working with customer staff, then tailor systems to existing workflows, such as documentation, medical coding, prior authorizations, and compliance reviews in healthcare. The company will also join Anthropic’s Claude Partner Network, extending delivery capacity alongside systems integrators that serve large enterprises.


### [Learnings from init to deploy](https://yomu.fyi/post/learnings-from-init-to-deploy.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Anna Spysz
- Published: May 4, 2026
- Type: Tutorial

An agent was used to build a transcription app for a team from initialization through deployment with Stripe Projects, OpenRouter, and Vercel. The app sends video-generated subtitles through an OpenRouter agent with a detailed system prompt and Stripe documentation tool calls, producing a formatted transcription whose speaker and service names are corrected. The author recommends treating agent-generated plans like design documents, verifying provider behavior, and using streaming for long transcripts because Vercel serverless limits made non-streaming responses prone to 504s for a roughly 1,500-line transcript. Testing led to removing an unreliable free-model tier, defaulting to openrouter/auto, and choosing Claude Opus 4.5-thinking over Claude Sonnet 4.5; shared agent files were committed, and a separate review agent was used before human review.


### [Gala supercharges analytics performance with ClickHouse on AWS](https://yomu.fyi/post/gala-supercharges-analytics-performance-with-clickhouse-on-aws.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse
- Published: May 4, 2026
- Type: Problem & solution

Blockchain gaming platform Gala struggled to ingest and query expanding volumes of telemetry and blockchain data on its Databricks-based data infrastructure, resulting in multi-minute query times and heavy engineering maintenance overhead. To solve these bottlenecks, the company migrated its data platform to ClickHouse on AWS, ingesting data through Kafka, Airbyte, Amazon S3, and Fivetran. Gala leveraged ClickHouse's indexing capabilities alongside saved queries and API endpoints to power company-wide Metabase business intelligence dashboards for non-technical teams. Following the migration completed in December 2024, data capacity expanded from 3 TB to 9 TB while query times dropped from minutes to sub-second on previously unoptimized tables. The switch also lowered initial infrastructure costs by 30 percent, with plans underway to implement ClickHouse ClickPipes for improved data processing pipelines.


### [The federal data paradox: Rich in data, poor in access](https://yomu.fyi/post/the-federal-data-paradox-rich-in-data-poor-in-access.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Kacey Hertan
- Published: May 1, 2026
- Type: Problem & solution

Federal agencies have invested substantially in data infrastructure, but program directors, policy analysts, oversight officials, and budget examiners often still rely on technical intermediaries to answer operational questions. The post frames this as the unresolved “last mile” of federal data modernization: data lakes, APIs, dashboards, evidence-based policymaking mandates, and agency CDO functions have advanced infrastructure without making it usable by most decision-makers. It presents Databricks Genie as a natural-language interface that lets staff query agency data in plain language, including questions requiring joins across disbursement, eligibility, and geographic data, while retaining existing access controls and policies. Genie runs on Unity Catalog with role-based access controls, audit logging, and data lineage; it also supports federated cross-agency queries and records queries, answers, and sources, which the post associates with oversight, FOIA readiness, and accountability.


### [Optimizing ML Workload Network Efficiency (Part I): Feature Trimmer](https://yomu.fyi/post/optimizing-ml-workload-network-efficiency-part-i-feature-trimmer.md)
- Company: [Pinterest](https://yomu.fyi/company/pinterest.md)
- Author: Pinterest Engineering
- Published: May 1, 2026
- Type: Problem & solution

Pinterest deployed a root-leaf architecture for online machine learning serving to decouple CPU-based feature preprocessing from GPU-based inference. Because the root cluster retrieved a union of features across models and fanned out the full set to every leaf, network bandwidth became a severe bottleneck. After initial tests with fbthrift LZ4 compression yielded a 20% bandwidth reduction at the cost of CPU and latency overhead, the team implemented Feature Trimmer. This mechanism uses exported model signatures as the source of truth to prune unused features on the root before transmission. By dispatching only required features per model, the system reduced network bandwidth, decreased latency, and enabled cluster rightsizing that saved over $4M annually.


### [Driving Budapest forward: How BKK uses Databricks to transform city mobility](https://yomu.fyi/post/driving-budapest-forward-how-bkk-uses-databricks-to-transform-city-mob.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Max von Münster, Estilla Híves, Istvan Viz, Engin Erzengin
- Published: May 1, 2026
- Type: Problem & solution

BKK, Budapest’s unified transport authority, manages public transit, shared mobility, infrastructure, and traffic systems for a city of 1.7 million people. Its legacy on-premises warehouse and fragmented Excel- and PowerBI-based reporting struggled with growing, varied data, limiting efficient access and analysis. BKK phased a migration to Azure Databricks, starting with mobility datasets and carefully modeling and validating data from vehicle GPS, passenger sensors, boarding schedules, and other systems. The Databricks Lakehouse lets analysts process large geospatial datasets in shared SQL, Python, or R notebooks, supporting minute-level shared-mobility tracking, route-performance analysis, airport-bus forecasting, and dynamic scheduling. These capabilities provide faster insights and more responsive decisions, while governed access and cost visibility support broader collaboration and BKK’s vision for a digital twin of Budapest’s mobility system.


[Newer posts](https://yomu.fyi/posts/page/133.md) · [Older posts](https://yomu.fyi/posts/page/135.md)