# Latest reads
> The engineering internet, summarised so you can actually read it.

## Articles

### [From weeks to a day: how we made LLM evaluation fast enough to iterate on](https://yomu.fyi/post/from-weeks-to-a-day-how-we-made-llm-evaluation-fast-enough-to-iterate.md)
- Company: [Airbnb](https://yomu.fyi/company/airbnb.md)
- Author: Baharak Saberidokht
- Published: Jul 14, 2026

Airbnb built a four-layer LLM infrastructure framework to reduce iteration and evaluation turnaround from weeks to a single day. By using per-sample caching for generated references and judge scores, they established a deterministic evaluation foundation that separates genuine model drift from measurement noise. This setup enables rapid, bounded model hotfixes using micro LoRA adapters and comprehensive end-to-end validation across component boundaries.


### [Shipyard: How We Built Slack’s Next-Generation EC2 Platform](https://yomu.fyi/post/shipyard-how-we-built-slack-s-next-generation-ec2-platform.md)
- Company: [Slack](https://yomu.fyi/company/slack.md)
- Author: Archie Gunasekara
- Published: Jul 14, 2026

Slack developed Shipyard to modernize its EC2 infrastructure by replacing continuously updated, mutable instances with immutable deployable artifacts. The platform adopts a layered image architecture starting with a standardized base image called slack-zero, which is built using AWS Image Builder and consumed by downstream service-specific machine images. Rather than running periodic Chef jobs in the background, configuration management is restricted to image baking and initial instance provisioning phases. Fleet management is handled through the Peekaboo inventory system, metric-driven progressive rollouts via Gondola, and automated instance rotation managed by a lifecycle component known as the Reaper. Developers validate cookbook updates prior to merging using Ship Quick, which provisions real infrastructure across dedicated worker fleets to verify provisioning workflows.


### [Training Orchestrator: Unifying Model Training at Yelp](https://yomu.fyi/post/training-orchestrator-unifying-model-training-at-yelp.md)
- Company: [Yelp](https://yomu.fyi/company/yelp.md)
- Author: Ying Wang and Nathan Sponberg, Software Engineer
- Published: Jul 14, 2026

Yelp developed Training Orchestrator to standardize machine learning pipeline execution and eliminate divergent training scripts across its applied engineering teams. Previously, monolithic Spark scripts coupled training logic directly to cluster runtimes, preventing local testing, slowing debug cycles, and causing inconsistent configuration drift across projects. The new framework introduces a declarative configuration layer backed by Pydantic models, enabling static type checking and step schema validation before runtime execution starts. Training workflows are defined as modular step hierarchies that the orchestration engine compiles into directed acyclic graphs for execution in topological order. By injecting Spark and MLflow contexts into steps, the architecture ensures reproducible tracking, automated Slack alerts, and parity across local development and production environments.


### [Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned](https://yomu.fyi/post/building-service-topology-at-scale-architecture-challenges-and-lessons.md)
- Company: [Netflix](https://yomu.fyi/company/netflix.md)
- Author: Netflix Technology Blog
- Published: Jul 13, 2026

Engineers at Netflix required a unified, real-time view of service dependencies to navigate distributed architecture and improve incident troubleshooting. Traditional batch systems introduced stale data, so the team created a streaming-first platform backed by reactive streams and backpressure handling to ingest flow records from multi-region Kafka streams and Server-Sent Events without data loss. The architecture partitions data into physically separate graph and columnar storage layers covering eBPF network flows, IPC metrics, and distributed traces. Network flow ingestion relies on a three-stage distributed aggregation pipeline using consistent hashing to resolve network intermediaries into logical application connections. The resulting production system serves time-travel and topology queries with sub-second latency while continuously updating dependency views.


### [Modernizing the Meta Ads Service With an Open-Source Kernel Scheduler](https://yomu.fyi/post/modernizing-the-meta-ads-service-with-an-open-source-kernel-scheduler.md)
- Company: [Meta](https://yomu.fyi/company/meta.md)
- Published: Jul 13, 2026

Meta's ad serving fleet encountered latency regressions and technical debt after Linux kernel upgrades introduced the default EEVDF scheduler. To resolve these performance drops across billions of daily requests, the engineering team implemented sched\_ext, an open-source, BPF-based scheduling framework integrated into Linux kernel v6.12. The custom scheduling policy uses domain knowledge to soft-partition CPUs dynamically into separate pools for latency-critical request threads and background work, enhancing L3 cache locality. Packaged as a user-space binary, the scheduler allows policy iterations and rollouts via process restarts without kernel rebuilds. The deployment achieved a 28% reduction in p99 ads retrieval latency, a 1.1% increase in ranked ads, and 3.28 megawatts in fleet-wide power savings.


### [Better tools made Copilot code review worse. Here’s how we actually improved it.](https://yomu.fyi/post/better-tools-made-copilot-code-review-worse-here-s-how-we-actually-imp.md)
- Company: [Github](https://yomu.fyi/company/github.md)
- Author: Napalys Klicius
- Published: Jul 10, 2026

When migrating GitHub Copilot code review from custom navigation tools to shared Unix-style CLI utilities—grep, glob, and view—benchmarks revealed higher review costs and fewer caught issues. Rather than an issue with the underlying tools, trace analysis showed the agent used general-purpose coding assistant instructions that triggered expansive repository browsing loops. In response, the team rewrote the tool guidance to enforce a review-specific workflow anchored to pull request diffs. The updated instructions direct the agent to narrow candidate call sites and files using batched discovery before reading minimal line ranges with view. This workflow tuning reduced average review costs by approximately 20% in production while maintaining review quality.


### [Scaling Grab's Data Lake: Our journey to Apache Iceberg adoption](https://yomu.fyi/post/scaling-grab-s-data-lake-our-journey-to-apache-iceberg-adoption.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Rahul Penti
- Published: Jul 10, 2026

Grab transitioned its petabyte-scale data lake from Hive Parquet to Apache Iceberg to resolve metadata bottlenecks, small file fragmentation, and lack of native ACID support. Implementing Iceberg with Z-ordering reduced navigation query runtimes from 70 seconds to 6 seconds and cut daily S3 API costs by up to 95% on key operational datasets. To seamlessly manage concurrent table formats without forcing engineers to specify format prefixes, Grab developed and open-sourced UnifiedSparkCatalog. This custom catalog transparently detects table types via the Hive Metastore, routes queries to format-specific handlers, and retains backwards compatibility for legacy Hive operations.


### [From Day 1 to Production: Building Lyft’s Analytics & Rides Intelligence Assistant as Onboarding…](https://yomu.fyi/post/from-day-1-to-production-building-lyft-s-analytics-rides-intelligence.md)
- Company: [Lyft](https://yomu.fyi/company/lyft.md)
- Author: Sagarbaronia
- Published: Jul 9, 2026

Lyft developed Aria, an AI-powered assistant allowing approved employees to query ride analytics via natural language through a backend with LangGraph orchestration and SQL generation. However, Aria's user interface was limited to a Streamlit prototype that lacked standard service framework integration, Lyft authentication, and multi-user scalability. To resolve these limitations, a new engineer built a production-grade web client from scratch using Lyft's internal Node.js framework and Next.js as a three-week onboarding assignment. The implementation involved configuring Envoy routing, resolving authentication plugin incompatibilities, debugging service connectivity with Grafana, and managing interface state with an XState state machine. Aria was successfully hardened and launched into production, expanding natural language analytics access across the company.


### [When and what should I be logging?](https://yomu.fyi/post/when-and-what-should-i-be-logging.md)
- Company: [Sentry](https://yomu.fyi/company/sentry.md)
- Author: Ben Coe
- Published: Jul 9, 2026

Targeted structured logging provides a fast way to capture operational behavior in production without requiring new deployments for debugging. Developers benefit most from logging key runtime decisions, multi-step algorithm outcomes, mutating audit operations, and context around non-critical or retried errors. Rather than outputting plain text, applications should record structured key-value pairs that capture who performed an action, what occurred, and associated trace identifiers. Sensitive information such as secrets and personally identifiable data must be excluded or scrubbed, and large unparsed payloads should be avoided to prevent excessive costs. Managing log levels appropriately and treating logs as potentially temporary instrumentation helps maintain actionable telemetry without generating unnecessary noise.


### [Automating cross-repo documentation with GitHub Agentic Workflows](https://yomu.fyi/post/automating-cross-repo-documentation-with-github-agentic-workflows.md)
- Company: [Github](https://yomu.fyi/company/github.md)
- Author: David Pine
- Published: Jul 8, 2026

Maintaining documentation across separate repositories often leads to severe lag because technical writers must reverse-engineer shipped features weeks after release. To address this in the Aspire project, the team implemented an automated pipeline using GitHub Agentic Workflows to bridge the product and documentation repositories. When product pull requests merge, a bash step maps milestones to docs release branches before an LLM agent evaluates the diff, drafts documentation updates, and emits structured pull request intents. A dedicated safe-outputs handler materializes these drafts via a scoped GitHub App and assigns the original code reviewers to verify accuracy. Across 396 product pull requests, the system generated 82 documentation pull requests that all merged with a median turnaround time of 44.8 hours.


### [Introducing Okta Journeys: A Better Way for Developers to Learn Identity](https://yomu.fyi/post/introducing-okta-journeys-a-better-way-for-developers-to-learn-identit.md)
- Company: [Okta](https://yomu.fyi/company/okta.md)
- Author: Dan Maharry
- Published: Jul 7, 2026

Okta has introduced Journeys, a task-oriented documentation format designed to streamline how developers learn and implement identity management. Rather than requiring engineers to manually assemble information across disparate guides, articles, and API references, each Journey provides an end-to-end curriculum for a specific project. Content is organized into three distinct phases: foundational knowledge in Learn, architectural decision-making in Plan, and curated implementation steps in Build. The initial release includes six Journeys aimed at Okta Customer Identity builders who are implementing authentication, registration, branding, and user management. Future additions will address AI agent governance, integrations for the Okta Integration Network, and migrations from the Classic Engine to the Okta Identity Engine.


### [When the sensor starts thinking: SnortML, agentic AI, and the evolving architecture of intrusion detection](https://yomu.fyi/post/when-the-sensor-starts-thinking-snortml-agentic-ai-and-the-evolving-ar.md)
- Company: [Stack Overflow](https://yomu.fyi/company/stack-overflow.md)
- Author: Samaresh Kumar Singh
- Published: Jul 6, 2026

Classic intrusion detection signatures suffer from an exposure window between zero-day discovery and rule distribution, leaving slight exploit variations unflagged. Cisco Talos addressed this by embedding SnortML into Snort 3 to perform local, sub-millisecond machine learning inference alongside signature checks. SnortML employs an LSTM preceded by a byte-embedding layer and XNNPACK acceleration to score URI queries and POST bodies for vulnerabilities like SQL injection, XSS, and command injection. Running both detection mechanisms concurrently provides independent error profiles and boosts detection confidence when both fire. The broader defense landscape is shifting toward connecting these packet-level ML detectors with agentic AI pipelines and supervised feedback loops.


### [Migrating Counter Service storage: Design choices and learnings](https://yomu.fyi/post/migrating-counter-service-storage-design-choices-and-learnings.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Jia Long Loh
- Published: Jul 3, 2026

Grab migrated its Counter Service storage backend from a legacy wide-column database to Aerospike to support real-time anti-fraud windowed aggregations across tens of thousands of queries per second. To decouple storage from the Rust reader service, engineers introduced a storage facade using enum dispatch, avoiding the per-query heap allocations of boxed trait objects. The reader used configuration-driven operating modes to support shadow reads and deterministic traffic splitting without requiring code updates. On the write path, the schema was redesigned to collapse multiple bucket records into a single sorted map per counter, using atomic server-side operations to increment counters and prune expired entries. This data model redesign reduced total record counts and in-memory primary index usage by over an order of magnitude while enabling a zero-downtime transition.


### [Any Apple update can break our app. Here's how we find out first.](https://yomu.fyi/post/any-apple-update-can-break-our-app-here-s-how-we-find-out-first.md)
- Company: [Sentry](https://yomu.fyi/company/sentry.md)
- Author: Dan Mindru, Oleh Stasula
- Published: Jul 2, 2026

Usage is an Apple ecosystem resource monitor that relies on undocumented APIs vulnerable to silent data structure changes across OS updates. To maintain stability across iPhone, iPad, and Mac without degrading device performance, the development team uses Sentry across separate component projects in a unified workspace. Beyond capturing standard crashes, the team wraps Swift errors with unexpected raw data payloads to quickly diagnose undocumented API structure drift while scrubbing sensitive user values. They also employ Sentry Size Analysis as a regression gate to monitor release payload deltas, preventing unintended asset omissions or bloat from slipping through code reviews. This multi-layered monitoring setup enables the team to detect and resolve platform-induced breakages before most end users encounter them.


### [How do you turn AI coding chaos into a repeatable playbook?](https://yomu.fyi/post/how-do-you-turn-ai-coding-chaos-into-a-repeatable-playbook.md)
- Company: [Stack Overflow](https://yomu.fyi/company/stack-overflow.md)
- Author: Eira May
- Published: Jul 2, 2026

Software development dynamics are shifting as code generation costs decrease, requiring engineering organizations to adapt their leadership and processes. Snowflake addressed this transformation across its software lifecycle by first encouraging widespread, unrestricted experimentation with coding agents before formalizing successful practices. The engineering organization codified these practices into fourteen distinct AI design patterns, including strategies such as planning in markdown and isolating parallel agents. This structured approach, combined with dedicated focus weeks and a progress scale, allowed engineering teams to accelerate both inner and outer loops of development. Consequently, a team of three engineers leveraged coding agents to deliver a 40x improvement to Snowflake's query compiler while significantly shortening release validation times.


### [Meta’s AI Storage Blueprint at Scale](https://yomu.fyi/post/meta-s-ai-storage-blueprint-at-scale.md)
- Company: [Meta](https://yomu.fyi/company/meta.md)
- Author: By Sidharth Bajaj, Venkatraghavan Srinivasan
- Published: Jul 1, 2026

Meta redesigned its BLOB-storage architecture on top of the foundational Tectonic block layer to eliminate GPU stalls and accelerate AI training workflows. The legacy storage system suffered from multi-layered metadata lookups, cross-region latency, and dataplane proxy bottlenecks that conflicted with the millisecond access requirements of flash-based AI clusters. To resolve these issues, the team collapsed disparate metadata into a single flat schema backed by ZippyDB for O(1) path lookups and introduced a fat client SDK capable of streaming data directly from storage servers. The updated stack also leverages spare GPU host memory as a distributed data cache alongside distributed read-plan caches to handle severe traffic spikes during checkpointing and model loading. These protocol and architectural changes deliver predictable latencies, reduce power consumption, and maintain high throughput during distributed training.


### [Reading the agent traces is how you make the call your eval can't](https://yomu.fyi/post/reading-the-agent-traces-is-how-you-make-the-call-your-eval-can-t.md)
- Company: [Sentry](https://yomu.fyi/company/sentry.md)
- Author: Sergiy Dybskiy
- Published: Jul 1, 2026

Automated evaluations often miss unknown failure modes in AI agents because developers only write tests for anticipated errors. While building an itinerary agent for AI Engineer World's Fair, an open-weight model routed a speaker query to a tool returning only tracks, leading the model to hallucinate famous speakers and falsely claim the tool provided that data. Because a tool was executed, the trace appeared grounded until inspectable tool outputs revealed the discrepancy. To resolve this, the author retained the inexpensive model, corrected query routing, tightened system prompts against ungrounded entities, and added a custom entity-diffing eval in CI. Ultimately, directly reading agent telemetry traces alongside errors and logs is essential for diagnosing root causes and deciding trade-offs between model costs, prompt adjustments, and routing fixes.


### [10 Years of Meta’s Commitment to Python](https://yomu.fyi/post/10-years-of-meta-s-commitment-to-python.md)
- Company: [Meta](https://yomu.fyi/company/meta.md)
- Published: Jun 30, 2026

Meta has marked ten consecutive years as a sponsor of the Python Software Foundation (PSF) to ensure the language remains robust and sustainable. Python represents the most widely used programming language across Meta's infrastructure, powering applications like Instagram and Threads as well as internal AI research. Beyond financial backing, Meta engineers actively contribute core language features, author Python Enhancement Proposals (PEPs), and build open-source tools such as the Pyrefly type checker. Company sponsorship directly funds the PSF's Developer-in-Residence program, core infrastructure, and security enhancements for the Python Package Index (PyPI). Continued support of the PSF functions as a strategic investment in maintaining the stability of the broader open-source ecosystem.


### [Why intent prediction needs more than an LLM](https://yomu.fyi/post/why-intent-prediction-needs-more-than-an-llm.md)
- Company: [Stack Overflow](https://yomu.fyi/company/stack-overflow.md)
- Author: Phoebe Sajor
- Published: Jun 30, 2026

Traditional large language models rely on next-token prediction trained over broad text sequences, excelling at conversational synthesis and code generation while struggling with forecasting future user behavior. Predicting intent inherently requires modeling decision-making under conditions of uncertainty, a capability not directly produced by general text pretraining. Dedicated behavioral foundation models address this limitation by training representations directly on proprietary, sensitive behavioral data, such as anonymous browsing sessions and product identifiers. By learning base representations of behavior, these models generalize to unseen predictive tasks, such as optimizing ad campaigns for products absent from the original training dataset. Implementing these behavioral models also necessitates privacy-centric machine learning techniques, ranging from differential privacy concepts to homomorphic machine learning.


### [GenPage: Towards End-to-End Generative Homepage Construction at Netflix](https://yomu.fyi/post/genpage-towards-end-to-end-generative-homepage-construction-at-netflix.md)
- Company: [Netflix](https://yomu.fyi/company/netflix.md)
- Author: Netflix Technology Blog
- Published: Jun 29, 2026

Netflix traditionally constructs its structured two-dimensional homepage through a complex, multi-stage recommender pipeline that separates candidate generation and ranking across rows and entities. To simplify this architecture and optimize directly for whole-page user satisfaction, Netflix developed GenPage, an end-to-end generative transformer model that autoregressively builds the entire homepage from raw tokenized context. The system relies on a domain-specific tokenizer to compress engagement history, context injection for cold start, hybrid row decoding to minimize decoding steps, and reinforcement learning post-training. In online A/B testing against the production baseline, GenPage delivered statistically significant gains in core user engagement metrics while cutting end-to-end serving latency by 20 percent. Offline evaluations further showed that enriching context prompts improved recommendation quality more effectively than increasing model capacity in the current operating regime.


[Newer posts](https://yomu.fyi/page/7.md) · [Older posts](https://yomu.fyi/page/9.md)