Loading…
Browse by type
Problem & solution reads
Problem & solution posts on Yomu, summarised newest first.
- 925 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Problem & solution posts
Beyond pass or fail: The new era of chaos engineering
Binary pass/fail uptime testing no longer suffices for cloud-native deployments because a cluster can remain operational while dropping packets, breaching SLOs, or suffering performance degradation. Krkn addresses this with a trifecta of telemetry: a weighted Resiliency Score, continuous HTTP Health Checks, and KubeVirt Checks. The Resiliency Score is a 0–100% macro-health metric that evaluates live Prometheus data against pre-defined SLOs using custom weights, allowing engineers to prioritize business-critical components like authentication or payment gateways. HTTP Health Checks probe application endpoints at configurable intervals, capturing exact downtime start and end times alongside HTTP status codes, enabling teams to track self-healing duration. KubeVirt Checks monitor virtual machine instances via real-time SSH connectivity and record node, IP address changes, and exact down-time, with a final post-chaos check that flags VMs remaining unreachable after the experiment concludes.
Stripe’s Payment Method Factory: Orchestrating agents for repeated, custom integrations
Stripe offers over 125 different payment methods globally, each requiring integrations that previously took up to six months because teams started from scratch. They hypothesized that reusable prompts with hardening could reduce effort, and a bake-off showed vanilla Claude Code took 20 days versus 4 days with their prompts. The resulting factory has created three new payment method integrations and migrated ten existing ones to a modern stack, reducing timelines from six months to two to six weeks. The factory also uses code-based orchestration, continuous knowledge loops where agents suggest prompt improvements, and agents testing their own code against sandbox environments. The factory’s approach has since been extended to Stripe’s tax and verification teams for building their own connectors.
Getting the Source Right, Not Just the Fact: Source-Aware Verification for MCP Agents
MCP agents can produce claims that are factually true but attributed to the wrong source, a failure known as cross-source conflation that source-blind checkers miss. ProvenanceGuard preserves source identity by decomposing answers into claims, routing each to its relevant tool output, and checking support and attribution separately, using local models like MiniLM and DeBERTa. On medical agent traces with 281 interactions, it caught 138 of 139 claims experts flagged, scoring 0.802 F1 on blocking decisions and correct source 86% of the time. It flagged 67 supported claims for review under a cautious policy and added roughly half a second per answer in local tests. Blocked answers can be fixed through a repair loop for rewriting or fallback, avoiding unverifiable responses while keeping sensitive traces in a controlled environment.
How we cut CI failure triage from hours to minutes using AI
The Pipeline Failure Analyzer (PFA) cuts CI failure triage from hours to minutes by confining AI to exactly one of five pipeline stages. Four stages remain deterministic: Prepare cleans raw logs by stripping ANSI codes and progress bars, Summarize assembles structured reports, Notify creates tickets, and Publish commits to wikis. The AI Analyze stage groups failures by root cause using semantic reasoning, overcoming traditional string-based algorithms that fail when the same missing dependency produces different error messages across CPU and GPU architectures. Prepare achieves 95–98% compression on 10,000-line traces, reducing them to 200–500 lines with virtually no signal loss by extracting error anchors and merging overlapping context windows. Before AI output proceeds downstream, five validation layers—including 13-field checks, cross-references, path audits, JSON schema, and trace verification—prevent hallucinated diagnoses from entering tickets or reports.
86% of brands think their AI works. Only 51% of customers agree.
A Twilio research report identifies a 35-point satisfaction gap: 86% of brands believe their AI customer service satisfies customers, but only 51% of consumers agree. The discrepancy arises because companies measure metrics such as conversation completion surveys and containment rates, which systematically exclude abandoned or unresolved interactions. 71% of consumers abandon a conversation when the AI fails to recognize them, yet those exits rarely appear in satisfaction data. 49% of consumers say AI never resolved their issue. To close the gap, Twilio recommends tracking abandonment by conversation stage, resolution of the underlying issue across sessions, repeat contact within seven days, containment split by outcome, and escalation quality.
How Databricks rolls out frontier models to 14,000 employees on Day 1
Databricks provides employees Day 1 access to new AI models using Unity Gateway for governance and cost management. The company uses a three-step pipeline: immediate experimental availability, per-user budget constraints, and data-driven promotion decisions. Unity Gateway CLI distributes model configurations to employee laptops, tagging new models as experimental. Per-user budgets include a monthly maximum, daily runaway limit, quality frontier budget, and experimental budget to control costs. Within three days of the Opus 5.5 and GPT-6 Sol launches, Databricks gathered benchmark, user feedback, and cost data to confirm they were on the efficiency frontier and promoted them, with Opus 5.5 showing a 29% cost reduction per session over Opus 4.8 and GPT-6 Sol showing a 48% reduction over GPT-5.6 Sol.
Engineering Multi-Agent AI Teams That Build and Test Themselves
Salesforce Marketing Cloud’s AI Delivery Lead Sohini Arya and team built Agent Designer, a system that automates the creation and testing of multi-agent AI teams. The orchestrator cannot write agent files itself; instead, seven specialist agents analyze architecture, failure modes, and compliance before the orchestrator synthesizes findings for human review. The system transforms an engineer’s request into a tested multi-agent team in 15-30 minutes, down from 2-4 hours manually, and helped approximately 200 engineers adopt the manager-of-agents model. Governance challenges included enforcing write-scope boundaries via prompts, reconciling independently produced design conflicts, and applying topology-specific budget formulas that account for flat, swarm, or team structures. Verifier agents return structured PASS/WARN/FAIL blocks, and self-healing is limited to two fix attempts and a $3 ceiling to prevent runaway repairs, with the orchestrator’s 500-turn budget divided across analysis, approval, generation, testing, and retries.
Why model versioning is not enough for production AI
Model versioning alone is insufficient for production AI because changes in inputs, preprocessing, prompts, retrieval, tool contracts, and serving settings can alter behavior independently. The post proposes a versioned release manifest that captures all interdependent components, including model revision, prompt revision, retrieval index revision, embedding revision, pipeline revision, and runtime revision covering token limits, batching, and timeouts. This manifest enables atomic release testing and rollback, with an evaluation gate that tests the full retrieval-generation path using deterministic checks and semantic rubrics on a versioned dataset. Practical steps include canary deployments that compare candidate and control signals, with explicit stop conditions and rollback procedures that restore compatible dependencies rather than just older model weights. The post concludes that the smallest useful implementation is a manifest, evaluation job, load test, release-aware traces, and a rehearsed rollback, and asks whether the on-call engineer can identify and restore a compatible known-good version.
Is Your “Human-in-the-Loop” Actually Slowing You Down? Here’s What We Learned
Human-in-the-loop (HITL) frameworks integrate human judgment into automated AI workflows to improve reliability and safety, but without careful design they can create bottlenecks. The post reveals key trade-offs: more automation increases speed and reduces cost but raises risk, while more human oversight boosts accuracy and safety but adds latency and cost. Common failure modes include over-reviewing trivial cases, lacking clear trigger logic, and expecting human reviewers to scale linearly. The proposed solution is a tiered HITL system applying the Pareto principle: automation handles 85% of high-confidence decisions using confidence calibration, with three tiers addressing the remaining cases under 3 ms latency, asynchronous review within hours, and real-time oversight with 30-60 second windows. Success requires careful routing, reviewer tooling, and closing feedback loops across seconds, days, weeks, and quarterly cycles to continuously improve model performance and reduce human involvement over time.
Scaling Beyond Borders: The 2026 Communication Blueprint for Chinese Brands
Chinese enterprises expanding overseas face fragmented communication channels, strict compliance regimes, and data residency mandates across 180+ markets. A programmable communication platform (CPaaS) eliminates the need to negotiate carrier-by-carrier in every country. Twilio’s global infrastructure — 335,000+ customers, 218+ voice-capable countries, and a 99.95% API SLA — provides a single integration point for SMS, voice, email, WhatsApp, and RCS worldwide. The platform comprises five layers: programmable messaging, voice APIs, SendGrid email, Twilio Verify, and Twilio Segment. Compliance features such as 3-region data residency and email authentication help brands navigate GDPR, TCPA, and other regulations.
Faster Rates for Federated Variational Inequalities
This paper studies federated optimization for solving stochastic variational inequalities (VIs), a field with growing attention but a significant performance gap compared to federated convex optimization. The authors address this gap by establishing a series of improved convergence rates. They first refine the analysis of the classical Local Extra SGD algorithm, providing tighter guarantees for smooth and monotone VIs. However, they identify an inherent limitation of Local Extra SGD: excessive client drift. To overcome this, they propose the Local Inexact Proximal Point Algorithm with Extra Step (LIPPAX), which mitigates drift and achieves better rates under bounded Hessian, bounded operator, and low-variance regimes. The work also extends these improved guarantees to federated composite variational inequalities.
Cloudflare’s 2026 Annual Founders’ Letter
Cloudflare’s 2026 founders’ letter, marking the company’s 16th birthday, examines rapid changes in internet traffic driven by AI. Since mid-2025, new website creation has exploded, fueled by ‘vibe coding’ tools that allow non-programmers to build apps, with over 7 million developers now using Cloudflare’s platform. Automated traffic from AI agents and crawlers is projected to surpass human traffic by May 2026 and could reach 1,000 times human levels within five years. This surge risks a tragedy of the commons: agents consume resources from many sites but benefit only one, threatening small businesses that lack the information volume older competitors hold. Cloudflare proposes reducing crawler load by fetching only changed content and enabling creators to receive payment when agents use their content, alongside partnerships to shape a fairer internet.
From Data to Dialogue: How S&P Global Energy Made Its Structured Data Estate Conversational with Databricks Genie Agents and MCP
S&P Global Energy aimed to make its structured data estate—spanning LNG, Chemicals, Crude Oil, and other commodities—conversationally accessible via AI agents. The team evaluated approaches and chose Databricks Genie Agents, where subject matter experts curate one agent per dataset group without writing code. Each Genie Agent is automatically exposed as a managed MCP server, providing a governed, zero-deployment integration point. A FastMCP-based proxy composes multiple Genie MCP servers into composite endpoints for cross-domain questions. This architecture shortened time to market from months to days, with governance via Unity Catalog and an open MCP standard enabling both internal and external consumers.
A kernel-centric path to real-time video generation on Trainium
Reactor and the AWS Neuron Science team collaborated to optimize real-time video generation on AWS Trainium. They focused on autoregressive diffusion models, which combine sequential next-token prediction with iterative diffusion to generate video frames that respond to user input. The teams used the Rolling Forcing model as a proxy, addressing challenges such as dynamic shapes, unusual memory access patterns, and heavy cache management that generic compilers struggle with. Optimizations included splitting SDPA heads across four cores and sequences across two, and batching common components of the diffusion and cache update phases to improve hardware utilization. The collaboration achieved real-time video generation above 16 frames per second on a single Trainium2 chip, a result that generalizes across autoregressive diffusion models.
Automated merchant review summary system with integrated feedback
The system uses representative sampling, large language model (LLM) summarization, automated quality checks, and consumer feedback to generate concise summaries from written merchant reviews. Consumers can scan an overall summary or focus on specific topics such as taste, packaging, and portion size, while merchants can identify recurring themes without reading every comment. The pipeline calculates the merchant’s rating distribution, uses a sentiment model to score reviews, and applies stratified sampling to select a representative subset before sending them to an LLM for summary generation. A separate LLM validates that each claim in the generated summary is supported by the source reviews, and summaries that fail this automated assessment are withheld from publication to reduce unsupported claims. The system regenerates summaries when new reviews or negative consumer feedback reach configured thresholds, and the feedback-driven regeneration includes anti-abuse controls to maintain trustworthiness.
How Intelligent Load Shedding Prevents Cascading Failures in Tier-0 Systems
Salesforce’s Cloud Atlas team operates a globally distributed identity data store that must maintain five nines of availability, roughly five minutes of downtime per year. The team redesigned service protection after per-instance rate limiting proved insufficient under autoscaling and increasingly bursty, unpredictable agent-driven traffic. The new approach combines intelligent load shedding, which monitors queue time as an early indicator of system pressure before error rates spike, with coordinator-free global quota management. Servers independently measure and share customer usage to throttle noisy tenants without a centralized coordinator, preventing one customer’s surge from affecting others. This design prevents cascading failures by shedding lower-priority work before overload spreads across dependent services, ensuring graceful degradation rather than widespread outage.
Designing the Kotlin Multiplatform and TeamCity Integration
The post describes the design of a new TeamCity integration for Kotlin Multiplatform (KMP) developers building iOS apps. The integration addresses common pain points: iOS builds require macOS infrastructure, Apple signing is complex, and there is no guided path from IDE to CI/CD. The solution provides a progressive onboarding flow within the IDE, starting with a tooltip after the first Git push. The flow generates four configuration files (teamcity.yaml, Fastfile, Appfile, Gemfile) and runs a build and tests on a hosted macOS agent, then guides the developer through a single form for Apple signing and TestFlight publishing, and finally automates the pipeline on GitHub pushes. TeamCity Cloud provides hosted macOS agents and a free starting tier, so no Mac is required. The design was validated through usability sessions and focuses on meeting the developer at the right moment with minimal configuration.
How I built agent-based security reviews on Databricks
Security reviews at Databricks relied on automation but still consumed expert time on predictable work, leaving less room for genuinely novel, high-risk architectures. The author built an agent-based layer on Databricks using Unity Catalog for governed tables containing standards, request data, and decisions; Databricks-hosted foundation models (Claude Haiku, Sonnet, Opus) for classification and reasoning; and Lakeflow Jobs for orchestrating notebook-based workflows on serverless compute. A conversational intake app built with Databricks Apps turns plain-language descriptions into structured requests and provides preliminary risk indications before a formal request is created. Seven focused agents each have a bounded responsibility—intake, risk assessment, requirements, specialized reviews, validation, workflow, and learning—ensuring behavior remains inspectable and testable. Routine requests that once waited days in the queue can now be completed in minutes, and security engineers spend more time on designs requiring genuine expertise.
Hitting a billion tokens per minute on one GPU by combining a query planner and an inference engine
Modal and CMU built Quail, a query-aware inference layer for AI-SQL that combines a query planner with an inference engine to process structured LLM transformations of database entries. On a multi-join query, Quail achieves over a billion tokens per minute per H100 GPU, over 10x faster than vLLM, costing under 6 cents per billion tokens on Modal. The gain comes from using the query plan to order requests for better KV cache management, including a revised Hydragen-style cascade attention and batching small requests to reduce host overhead. On a new AI-SQL benchmark, Quail runs 1.84x faster than vLLM, but falls behind on agent trace processing. The authors see this as part of a trend toward custom inference engines for specific workloads, with a reusable core like databases post-DataFusion, and note that much work was done by prompting coding agents.
A Practical Recipe for Semi-Supervised Federated ASR: Online Pseudo-Labels with Server Update Stabilization
Semi-supervised federated learning (SSFL) for ASR uses a teacher to generate pseudo-labels on unlabeled client data while the server trains on a small labeled seed. The paper finds that pseudo-label errors compound across the output sequence and training rounds, causing divergence and a large gap to fully-supervised FL. Two coupled design axes close this gap: the teacher type (per-client online, global broadcast, or transitioning) and the anchor (server-side labeled-data updates that stabilize training). A per-client online teacher diverges on its own but, once stabilized, matches or beats the global teacher in-domain and under domain shift. The server must keep training on labeled data between rounds; otherwise the online teacher drifts, and this interleaving governs convergence more than the seed model does. The proposed guidelines improve over the strongest prior method on 9 of 11 pairs, by 20.8% in-domain and 10.0% cross-domain on average.
Top topics
- Architecture834
- Performance816
- LLMs668
- Machine Learning622
- Open Source567
- AI Agents559
- AI476
- Developer Experience448
Weekly digest
The best engineering reads, once a week.