Loading…
Latest reads
The engineering internet, summarised so you can actually read it.
Automated Schema Evolution in Pinterest’s Next-Generation DB Ingestion Framework
Pinterest's CDC ingestion platform relies on a multi-stage pipeline built with Kafka, Flink, Spark, and Iceberg, where upstream schema updates risk disrupting data transformations and offline storage. To prevent pipeline failures and schema drift, Pinterest introduced an automated schema evolution framework based on a staged convergence model. Upstream changes are detected via push-based DDL messages and daily pull-based validation checks. The workflow automatically restricts automated updates to additive changes and numeric precision widening, updating Iceberg table definitions and regenerating Flink and Spark code through auditable pull requests. By decoupling schema divergence, code deployment, and data convergence, the system maintains continuous ingestion availability while working toward eventual schema consistency across all storage layers.
Pinterest EngineeringYour AI shipped a backend that boots. That is the whole problem.
Backend security vulnerabilities often emerge from permissive default settings in web frameworks, a risk amplified when AI agents generate code focused solely on passing tests. Common failure points include unmetered request body parsing, unvalidated schemas, prototype pollution, and wildcard cross-origin resource sharing policies. To eliminate these foot-guns, the TypeScript framework DaloyJS enforces secure-by-default behavior across its entire routing and middleware model. It provides built-in request timeouts, streamed body size caps, prototype-stripping JSON revivers, and unified Zod schemas that simultaneously validate payloads and generate OpenAPI contracts. While secure defaults lower the risk of common infrastructure exploits, developers still remain responsible for application-level authorization and business logic correctness.
Devlin DuldulaoThe 2026 Developer Survey is now open (for human developers only)!
The 2026 Developer Survey is open to gather data on software builders' tools, working lives, and emerging practices. Marking fifteen years since its initial 2011 release, the annual survey focuses heavily on the role of artificial intelligence and autonomous agents across the software development lifecycle. Past survey results indicated that while developer adoption of AI and agents doubled, user trust declined alongside escalating concerns regarding generated code quality. In addition to modern AI trends and financial return considerations, the questionnaire maintains traditional inquiries regarding day-to-day technologies and developer workflows. The survey is hosted on the Qualtrics platform, which requires users to pause third-party ad blockers and adheres to regional access restrictions across multiple countries.
Ryan DonovanNetflix ·
Toward More Controllable AI Video Editing: An Early Research Exploration at Netflix
Generative video editing tools often regenerate entire video clips, which inadvertently modifies untouched scene details or violates physical continuity during object deletion. To provide artists with precise control, two research prototypes were developed: Vera and VOID. Vera uses a layered video diffusion framework with a Mixture-of-Transformers architecture to jointly generate separate edit and alpha matte layers, compositing them with original footage while leaving unchanged pixels untouched. VOID addresses object deletion by conditioning on interaction-aware quadmasks and using a second-pass appearance refiner to reconstruct scenes with plausible physical continuity. In a user study evaluated across 75 real-world scenarios, reviewers selected VOID 64.8% of the time over six baselines.
Netflix Technology BlogNetflix ·
How Netflix Simplified Batch Compute with Kueue
Netflix transitioned its managed batch compute infrastructure from a homegrown solution called Compute Managed Batch to Kueue on its Titus container platform. CMB previously relied on custom scheduling and admission-only fair sharing without preemption, making feature development cumbersome as the Kubernetes ecosystem evolved. To modernize the platform transparently, Netflix mapped internal tenants to Cohorts and leaf tenants to ClusterQueues and LocalQueues while routing jobs through a custom Kueue router. Kueue operates alongside existing Titus scheduling profiles rather than replacing the kube-scheduler, preserving cluster placement efficiency. The migration was completed in four weeks across millions of batch workloads, significantly increasing average resource utilization through preemption-based fair sharing.
Netflix Technology BlogGrab ·
Scaling out Distroless adoption With AI
Grab is transitioning its microservices to Distroless base images to eliminate unnecessary binaries and reduce vulnerability risks, but the migration risks runtime failures from missing shared objects and system utilities. To safely validate container execution in continuous integration without staging dependencies, the team relied on medium tests that run containerized services alongside internal dependencies managed by Testcontainers. Because hundreds of services lacked this test harness, Grab implemented an agentic workflow using Claude Code and Model Context Protocol integrations to inspect repositories, generate test boilerplate, and resolve configuration errors. Once test baselines are established, an automated patch-test-compare pipeline updates Dockerfiles, constructs multi-stage builds for necessary dynamic libraries, and creates draft merge requests for human approval.
Jia Yee ChongGrab ·
Palana (Part 2): Architecting isolation, identity, and auditability for AI agents
Grab's Palana platform provisions isolated, Kubernetes-native runtime environments for autonomous AI agents using dedicated per-agent namespaces and role-based access controls. The architecture separates network enforcement across layers, applying Layer 3 and Layer 4 containment with Cilium and NetworkPolicy alongside Layer 7 application filtering evaluated by Open Policy Agent. Agent interactions with large language models route through a LiteLLM proxy wrapper that retrieves credentials from HashiCorp Vault based on Kubernetes pod context rather than client headers. Secrets management is divided between directly readable agent paths and proxy-only placeholder paths that prevent raw tokens from residing in runtime filesystems. An automated reaper monitors multi-source activity signals to shut down idle compute resources while preserving persistent storage and configuration state.
Kevin LittlejohnNetflix ·
The Data Canary: How Netflix Validates Catalog Metadata
A manual mitigation action during an incident corrupted a data feed for a subset of titles, causing playback issues and catalog service failures that existing code canary systems failed to catch. To protect streaming reliability, Netflix built an automated data canary system that validates transformed catalog metadata prior to publication. The architecture utilizes a dedicated orchestrator alongside permanent baseline and canary service clusters to coordinate validation using real production traffic. By leveraging custom chaos experiment thresholds, sticky session affinity, and Starts Per Second playback metrics, the system detects regressions in under ten minutes and blocks publication automatically. Controlled failure injection experiments routing approximately 0.2% of global traffic confirmed that issues could be identified in 2.5 to 4 minutes.
Netflix Technology BlogDispatches from O'Reilly: From capabilities to responsibilities
High-stakes AI agents capable of mutating external state often face governance failures when relying on system prompts or manual Human-in-the-Loop approval queues that quickly degrade into alert fatigue. The Responsibility-Oriented Agent architecture addresses this operational bottleneck by shifting system design from open-ended capability framing to deterministic, contract-enforced responsibilities. Under this model, underlying orchestration frameworks like LangChain operate in User Space with their side-effecting tools removed, isolating the agent to epistemic reasoning. The agent expresses its intended action exclusively by emitting a structured policy proposal to a privileged Kernel Space runtime. The runtime deterministically evaluates the proposal against versioned YAML contracts registered in an agent registry, ensuring that only genuine policy exceptions are escalated to human supervisors.
Artur HukGrab ·
Palana (Part 1): Why Grab built a secure platform for autonomous AI Agents
Autonomous AI agents introduce significant operational and security risks when granted network access, persistent state, and credentials. To address these concerns without impeding developer productivity, Grab created Palana, an in-house Kubernetes-native execution substrate. The platform isolates each agent workload within its own namespace, pairing it with dedicated storage, network policies, and role-based access control. Network egress is funneled through an Envoy and Open Policy Agent proxy layer that audits requests and injects credentials from HashiCorp Vault using placeholder tokens, keeping raw secrets outside the agent runtime. This design allows Grab to securely host hundreds of long-running workflows, remote coding environments, and automation bots.
Kevin LittlejohnThe new bottleneck
AI coding tools have significantly lowered the cost of generating software, yet many engineering organizations fail to realize overall delivery speed improvements. Applying the Theory of Constraints reveals that eliminating code production as a bottleneck shifts inventory directly into surrounding legacy processes that remain unadjusted. New friction points consistently emerge across underspecified requirements, prolonged design handoff gates, senior engineer review capacity, and external sign-offs from legal or security. Organizations can address these blockers by interrogating legacy agile ceremonies checkpoint by checkpoint to determine if their original constraints still exist. Practical remedies include adopting real-time co-development between product and engineering, treating initial designs as fluid starting points, and restructuring processes around running rapid, high-volume experiments.
Eira MayAI agents are a confused deputy with the keys to your kingdom
Attackers recently compromised over twenty thousand Instagram accounts by manipulating Meta's AI support assistant to rebind recovery email addresses without verifying account ownership. This incident illustrates the classic confused deputy security problem, where a privileged process is persuaded by an unprivileged user to perform unauthorized operations. Because large language model interfaces operate purely on natural language and cannot distinguish instructions from data, the model itself cannot serve as an authorization boundary. Securing AI agents requires verifying caller identity through external policy checks against authenticated sessions rather than relying on chat context or prompt engineering. Teams must enforce least privilege with short-lived scoped credentials, place irreversible actions behind hard policy gates or human approvals, and maintain audit trails of agent actions.
Fabio SalvadoriBuilding AI Agents for AR Glasses and XR Devices with NVIDIA XR AI
Building AI experiences for augmented reality and wearable devices requires bridging hardware with live media streams, multimodal models, enterprise tools, and runtime infrastructure. NVIDIA XR AI provides an open-source, modular framework connecting extended reality headsets and smart glasses to GPU-accelerated services across cloud, edge, and workstations. In this architecture, camera frames and microphone audio ingest into an XR Media Hub that routes data while keeping raw video pixels in shared memory to minimize overhead. The ecosystem uses NVIDIA Cosmos models for vision-language grounding, NVIDIA Nemotron models for reasoning and tool invocation, and the Model Context Protocol for enterprise integrations. Optional agent orchestration via NVIDIA NeMo Agent Toolkit and spatial streaming through NVIDIA CloudXR support complex workflows across healthcare and manufacturing.
Greg BarboneNVIDIA Blackwell Tops MLPerf Training 6.0 with Industry-Leading Scale and Performance
NVIDIA submitted results across all MLPerf Training v6.0 benchmarks, setting performance records on workloads including the DeepSeek-V3 and GPT-OSS-20B Mixture of Experts models. To scale training up to 8,192 Blackwell GPUs, NVIDIA combined hardware cluster designs with scale-out networking via Spectrum-X Ethernet and Quantum InfiniBand. The engineering team eliminated CPU-GPU synchronization bottlenecks in token-dropless MoEs by implementing full-iteration CUDA graphs with synchronization-free operators and paged stashing. Further software optimizations included CuTe DSL kernel fusions, an MXFP8 attention block in the Transformer Engine, and transitioning router elementwise math to FP32. Across these benchmarks, the GB300 NVL72 platform achieved the fastest time to train at scale and the highest normalized per-accelerator performance.
Farshad GhodsianBuild On-Device AI Companions with the NVIDIA ACE Game Agent SDK and Unreal Engine 5 Plugins
Game developers integrating on-device artificial intelligence companions face challenges such as conversational latency, state synchronization, and loop prevention. NVIDIA announced the open-source ACE Game Agent SDK alongside a suite of Unreal Engine 5 plugins to streamline native, hardware-accelerated character pipelines. The SDK offers Agent, Chat, and RAG APIs, enabling characters to execute multi-step tool-assisted reasoning and query game databases. Unreal Engine 5 plugins introduce local runtime models including nemo-conformer-ctc-120m for speech recognition, Qwen 3.5 4B for language generation, and Chatterbox Turbo 350M for speech synthesis. Additional tooling expands to motion generation via Animotive Kimodo and rendering updates with the DLSS 4.5 Unreal Engine plugin.
Phillip SinghDropbox ·
How Dropbox uses MCP and Dash to close the design-to-code security gap
Dropbox developed a system using the Model Context Protocol (MCP) and Dash's semantic search to bridge the gap between security threat models and code implementation. By retrieving original security documents during pull requests, an LLM agent automatically evaluates whether the proposed code adheres to previously agreed-upon security requirements. This approach surfaces design regressions and missing controls that traditional static analysis tools miss.
Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan MishraSentry ·
Better, faster, less wrong: Enhancing issue grouping
Sentry upgraded its AI-driven issue grouping system to better prevent duplicate issues without merging distinct application errors. The team trained lightonai/modernbert-embed-large using Matryoshka Representation Learning on hundreds of thousands of stacktrace pairs labeled by Claude Sonnet 4.5. To optimize the high-throughput ingestion pipeline, embeddings were truncated from 768 to 64 dimensions, combined with bfloat16 precision, PyTorch SDPA, and CUDA graph compilation. A phased live rollout used threshold-gated index backfilling and automatic fallback to prevent issue spikes during migration. In production, the v2 model cuts the overgrouping rate from 8% to 4%, increases prevented duplicate issues to 70%, and delivers 6x faster inference.
Kush Dubey, Yuval MandelboumSlack ·
Agentic Testing: Where Agents Fit in the E2E Testing Stack
Traditional end-to-end tests validate rigid user journeys, whereas agentic tests verify whether broad goals can be achieved by adapting actions dynamically. To evaluate agentic testing tradeoffs, researchers executed over 200 runs across Playwright Model Context Protocol (MCP), Playwright CLI, and agent-generated Playwright tests using Claude models. Playwright MCP demonstrated high reliability with failure rates of 0% on simple thread replies and approximately 12% on complex search discovery flows. Playwright CLI and generated code struggled more on complex workflows, exhibiting failure rates of approximately 20% and 48% respectively. Although generated tests were faster with average runtimes of roughly three minutes, agentic testing provides a distinct exploratory layer atop deterministic CI test suites.
Sergii GorbachovSentry ·
Catch visual regressions with Snapshots, now in beta
Sentry has released the beta version of Sentry Snapshots to catch visual regressions on pull requests across any platform with a frontend. The tool captures screenshots of an application across various viewports, themes, and languages, compares them against a baseline, and blocks pull requests when diffs are detected. Teams can attach custom context metadata, including test file paths, to help developers and AI agents locate test sources and generate additional test cases. By leveraging the sentry-mcp integration, REST API, and sentry-cli, workflows can retrieve base images, diff masks, and run local diff verification before pushing commits. Originally created at Emerge Tools for mobile applications, this iteration expands automated visual testing capabilities to broader frontend codebases.
Max TopolskyLyft ·
Metric Semantic Layer: How Lyft Governs and Scales Key Data Definitions
As Lyft scaled, different teams developed conflicting definitions for key business metrics due to the lack of centralized version control and shared standards. To resolve this, Lyft implemented an internal Metric Semantic Layer as a Python package that stores authoritative metric definitions in YAML files with Jinja SQL templates. The system restricts onboarding to Golden Metrics used across multiple applications, requiring team-based approval from both Business and Operational Owners for any definition changes. Standardized definitions are exposed through Python APIs, integrated into the Amundsen data catalog and self-service user interfaces, and surfaced to AI tools via a Model Context Protocol.
Iraklikhorguani