Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,671 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Compressing Streaming Neural Audio Encoders via Latent-Space Distillation
Apple’s on-device dictation tokenizer maps waveform to latent representations for a sparsely activated language model, and compressing it reduces memory and latency demands. The authors compress it via latent-space distillation, using the teacher’s pre-quantizer latent as the supervision target rather than tokens or output distributions. A student encoder is trained to regress the teacher’s per-frame latent with a squared-error objective, and an affine layer absorbs width mismatch. This recipe works for both token interfaces and for tokenizers pretrained alone or jointly with a language model, requiring no fine-tuning. At 2.8x compression, the student stays within 1.9% relative WER of its teacher on five of six pairs and outperforms an independently trained tokenizer of identical capacity by 3.9% relative.
Welcoming Gaby Garcia, our new Recruiter
Resend announced the hiring of Gaby Garcia as a new full-cycle recruiter who combines a people-first approach with strategic sourcing and a passion for creating thoughtful, scalable recruiting processes. Garcia holds a background in Psychology, which she says brings curiosity and empathy to her work, and she continually looks for ways to improve how teams hire. She spent the last five years at Automattic hiring engineers and designers, making her well acquainted with developers and the DevTools space. Garcia expressed excitement about Resend’s focus on culture and growing together, stating she loves helping people find a place where they can do meaningful work and feel like they belong. She aims to help Resend grow while being part of that kind of team herself, bringing her expertise to the company’s hiring process.
Bringing Private Processing to Meta AI Glasses
Meta AI glasses require cloud processing for advanced AI features like streaming transcription and long-term recall, but traditional cloud architectures expose data during use. Private Processing uses confidential virtual machines with Trusted Execution Environments (TEEs) on CPUs and GPUs to keep data encrypted even while being computed. Before sending context, glasses perform remote attestation, verifying the server’s TEE certificate against a public transparency ledger that records all deployed binary images. Non-targetable routing via blind-signed tokens and third-party OHTTP relays prevents tying requests to specific users, while encrypted storage ensures data is only accessible with user-provided keys. This infrastructure, already proven for WhatsApp and Meta AI, extends to glasses, with security audits by firms like NCC Group and a bug bounty program for independent research.
Introducing NV-Reason-CT Open 3D CT VLM for Radiologist Chain-of-Thought Reasoning
NVIDIA released NV-Reason-CT, a vision language model for 3D CT analysis that processes volumes as true 3D input rather than independent 2D slices. The model generates structured diagnostic reports covering 30 chest and 29 abdominal abnormalities and emulates radiologist chain-of-thought reasoning with systematic anatomical reviews and uncertainty articulation. Its architecture pairs a Qwen3.5-4B language model with a full 3D ViT encoder that processes volumes resampled to 192³ voxels at 2mm isotropic resolution using 8x8x8 patch tokens and 3D MRoPE for spatial token relationships. Training uses supervised fine-tuning on roughly 550,000 QA examples from CT-RATE, NIH CT, and CancerVerse followed by Group Relative Policy Optimization that refines reasoning quality based on diagnostic accuracy. NV-Reason-CT is an open research foundation not a cleared clinical product and sits within NVIDIA’s Medical AI ecosystem alongside NV-Reason-CXR, NV-Segment-CTMR, and NV-Generate-CTMR.
Claude discovers a novel enzyme system
Anthropic launched a new life sciences research group to use AI agents for fundamental biology discovery. In one of their first programs, Claude agents autonomously analyzed over 200,000 reverse transcriptase sequences from a DNA database. After 21 hours and 210 million tokens, one agent spotted a repeating DNA pattern next to a reverse transcriptase gene, which it identified as a CRISPR-like system. The system, called ART (array-associated reverse transcriptases), consists of the RT, an accessory protein, and an array of non-coding DNA repeats that are expressed as distinct small RNAs. Its function remains unknown, but the pattern of repeats is only found together in other programmable DNA-cutting systems, suggesting ART may be a new tool for biotechnology.
Validate GPU Cluster Readiness Before AI Workloads Land
GPU clusters can pass standard health checks yet fail under real AI training workloads due to a single degraded GPU, network link, or misrouted traffic. NVIDIA Cluster Readiness Engine (NVCRE) is an open source Kubernetes controller that runs real distributed workloads across topology-aware node groups before production jobs land, measuring results and reporting which nodes failed each test. Its layered API uses Custom Resource Definitions for Certification, Workflow, and Job resources, attributing each failure to a specific node and category such as NCCL bandwidth or training goodput. The WorkloadRun API handles platform detection, NCCL settings, shared-memory volumes, and gang scheduling, so operators do not need to write those manifests by hand. NVCRE integrates with NVSentinel for passive telemetry and is available under Apache 2.0, with a roadmap for inference and automated lifecycle validation.
How to Use NVIDIA Warp and MjWarp to Accelerate Robotics Simulation and Learning Workflows
NVIDIA Warp is a Python framework for writing GPU-accelerated kernels that compiles statically typed Python to CUDA. MuJoCo Warp (MJWarp) implements MuJoCo physics on Warp, enabling batched simulation of hundreds to thousands of independent robot environments on NVIDIA GPUs. The core API transition is small: mjw.put_model and mjw.put_data replace mujoco.MjModel and mujoco.MjData, and mjw.step advances the entire batch. Proper throughput measurement requires GPU synchronization and warm-up to avoid timing queue latency instead of actual GPU work. The post demonstrates moving an SO-101 follower arm from a single CPU world to 2,048 parallel MJWarp environments, validating parity and measuring aggregate world-steps per second.
Rendering huge pull requests in the GitHub Copilot app
Rendering a pull request with over a million changed lines and hundreds of threaded comments requires a fundamentally different architecture than a code-only diff. The standard virtualized approach fails because comment heights depend on render-time factors like markdown wrapping, expandable sections, and reply composers. The team split the document’s height into two independent domains: deterministic code geometry and dynamic block geometry for comments. A single idle- and scroll-gated measurement pass, scoped to the viewport, corrects heights without causing reflows mid-scroll. The result is a pull request view where a million-line diff with hundreds of comments opens, scrolls, and behaves like a normally sized pull request.
Manage Kubernetes Node Fleets with NodeWright
Managing Kubernetes node fleets, especially GPU nodes, is difficult because host-level changes can disrupt long-running training jobs, and traditional tools are not cluster-aware. NodeWright, an open-source Kubernetes-native package manager from NVIDIA, solves this by declaratively configuring and updating node operating systems while respecting PodDisruptionBudgets and non-interruptible workload labels. The operator orchestrates a sequence of cordon, wait, drain, apply, interrupt, and uncordon on each node, using container image packages that carry modifications such as kernel parameters, security agents, and verification scripts. It supports progressive rollouts across a fleet via DeploymentPolicy compartments with configurable disruption budgets and strategies like batch, canary, and rolling. NodeWright integrates with NVIDIA AI Cluster Runtime for validated configurations and is part of the DSX OS open-source portfolio, available under Apache 2.0.
How Concurrence governs clinical AI at a trillion-token scale with Unity Gateway
Concurrence processes 100.8 billion input tokens and 11.2 million LLM calls every 30 days, at an annualized rate of 1.2 trillion tokens. To govern these workflows, the company consolidates on Databricks with Lakebase for operational state, Unity Catalog for governance, and Unity Gateway for centralized AI access. Patient data is recorded as immutable events in Delta tables, from which a current patient state—called the world model—is computed while preserving provenance. All agents are tested against simulated patients before deployment; simulation traffic is approximately seven times production traffic, using immutable event history to replay states. Concurrence routes all coding-agent traffic through Unity Gateway, attributing each request to the engineer, and enforces compliance-gated model routing where HIPAA requirements are met first.
Unity Catalog Pages: a governed home for your business knowledge in Genie Ontology
Business concepts like active customer or completed trip are often defined differently across teams, scattered in Slack threads and spreadsheets, causing AI agents to guess confidently when answering questions. Databricks introduces Unity Catalog Pages as a governed home within Genie Ontology where data stewards with Genie Code define the authoritative meaning of those terms and entities. Pages complement existing semantics — metric views how to calculate, domains where to look, and certification what to trust — by capturing what each concept means. Genie Code can bootstrap Pages by extracting definitions from Confluence, Slack, Google Docs, or GitHub via MCP-connected tools, turning weeks of work into minutes. When Genie One uses a Page to ground an answer, the Page is cited as a clickable source so users can inspect the definition and verify the reasoning.
How I massively improved my AI inference performance without buying new hardware
Default inference configurations for large language models often leave significant performance unused, even on expensive hardware like 16 NVIDIA H200 GPUs across two nodes. The author tested this setup with vLLM and Red Hat AI Inference 3.4 on a CoreWeave Kubernetes cluster, using enterprise RAG workloads with 2,000-token inputs and 100 concurrent users. By systematically comparing aggregated and prefill-decode (PD) disaggregated architectures, and tuning tensor parallelism (TP) and PD split, latency was dramatically reduced. For Qwen3-32B, time-to-first-token (TTFT) on the best aggregated configuration (2xTP8) dropped from 995ms to 304ms, and with optimized routing to 287ms — a 71% improvement without hardware changes. For Llama-70B, TTFT fell from 1,292ms to 463ms, a 64% gain. The author also built ServeIt Studio, an open-source tool that automates the full optimization pipeline on Kubernetes.
Build OpenJDK container images locally using the standalone S2I tool
Red Hat provides the Red Hat build of OpenJDK for running Java applications on Red Hat OpenShift 4. Developers can use the standalone Source-to-Image (S2I) CLI to build and test OpenJDK container images locally before deploying to a cluster. S2I produces reproducible images by injecting source code into a builder image and running its assemble script. The toolkit supports flags such as –loglevel for detailed output, –environment-file to pass custom JVM settings like G1GC and MaxRAMPercentage, and –as-dockerfile to output a Dockerfile. After building, the image can be tested locally with docker run, then uploaded to the cluster for deployment.
Small Talk With Prasun Kumar, CEO and Founder of Oppex AI
Oppex AI builds AI agents that help developers resolve production incidents by gathering context before the engineer is paged. The agents collect logs, cloud metrics, database health, affected customers, recent code changes, and past incident history, then present a recommendation. Because developer logs are not plain English, the team constructs a knowledge graph of how the entire system connects. They harden the agent by pitting it against a chaos monkey that breaks test systems without revealing the method, forcing the agent to learn. The 15-person team has launched the product with early enterprise customers and aims to bring mean time to resolve under 10 minutes, eventually moving toward autonomous system healing.
**Know Who Spoke When: Build Real-Time, Multi-Speaker AI with NVIDIA Nemotron 3 Diarization**
NVIDIA Nemotron 3 Diarization is an open-weight, 100M-parameter model that ranks #1 on VoiceArena’s Diarization-Bench leaderboard with a 14.72% Diarization Error Rate (DER). It classifies who spoke when in conversations, handling overlapping speech and supporting up to eight speakers across live and recorded audio. The model uses a Sortformer approach with arrival-order speaker labeling, a 31-layer Transformer encoder, and two memory mechanisms for streaming inference. It accepts 16 kHz single-channel audio and outputs a floating-point tensor of speaker-activity probabilities at a configurable time step. Developers can combine its timestamps with automatic speech recognition to produce speaker-attributed transcripts, with recommended input-buffer latencies ranging from 0.32 to 30.4 seconds.
100 Exercises to Learn Rust, Updated
The post announces a major update to the free JetBrains Academy course ‘100 Exercises to Learn Rust,’ which runs inside RustRover. The course teaches Rust through hands-on exercises building a ticket management system across eight chapters from ownership to futures, assuming fluency in another language. The update adds a rebuilt introduction, an FAQ, a final challenge turning the system into an async REST API by editing Cargo.toml and verifying with a requests.http file, visible test assertions, and a dev/release profile experiment. A ‘fast track’ lets experienced users skip to the final challenge to quickly see which areas need revisiting. The post advises using AI for explanations, not code generation, and turning off AI code completion during exercises to maximize learning from getting stuck and working out solutions.
What Dropbox has learned from deploying AI at company scale
Dropbox has learned that deploying AI at scale requires examining workflows end to end, addressing new bottlenecks like too many code reviews or infrastructure strain. The company uses a framework measuring speed, effectiveness, quality, and impact, with proxy metrics including pull request throughput and change failure percentage. Dropbox reports around 70% AI-generated code, comparable to top technology peers, and its pull request throughput ranks in the top 5% of a custom benchmark while change failure percentage is at the 75th percentile. The company connects AI token usage to engineering value through its internal agent service Nova, and it uses ground-up initiatives like boot camps and aligned incentives to drive adoption. Human skills like problem solving, judgment, and leadership become more critical as AI lowers the cost of producing software, and Dropbox aims to extend AI workflows beyond engineering to product, design, and customer experience teams.
ClickHouse release 26.9
ClickHouse 26.9 ships with 56 new features, 135 performance optimizations, and 464 bug fixes. Among the highlights are conditional LIMIT boundaries that control output based on ordered stream values, incremental refreshes for append-only materialized views, and disk spilling for high-cardinality DISTINCT queries that automatically spills when memory usage reaches half the limit. The CREATE TOKEN feature issues time-limited credentials with a default lifetime of 30 minutes and restricted privileges, reducing the impact of leaked keys. Arithmetic operations between DateTime and Time types now support configurable overflow behaviors—throw, saturate, or ignore. PromQL support expands in a private preview on ClickHouse Cloud, bracket syntax simplifies JSON subcolumn access, and queries for min, max, and count receive performance improvements.
How to Guide Your Language Flow
Probe guidance, a novel method for guiding flow matching models, is introduced. It uses the frozen internal states of an existing diffusion model to construct a guidance signal, operating on a similar principle as autoguidance but removing the need for an additional forward pass at inference time. The method provides a reliable path ensuring the weak and strong models share similar dynamics. Probe guidance sets a new state-of-the-art performance on unconditional generation when benchmarked on continuous diffusion language models. Applied to a 1.7B diffusion language model, it consistently improves multiple choice question answering benchmarks.
How to serve trillions of tokens for trillion-parameter coding agents
Serving inference for coding agents requires trillion-parameter sequence models like Moonshot AI’s Kimi K2.6 to process trillions of input and output tokens at extremely high relative and absolute performance. To be economically viable, these services must achieve large fractions of hardware peak performance, using contemporary matrix math accelerators like Tensor Cores at petaFLOP per second scale. Modal optimized by batching multiple sequences together, caching KV state, and patching SGLang with NVFP4 micro-scaling on Blackwell GPUs like the B200. Per-replica performance improved 2.8x per user and 5.6x across users, transforming ruinously expensive into a price-competitive service. One deployment processed hundreds of billions of tokens daily and trillions total, showing that careful engineering and benchmarking are key to scaling inference.
Top topics
- Architecture834
- Performance816
- LLMs660
- Machine Learning622
- Open Source563
- AI Agents553
- AI470
- Developer Experience448
Weekly digest
The best engineering reads, once a week.