--- title: "Latest reads" description: "The engineering internet, summarised so you can actually read it." --- # Latest reads > The engineering internet, summarised so you can actually read it. ## Articles ### [Continuous batching from first principles](https://yomu.fyi/post/continuous-batching-from-first-principles.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Rémi Ouazan Reboul, Arthur Zucker, Luc Georges - Published: Nov 25, 2025 Large language model serving requires running expensive next-token generation across multiple concurrent user requests. Traditional batching approaches introduce severe padding inefficiencies when mixing variable-length prompts and different generation phases, especially under static shape constraints like CUDA graphs. Continuous batching resolves these inefficiencies by combining key-value caching, chunked prefill, ragged batching, and dynamic request scheduling. Ragged batching eliminates the traditional batch axis by concatenating token sequences into a single tensor and using boolean attention masks to isolate independent sequences. By dynamically removing completed prompts and packing decoding tokens alongside chunked prefill tokens up to a hardware memory budget, serving systems maintain high hardware utilization and throughput. ### [Building Deep Research: How we Achieved State of the Art](https://yomu.fyi/post/building-deep-research-how-we-achieved-state-of-the-art.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Michael Griff, Dean Sacoransky, Noah Nefsky - Published: Nov 24, 2025 Building production AI research agents presents challenges around context window pollution, escalating token costs, and architectural brittleness across model updates. To resolve these issues, Tavily rebuilt its deep research system around simplified orchestration, compact tooling, and active context curation. The agent mimics human research workflows by distilling tool outputs into concise reflections for ongoing reasoning, withholding raw retrieved web data until the final deliverable stage. Compared to traditional ReAct propagation architectures where token consumption scales quadratically, this reflection-based approach achieves linear token growth. As a result, the system reduced token consumption by 66% compared to Open Deep Research while reaching state-of-the-art performance on DeepResearch Bench. ### [OVHcloud on Hugging Face Inference Providers 🔥](https://yomu.fyi/post/ovhcloud-on-hugging-face-inference-providers.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Gilles Closset, Fabien Ric, Elias Tourneux - Published: Nov 24, 2025 OVHcloud is now integrated as a supported Inference Provider on the Hugging Face Hub, expanding serverless inference options across model pages and client SDKs. The integration enables access to open-weight models, including gpt-oss, Qwen3, DeepSeek R1, and Llama, using European infrastructure with pay-per-token pricing starting at €0.04 per million tokens. Developers can connect via Python and JavaScript SDKs using either direct custom API keys or automatic routing through Hugging Face tokens. The service supports structured outputs, function calling, multimodal workflows, and embedding models while providing sub-200ms first-token response times. Calls routed through Hugging Face pass through standard provider pricing without markups, and PRO tier subscribers receive monthly inference credits. ### [SpellVault’s evolution: Beyond LLM apps, towards the agentic future](https://yomu.fyi/post/spellvault-s-evolution-beyond-llm-apps-towards-the-agentic-future.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Felix Haryanto Lie - Published: Nov 21, 2025 Grab developed SpellVault as an internal no-code platform to democratize the creation of AI applications backed by Retrieval-Augmented Generation (RAG) and plugin integrations. To advance beyond static retrieval and linear input-output processing, the platform transitioned from its legacy executor to a graph-based execution model supporting branching, looping, and ReAct agent patterns. Capabilities like Python code execution and internal repository searching were unbundled from the prompt builder and consolidated alongside user plugins into unified Native and Community Built Tools. The platform also introduced a drag-and-drop deterministic workflow designer, automated task scheduling, and support for the Model Context Protocol (MCP). ### [20x Faster TRL Fine-tuning with RapidFire AI](https://yomu.fyi/post/20x-faster-trl-fine-tuning-with-rapidfire-ai.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Kamran Bigdely, Arun Kumar, Quentin Gallouédec - Published: Nov 21, 2025 Hugging Face TRL now officially integrates with RapidFire AI to accelerate LLM fine-tuning and post-training experimentation through concurrent configuration evaluation. Instead of executing training configurations sequentially, the platform shards datasets into discrete chunks and cycles multiple model configurations across GPUs at chunk boundaries. The runtime handles automatic checkpointing via a shared-memory-based adapter and model spilling mechanism to maintain consistent training state. Experimenters can monitor live metrics on an MLflow-based dashboard and use Interactive Control Ops to stop underperforming runs or clone and warm-start promising configurations mid-flight. Benchmark results on NVIDIA A100 GPUs demonstrate experimentation speedups ranging from 15x to 20x over sequential runs when training TinyLlama-1.1B and Llama-3.2-1B models. ### [Open ASR Leaderboard: Trends and Insights with New Multilingual & Long-Form Tracks](https://yomu.fyi/post/open-asr-leaderboard-trends-and-insights-with-new-multilingual-long-fo.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Eric Bezzam, Steven Zheng, Eustache Le Bihan, Vaibhav Srivastav - Published: Nov 21, 2025 The Open ASR Leaderboard expanded its evaluation framework by introducing dedicated multilingual and long-form transcription tracks across eleven datasets. Benchmarking more than sixty open- and closed-source models from eighteen organizations revealed clear operational trade-offs between accuracy and inference speed. Systems combining Conformer encoders with large language model decoders achieved the lowest word error rates in short-form English speech recognition. In contrast, connectionist temporal classification and token-and-duration transducer decoders yielded tenfold to hundredfold improvements in throughput with only slight accuracy penalties. While closed-source commercial systems maintained performance leads in long-form audio transcription, specialized single-language models consistently outperformed generalized systems supporting hundreds of languages. ### [Introducing AnyLanguageModel: One API for Local and Remote LLMs on Apple Platforms](https://yomu.fyi/post/introducing-anylanguagemodel-one-api-for-local-and-remote-llms-on-appl.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Mattt - Published: Nov 20, 2025 Apple developers frequently face integration friction when trying to support a mix of local and cloud language models across disparate APIs. To resolve this fragmentation, AnyLanguageModel introduces a unified Swift package that acts as a drop-in replacement for Apple's Foundation Models framework. The package standardizes interactions across local backends such as Core ML, MLX, and llama.cpp alongside cloud providers like Anthropic and OpenAI. To avoid dependency bloat from multi-backend support, the library employs Swift 6.1 package traits so projects only import the specific runtime engines they require. Furthermore, the library extends beyond current Foundation Models limitations by adding prompt image support for vision-language models. ### [Apriel-H1: The Surprising Key to Distilling Efficient Reasoning Models](https://yomu.fyi/post/apriel-h1-the-surprising-key-to-distilling-efficient-reasoning-models.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Torsten Scholak, Oleksiy Ostapenko, Raymond Li, Luke Kumar, Joel Lamy-Poirier - Published: Nov 19, 2025 Retrofitting efficiency into existing large language models often fails when naive distillation disrupts complex multi-step reasoning mechanisms. The Apriel-H1 project addresses this challenge by converting a 15B full-attention reasoning model into a Mamba hybrid using targeted distillation on high-quality SFT reasoning traces rather than diffuse pretraining data. Built on the open-source Fast-LLM framework, the process applies reverse KL divergence alongside a staged layer replacement strategy combining Leave-One-Out analysis and dynamic loss evaluations. Across fifty total blocks, replacing thirty attention layers with Mamba mixers yielded the flagship Apriel-H1-15b-Thinker-SFT checkpoint. This configuration achieved a 2.1x throughput increase with minimal benchmark degradation after 76.8B training tokens. ### [Easily Build and Share ROCm Kernels with Hugging Face](https://yomu.fyi/post/easily-build-and-share-rocm-kernels-with-hugging-face.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Abdennacer Badaoui, Daniel Huang, colorswind, Zesen Liu - Published: Nov 17, 2025 Compiling custom deep learning GPU kernels across architectures often leads to complex build flag management, compiler errors, and ABI issues. Hugging Face's kernels library and kernel-builder streamline this workflow by supporting backends such as ROCm, CUDA, Metal, and XPU. Using the RadeonFlow FP8 block-wise GEMM kernel optimized for the AMD Instinct MI300X GPU as an example, developers organize files into specific headers and HIP implementations. The build is configured via a build.toml manifest and flake.nix to ensure reproducible compilation across supported PyTorch and ROCm versions. Artifacts can then be pushed to the Hugging Face Hub, allowing users to load and execute custom operators directly via get\_kernel without traditional installation steps. ### [Join the AMD Open Robotics Hackathon](https://yomu.fyi/post/join-the-amd-open-robotics-hackathon.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Eric Ma, Guruprasad MP - Published: Nov 13, 2025 AMD, Hugging Face, and Data Monsters announced the AMD Open Robotics Hackathon, an in-person competition scheduled for Tokyo and Paris in December 2025. Participating teams of up to four adults will compete over three days across two core missions. The first mission covers an instructor-led session on configuring the LeRobot development environment using AMD AI solutions. During the second mission, teams spend two days developing a freestyle project to address a real-world problem using LeRobot. Participants receive access to SO-101 robotics kits, AMD Ryzen AI laptops, and AMD Instinct MI300X GPUs via AMD Developer Cloud, with prizes awarded to the top seven teams in each city based on creativity, difficulty, ease-of-use, and practicality. ### [Treating Data as Code at Two Sigma](https://yomu.fyi/post/treating-data-as-code-at-two-sigma.md) - Company: [Two Sigma](https://yomu.fyi/company/two-sigma.md) - Author: Emily Majewski - Published: Nov 13, 2025 Scaling research and trading platforms across thousands of data sources created bottlenecks at Two Sigma, where reliance on database snapshots and fragmented infrastructure slowed dataset delivery to data scientists. To address operational costs and architectural complexity, the data engineering team adopted software development principles by treating data as code. The organization migrated to Google BigQuery's serverless architecture, standardized SQL transformations using dbt, and defined declarative pipelines with Terraform under continuous integration workflows. Internal tooling was introduced to track directed acyclic graph dependencies, automate anomaly detection, and streamline data discovery. These shifts eliminated manual data movement routines, reduced operational overhead, and enabled formal data contracts to safeguard downstream consumers and emerging language model integrations. ### [Building for an Open Future - our new partnership with Google Cloud](https://yomu.fyi/post/building-for-an-open-future-our-new-partnership-with-google-cloud.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Jeff Boudier, Simon Pagezy - Published: Nov 13, 2025 Hugging Face and Google Cloud announced an expanded strategic partnership designed to streamline the deployment and management of open models on Google Cloud infrastructure. Prompted by a tenfold increase in Hugging Face usage on Google Cloud over three years, the collaboration introduces a joint CDN Gateway using Hugging Face Xet technology to cache models and datasets directly on Google Cloud. This caching mechanism aims to shorten download times, strengthen model supply chain resilience, and accelerate time-to-first-token across Vertex AI, Google Kubernetes Engine, Cloud Run, and Compute Engine virtual machines. Hugging Face plans to integrate native library support for Google Cloud TPUs, lower instance prices on Inference Endpoints, and enhance Hub security scanning via VirusTotal, Google Threat Intelligence, and Mandiant. ### [Grab's Mac Cloud Exit supercharges macOS CI/CD](https://yomu.fyi/post/grab-s-mac-cloud-exit-supercharges-macos-ci-cd.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Madushan Gamage - Published: Nov 6, 2025 Scaling iOS CI/CD workloads across multiple large mobile applications led to substantial expenses and queue delays under rented cloud macOS infrastructure. To mitigate network latency to Git servers and cut compute overhead, Grab relocated its build fleet from a US cloud provider to a colocation data center in Malaysia. The infrastructure deployment relies on over 200 bare-metal Mac minis across 42RU racks, purposefully avoiding virtualization solutions that introduced performance and stability penalties. Phased migration and guardrail monitoring verified cluster stability while achieving 20% to 40% reductions in pipeline and app build times. Operating the regional bare-metal cluster over an estimated three-year hardware replacement lifecycle is projected to yield 2.4 million USD in total cost of ownership savings. ### [How we built a custom vision LLM to improve document processing at Grab](https://yomu.fyi/post/how-we-built-a-custom-vision-llm-to-improve-document-processing-at-gra.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Jia Chen - Published: Nov 4, 2025 Document processing for identity verification across Southeast Asia presents challenges due to varied layouts and non-Latin scripts. Traditional OCR and off-the-shelf vision models struggle with accuracy, high latency, or lack of regional language training data. Grab addressed this by creating synthetic regional datasets, using an automated labeling pipeline named Documint, and evaluating open-source multimodal architectures. After initial LoRA fine-tuning failed on complex scripts like Thai and Vietnamese, full-parameter fine-tuning of Qwen2-VL 2B yielded substantial gains. To optimize deployment costs and latency, the team constructed a custom 1B parameter model pairing a Qwen2-VL vision encoder with a Qwen2.5 0.5B language decoder, achieving performance within 3 percentage points of the 2B model at significantly lower latency. ### [MongoDB Announces Leadership Transition](https://yomu.fyi/post/mongodb-announces-leadership-transition.md) - Company: [MongoDB](https://yomu.fyi/company/mongodb.md) - Author: MongoDB - Published: Nov 3, 2025 Dev Ittycheria announced his decision to retire as MongoDB President and Chief Executive Officer after eleven years of leading the company. Chirantan Desai will become the new Chief Executive Officer effective November 10, 2025. The transition follows a succession planning process initiated earlier this year when Ittycheria concluded he could not commit to an additional five-year term requested by the Board. Desai previously scaled ServiceNow organically from over $1 billion to more than $10 billion in revenue and served as President of Product and Engineering at Cloudflare. Ittycheria will remain on MongoDB's Board to support Desai through the transition into the company's next phase of growth. ### [Aligning to What? Rethinking Agent Generalization in MiniMax M2](https://yomu.fyi/post/aligning-to-what-rethinking-agent-generalization-in-minimax-m2.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: MiniMax - Published: Oct 30, 2025 MiniMax M2 addresses the performance gap between agent benchmark scores and real-world deployment across diverse scaffolding frameworks and environments. While initial alignment efforts relied on scaling tool variety, models continued to fail when subjected to slight variations in prompts, scaffolding, and tool responses. To resolve this instability, the team introduced interleaved thinking, which enables models to execute internal reasoning at arbitrary steps throughout long-horizon tasks to adapt dynamically to external tool outputs. Additionally, developers constructed a data pipeline designed for full-trajectory generalization by introducing perturbations across system prompts, user objectives, environments, and tool feedback. In internal evaluations, MiniMax M2 maintained robust tool-calling and instruction-following performance when tested against unfamiliar, cold-start agent scaffolding frameworks. ### [Machine-learning predictive autoscaling for Flink](https://yomu.fyi/post/machine-learning-predictive-autoscaling-for-flink.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Minh Nhat Nguyen - Published: Oct 30, 2025 Managing resource allocations for expanding Apache Flink stream-processing workloads often leads to over-provisioning or unstable reactive scaling. Reactive autoscaling with Kubernetes Horizontal Pod Autoscaling triggers severe restart spikes because pipelines must reprocess backlogged records from checkpoints, inducing feedback loops and scaling fluctuations. To prevent these spirals, a predictive vertical autoscaling system uses upstream Kafka message throughput as an independent workload metric. The architecture forecasts future workload using time-series models and maps the projected throughput to required TaskManager CPU allocations through a trained regression model. A custom controller vertically scales TaskManager CPU resources before traffic changes arrive, avoiding restart-induced latency spikes. ### [On the Shifting Global Compute Landscape](https://yomu.fyi/post/on-the-shifting-global-compute-landscape.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Tiezhen WANG, Irene Solaiman - Published: Oct 29, 2025 United States export controls on advanced artificial intelligence hardware have catalyzed rapid expansion across China's domestic silicon and open-weight model ecosystem. Faced with restricted access to high-end NVIDIA GPUs, Chinese developers accelerated deployments on domestic accelerators, including Huawei Ascend, Cambricon, and Baidu Kunlun. Hardware scarcity spurred architectural and algorithmic innovations in compute efficiency, notably DeepSeek's Multi-head Latent Attention and Group Relative Policy Optimization, alongside substantial post-training cost reductions. Organizations such as Baidu and Ant Group now train foundation models directly on domestic hardware, fostering non-CUDA software stacks and lowering inference costs globally. Consequently, the global artificial intelligence infrastructure is shifting from an exclusively American-focused paradigm toward a dual-ecosystem landscape powered by domestic chips and open-weight architectures. ### [Building a Healthcare Robot from Simulation to Deployment with NVIDIA Isaac](https://yomu.fyi/post/building-a-healthcare-robot-from-simulation-to-deployment-with-nvidia.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Steven Palma, Andres Diaz-Pinto - Published: Oct 29, 2025 Simulation in healthcare robotics has historically been slow, siloed, and difficult to translate into real-world physical systems. The NVIDIA Isaac for Healthcare v0.4 release introduces an end-to-end SO-ARM starter workflow to streamline simulation, training, and deployment for embodied surgical assistants. The architecture relies on a Sim2Real mixed training approach that fine-tunes the GR00T N1.5 foundation model on combined real-world and synthetic datasets using LeRobot. Synthetic data generated in simulation constitutes over 93 percent of the policy training data to bridge physical data collection constraints. Real-time inference on the SO-ARM101 precision manipulator utilizes RTI DDS communication, dual-camera vision, and TensorRT model conversion. ### [How to Build a Healthcare Robot from Simulation to Deployment with NVIDIA Isaac for Healthcare](https://yomu.fyi/post/how-to-build-a-healthcare-robot-from-simulation-to-deployment-with-nvi.md) - Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md) - Author: Asawaree - Published: Oct 28, 2025 Simulation in healthcare robotics has historically been slow, siloed, and difficult to translate to physical hardware. NVIDIA introduced the Isaac for Healthcare v0.4 developer framework to streamline this lifecycle through an end-to-end SO-ARM starter workflow. Developers collect mixed real-world and synthetic datasets using LeRobot and SO-101 teleoperation, with synthetic generation accounting for over 93 percent of training data. The pipeline post-trains GR00T N1.5 models using dual-camera vision inputs, evaluates policies in Isaac Lab, and deploys them to physical hardware with RTI DDS communication. This integrated approach allows surgical assistant robots to interpret natural language instructions and execute physical manipulation tasks reliably. [Newer posts](https://yomu.fyi/page/25.md) · [Older posts](https://yomu.fyi/page/27.md)