Loading…
Latest reads
The engineering internet, summarised so you can actually read it.
MongoDB ·
Token-count-based Batching: Faster, Cheaper Embedding Inference for Queries
Serving embedding models for short search queries often suffers from poor GPU efficiency because traffic is spiky and memory-bound. Traditional time-window and request-count batching strategies lead to inconsistent GPU utilization, while tensor padding wastes compute on empty tokens. To resolve these bottlenecks, Voyage AI implemented token-count-based batching paired with padding removal in inference engines like vLLM. The architecture uses Redis with Lua scripts to atomically aggregate pending requests until reaching an optimal hardware saturation token threshold. Across production deployments, this approach achieved a 50% reduction in GPU inference latency with 3X fewer GPUs and improved throughput by up to 8×.
Chengcheng Pei, Yuan LinGrab ·
How Grab is accelerating growth with real-time personalization using Customer Data Platform scenarios
Grab previously relied on daily batch attribute updates in its Customer Data Platform, which created engineering bottlenecks and hindered time-sensitive engagement opportunities. To address this limitation, the team introduced Scenarios, a self-serve real-time personalization capability embedded within the platform. The architecture processes real-time event triggers from Grab's Scribe platform using Apache Flink, enriches incoming events with historical context from StarRocks, and evaluates pre-trained machine learning classifiers. Computed outputs sync to Kafka streams or Amphawa, an internal feature store powered by AWS DynamoDB, maintaining end-to-end latencies under fifteen seconds. Across more than a dozen production deployments, including real-time interventions for subscription abandonment within fifteen minutes, the platform achieved over a 3% conversion uplift compared to batch campaigns.
Saubhagya AwaneeshTokenization in Transformers v5: Simpler, Clearer, and More Modular
Transformers v5 overhauls its tokenization framework by separating tokenizer architecture from trained vocabularies. In contrast to v4's dual slow Python and fast Rust files, v5 consolidates each model tokenizer into a single file defaulting to the Rust-backed TokenizersBackend. The pipeline stages—normalizer, pre-tokenizer, model algorithm such as BPE or Unigram, post-processor, and decoder—are now directly exposed and configurable rather than buried in serialized files. Practitioners can instantiate blank tokenizer architectures and train custom vocabularies directly from iterators using native methods like train_new_from_iterator while retaining model-specific formatting rules. The wrapper layer continues to bridge raw tokenization and model requirements by managing chat templates, context limits, and special token insertion.
Ita Zaporozhets, Aritra Roy Gosthipaty, Arthur Zucker, Sergio Paniego, merve, Pedro CuencaThe Open Evaluation Standard: Benchmarking NVIDIA Nemotron 3 Nano with NeMo Evaluator
Assessing whether large language model improvements stem from genuine advances or underspecified evaluation conditions remains a major challenge across the industry. Most published model evaluations omit critical execution parameters, prompt templates, harness versions, and runtime configurations. In response, NVIDIA released Nemotron 3 Nano 30B A3B alongside its complete, reproducible evaluation recipe built with the open-source NeMo Evaluator library. The library serves as an orchestration layer that unifies diverse benchmark harnesses under standard configurations while decoupling evaluation logic from underlying inference backends. Developers can execute the identical evaluation pipeline against local deployments or hosted endpoints using published YAML configurations and structured logging.
Seph Mard, Isabel Hulseman, Besmira Nushi, Piotr Januszewski, Grzegorz Chlebus, VivienneZhang, Wojciech Prazuch, Pablo Ribalta, Nik Spirin, Ferenc GalkoLyft ·
From Python3.8 to Python3.10: Our Journey Through a Memory Leak
During an initiative to upgrade Python services from version 3.8 to 3.10, Lyft engineers encountered severe latency spikes and timeouts in one service within their test environment. Profiling DynamoDB queries revealed that gevent thread joins were taking up to 30 seconds while pod memory consumption climbed steadily. To isolate the issue, engineers used an internal tracemalloc-based profiler triggered via SIGUSR2 signals, temporarily disabling Gunicorn preload to prevent workers from terminating prematurely on signal receipt. Memory traces pointed to botocore and an incompatibility between weakref.finalize and gevent monkey patching in urllib3 version 1.26.16, which prevented connections from returning to the pool. Downgrading urllib3 to version 1.26.15 immediately resolved both the timeouts and the memory leak before a permanent fix arrived in gevent and urllib3 updates.
Jay PatelCUGA on Hugging Face: Democratizing Configurable AI Agents
Many existing AI agent frameworks suffer from brittleness, tool misuse, and failures when executing complex workflows. To address these limitations, the open-source Configurable Generalist Agent (CUGA) introduces structured orchestration that decomposes user goals into programmatic subtasks tracked by a dynamic task ledger. The framework integrates agentic patterns like planner-executor and code-act, delegating subtasks to specialized agents that generate pseudo-code before running execution in a secure sandbox. Released under the Apache 2.0 license, CUGA integrates with Langflow for low-code visual workflow assembly and supports multi-tool environments through OpenAPI specs, MCP servers, and LangChain. Testing on inference platforms like Groq with open models such as gpt-oss-120b demonstrates rapid response times during multi-step planning and validation.
Jim Laredo, Avi Yaeli, Sami Marreed, Ayhan Sebin, Merve UnuvarNew in llama.cpp: Model Management
llama.cpp server now includes a router mode that enables dynamic loading, unloading, and switching between multiple LLMs without restarting the server. The architecture runs each model in an isolated process to ensure a single model crash does not affect other active instances. Running llama-server without specifying a model activates auto-discovery across the cache directory or a designated folder of GGUF files. In addition to on-demand loading and least-recently-used eviction capped by default at four models, the server provides endpoints for manual loading, unloading, and listing model statuses. Models inherit global configuration options or use dedicated configuration presets while also integrating directly into the built-in web UI.
Xuan-Son Nguyen, Victor MustarCodex is Open Sourcing AI models
Hugging Face Skills equips AI coding agents like OpenAI Codex to execute end-to-end machine learning workflows. By reading AGENTS.md files and interfacing via the Model Context Protocol, Codex automates dataset validation, training script updates, and job submissions to Hugging Face Jobs. The workflow supports methods including supervised fine-tuning, direct preference optimization, and reinforcement learning for models ranging from 0.5B to 7B parameters. Throughout execution, Codex tracks live metrics via Trackio, records benchmark evaluations against baselines, and maintains Markdown reports. Once training concludes, Codex merges LoRA adapters, applies GGUF quantization, and publishes the resulting models to the Hugging Face Hub for local deployment.
ben burtenshaw, shaun smithIntroducing swift-huggingface: The Complete Swift Client for Hugging Face
Hugging Face released swift-huggingface, a dedicated Swift package offering complete Hub API integration, reliable file downloads, and inference provider access. The library addresses previous limitations in swift-transformers 1.0, where interrupted multi-gigabyte model downloads could not resume and cache structures differed from Python. To resolve cache duplication, swift-huggingface implements a Python-compatible content-addressed storage layout using symlinks and flock file locking. Authentication is standardized through a TokenProvider pattern supporting auto-detection, Keychain integration, static CI/CD tokens, and OAuth 2.0 sign-in with automatic token refresh. The package also provides URLSession-backed snapshot downloads with granular progress tracking and will soon replace the HubApi implementation inside swift-transformers.
MatttDeepMath: A lightweight math reasoning Agent with smolagents
Mathematical reasoning in large language models often suffers from lengthy chain-of-thought traces and frequent arithmetic mistakes. To address these issues, DeepMath pairs a Qwen3-4B Thinking base model with a sandboxed Python execution environment built using the smolagents library and vLLM backend. The system offloads deterministic calculations by emitting concise Python snippets, executing them safely with restricted module imports and no network access, and folding returned values back into the context. Training utilizes Group Relative Policy Optimization on the Tool-Integrated Reasoning subset of OpenMathReasoning with temperature scheduling and length constraints. Across benchmarks including MATH500, AIME, HMMT, and HLE, DeepMath reduces output token lengths by up to 66% while increasing overall problem-solving accuracy.
Daniel Fleischer, Moshe Berchansky, Moshe WasserblatWe Got Claude to Fine-Tune an Open Source LLM
Hugging Face Skills equips coding agents like Claude Code, OpenAI Codex, and Google's Gemini CLI to manage end-to-end language model fine-tuning. Using the hf-llm-trainer skill, an agent validates dataset formats, selects cloud hardware, configures authentication, and submits jobs to Hugging Face Jobs. Supported techniques include Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Group Relative Policy Optimization (GRPO) for models ranging across various parameter sizes. The integration incorporates Trackio for real-time monitoring and automates LoRA configuration for larger models. Once training completes, the agent pushes artifacts to the Hugging Face Hub and can convert models to GGUF format for local execution.
ben burtenshaw, shaun smithEngineering the right opportunities for Thumbtack Pros.
Ashmann Syngle, a backend software engineer on Thumbtack's Pricing team, focuses on systems that connect local service professionals with homeowners. To improve matching and revenue operations, the team recently deployed new pricing features, launched experiments, and partnered closely with Data Science and Monetization Experience groups. Current engineering efforts center on infrastructure enhancements, specifically upgrading alerting and monitoring across owned services to raise system reliability and operational efficiency. Because the team maintains complex monetization systems within a codebase that has evolved over many years, engineers conduct deep system analyses to evaluate edge cases before implementing foundational platform modifications. Thumbtack supports these initiatives through a virtual-first operational model complemented by regular in-person offsites.
AshmannsyngleSlack ·
Streamlining Security Investigations with Agents
Slack's Security Engineering team handles billions of daily security events and needed a reliable way to streamline on-call alert triage. An initial prototype relying on a single 300-word prompt produced inconsistent results and frequently reached spurious conclusions without properly challenging assumptions. To gain precise control, the team decomposed the workflow into chained model invocations with structured JSON outputs organized across three agent personas: a Director, four domain experts, and a Critic. Domain experts gather raw evidence through tool calls, the Critic evaluates finding quality and synthesizes a timeline, and the Director steers investigation phases using tiered model costs. The multi-agent system enables engineers to supervise investigations via a real-time dashboard while uncovering emergent issues like credential exposures across process ancestry chains.
Dominic MarksGrab ·
A Decade of Defense: Celebrating Grab's 10th Year Bug Bounty Program
Grab's bug bounty program has operated for a decade in partnership with HackerOne, expanding from an initial cohort of 23 researchers to over 850 active participants across global regions. The program's scope broadened between 2023 and 2024 to encompass artificial intelligence systems, Indonesian financial services, and a dedicated bounty table for mobile-specific security issues. Grab extended external testing coverage through live hacking appearances at ThreatCon 2023 and DEFCON 32, as well as invite-only anniversary campaigns with regional clubs in Germany, Morocco, and India. Internal cybersecurity teams manage vulnerability reports by emphasizing rapid triage times, direct communication, and payouts upon triage. Over the decade, reported vulnerabilities transitioned from foundational flaws toward more sophisticated and emerging threat categories.
Pei Shan YapTransformers v5: Simple model definitions powering the AI ecosystem
Transformers v5.0.0rc-0 introduces major architectural updates focused on simplicity, training, inference, and ecosystem interoperability across modern AI workflows. The release adopts a modular modeling approach and centralizes attention implementations into a unified AttentionInterface abstraction to reduce contribution and code review overhead. Support for Flax and TensorFlow is officially sunset in favor of focusing on PyTorch as the primary backend, while tokenization is standardized around the tokenizers library. For execution workloads, v5 adds native continuous batching, paged attention mechanisms, and a dedicated transformers serve OpenAI-compatible serving system. Finally, weight loading is refactored to make low-precision quantization a first-class citizen alongside broad interoperability with formats such as GGUF, MLX, and TorchAO.
Lysandre, Arthur Zucker, Cyril Vallez, Vaibhav SrivastavGrab ·
Real-time data quality monitoring: Kafka stream contracts with syntactic and semantic test
Kafka streams often suffer from syntactic and semantic data quality issues that propagate undetected to downstream consumers without real-time validation. Grab addressed this challenge by developing a standardized contract testing and observability framework within its Coban platform. Stakeholders define schema rules and field-level semantic validations, which can be recommended using large language models and anonymized sample data. A transformation engine converts these contracts into inverse SQL queries executed continuously by a FlinkSQL Test Runner on a dedicated consumer group. Problematic records are published to an alert topic, archived to AWS S3, and surfaced via Slack notifications and UI field-highlighting across more than 100 critical Kafka topics.
Yuanzhe LiuDiffusers welcomes FLUX-2
Diffusers introduces support for FLUX-2 models through the Flux2Pipeline and Flux2Transformer2DModel classes. Running the 4-bit quantized checkpoint diffusers/FLUX.2-dev-bnb-4bit requires loading Mistral3ForConditionalGeneration as the text encoder and the transformer in bfloat16 precision with CPU offloading enabled. During image generation with a prompt and 50 inference steps, the pipeline encodes text embeddings through the Mistral 3 model before executing diffusion operations. However, executing the pipeline on a GPU with 14.56 GiB capacity triggers a CUDA OutOfMemoryError during 4-bit dequantization inside bitsandbytes matrix multiplication operations. Additionally, the execution logs warn that Flax classes are deprecated in Diffusers and will be removed in version 1.0.0.
YiYi Xu, Daniel Gu, Sayak Paul, Alvaro Somoza, Dhruv Nair, Aritra Roy Gosthipaty, Linoy Tsaban, Apolinário from multimodal AI artContinuous batching from first principles
Large language model serving requires running expensive next-token generation across multiple concurrent user requests. Traditional batching approaches introduce severe padding inefficiencies when mixing variable-length prompts and different generation phases, especially under static shape constraints like CUDA graphs. Continuous batching resolves these inefficiencies by combining key-value caching, chunked prefill, ragged batching, and dynamic request scheduling. Ragged batching eliminates the traditional batch axis by concatenating token sequences into a single tensor and using boolean attention masks to isolate independent sequences. By dynamically removing completed prompts and packing decoding tokens alongside chunked prefill tokens up to a hardware memory budget, serving systems maintain high hardware utilization and throughput.
Rémi Ouazan Reboul, Arthur Zucker, Luc GeorgesBuilding Deep Research: How we Achieved State of the Art
Building production AI research agents presents challenges around context window pollution, escalating token costs, and architectural brittleness across model updates. To resolve these issues, Tavily rebuilt its deep research system around simplified orchestration, compact tooling, and active context curation. The agent mimics human research workflows by distilling tool outputs into concise reflections for ongoing reasoning, withholding raw retrieved web data until the final deliverable stage. Compared to traditional ReAct propagation architectures where token consumption scales quadratically, this reflection-based approach achieves linear token growth. As a result, the system reduced token consumption by 66% compared to Open Deep Research while reaching state-of-the-art performance on DeepResearch Bench.
Michael Griff, Dean Sacoransky, Noah NefskyOVHcloud on Hugging Face Inference Providers 🔥
OVHcloud is now integrated as a supported Inference Provider on the Hugging Face Hub, expanding serverless inference options across model pages and client SDKs. The integration enables access to open-weight models, including gpt-oss, Qwen3, DeepSeek R1, and Llama, using European infrastructure with pay-per-token pricing starting at €0.04 per million tokens. Developers can connect via Python and JavaScript SDKs using either direct custom API keys or automatic routing through Hugging Face tokens. The service supports structured outputs, function calling, multimodal workflows, and embedding models while providing sub-200ms first-token response times. Calls routed through Hugging Face pass through standard provider pricing without markups, and PRO tier subscribers receive monthly inference credits.
Gilles Closset, Fabien Ric, Elias Tourneux