Loading…
Latest reads
The engineering internet, summarised so you can actually read it.
MongoDB ·
Vision RAG: Enabling Search on Any Documents
Traditional retrieval-augmented generation relies on text parsing and optical character recognition to extract data from complex visual documents. These legacy techniques introduce high engineering effort, inconsistent accuracy, and heavy scaling costs. Vision RAG resolves these challenges by embedding entire visual documents directly through multimodal models like Voyage AI's voyage-multimodal-3.5, removing the need for text extraction pipelines. In this pipeline, dense vector embeddings map both text queries and visual assets into a unified representation space. At query time, the system retrieves relevant document images and supplies them alongside the text prompt to Anthropic's Claude 3.5 Sonnet to generate visually grounded responses.
Thibaut GourdelLyft ·
Lyft’s Feature Store: Architecture, Optimization, and Evolution
Lyft built a centralized Feature Store to optimize feature management, deployment, and data consistency for machine learning workflows across its rideshare platform. The system ingests batch features defined by Spark SQL and JSON metadata into Airflow DAGs, which simultaneously write data to Hive for model training and to an online serving layer for low-latency inference. Real-time streaming features are processed with Apache Flink and Apache Kafka before being ingested into the serving infrastructure. The online layer, dsfeatures, couples DynamoDB with a ValKey write-through LRU cache and OpenSearch to serve embeddings. Infrastructure optimizations reduced standard P95 read latency by a third while supporting over a trillion additional read and write operations.
Rohan VarshneyNVIDIA Cosmos Reason 2 Brings Advanced Reasoning To Physical AI
NVIDIA released Cosmos Reason 2, an open reasoning vision-language model designed to improve multi-step planning, uncertainty handling, and physical AI decision-making. Available in 2B and 8B parameter configurations, the model increases input context support to 256K tokens, up from 16K in the prior version. The release adds spatial perception capabilities including optical character recognition, 2D and 3D point localization, bounding boxes, and trajectory generation for robotics. In domain adaptation tests on autonomous vehicle video captioning, fine-tuning the 8B model improved BLEU scores by 10.6% and LingoQA scores by 13.8%. The model leads the Physical AI Bench and Physical Reasoning leaderboards for open visual understanding models.
Tsung-Yi Lin, Debraj SinhaIntroducing Falcon-H1-Arabic: Pushing the Boundaries of Arabic Language AI with Hybrid Architecture
The Technology Innovation Institute introduced Falcon-H1-Arabic, a family of Arabic language models released in 3B, 7B, and 34B parameter sizes. Built on a hybrid architecture, each block runs Mamba State Space Models and Transformer attention in parallel before fusing their representations at the output projection. Pre-training incorporated around 300 billion tokens across Arabic dialects, English, and multilingual content filtered via linguistic analysis. A post-training pipeline combining supervised fine-tuning and direct preference optimization addresses long-context utilization and alignment. Across evaluations on the Open Arabic LLM Leaderboard, the 34B model achieves roughly 75%, exceeding the performance of larger baseline systems like Llama-3.3-70B.
Basma Boussaha, Mohammed Alyafeai, Ahmed Alzubaidi, Leen AlQadi, Shaikha Alsuwaidi, Omar saif alkaabi, Hamza Alobeidli, Hakim HacidAprielGuard: A Guardrail for Safety and Adversarial Robustness in Modern LLM Systems
Modern language model systems face security and safety threats spanning prompt injection, jailbreaks, and tool manipulation across long-context agentic workflows. To address these vulnerabilities, researchers introduced AprielGuard, an 8B parameter causal decoder-only transformer based on a downscaled Apriel-1.5 Thinker Base model. The model detects 16 safety risk categories alongside binary adversarial attacks across standalone prompts, multi-turn dialogues, and agentic workflows. Trained on synthetic datasets created with Mixtral-8x7B, NVIDIA NeMo Curator, and SyGra, AprielGuard operates in both explainable reasoning and fast non-reasoning modes. Evaluation shows high precision and recall on safety and adversarial benchmarks, though trade-offs exist between explainability latency and non-reasoning performance.
Jaykumar KasundraGrab ·
Demystifying user journeys: Revolutionizing troubleshooting with auto tracking
Tracking user journeys across hundreds of mobile UI components via manual clickstream instrumentation frequently caused data gaps and prolonged issue resolution. Grab designed AutoTrack, an SDK that automatically records application lifecycle states, launch triggers, user interactions, and screen contexts. The system monitors transitions across foreground, background, and inactive states alongside entry points like deep links and push notifications. On the client side, AutoTrack captures complete screen hierarchy metadata from native Android activities, fragments, and iOS view controllers, redacting sensitive data to comply with privacy regulations. This automated trace enables engineering teams to reproduce elusive bugs, route incidents to domain owners, and generate automated UI test cases from real-world usage patterns.
Alex ProkofievMongoDB ·
That’s a Wrap: MongoDB’s 2025 in Review & 2026 Predictions
MongoDB recapped its product releases, executive transitions, and enterprise adoption milestones throughout 2025 alongside predictions for 2026. The company acquired embedding and reranking specialist Voyage AI to mitigate large language model hallucinations and launched MongoDB AMP to accelerate legacy application modernization. Vector search and full-text search capabilities were expanded from MongoDB Atlas to Community Edition and Enterprise Server, enabling on-premises and hybrid retrieval-augmented generation workloads. Enterprise implementations included pharmaceutical distributor McKesson scaling tracking infrastructure to handle 1.2 billion containers annually after replacing its monolithic legacy system. Executive predictions for 2026 emphasized human-in-the-loop verification, grounding language models with factual retrieval, and building multi-cloud resilience against outages and data sovereignty constraints.
Kevin O'RourkeMongoDB ·
Token-count-based Batching: Faster, Cheaper Embedding Inference for Queries
Serving embedding models for short search queries often suffers from poor GPU efficiency because traffic is spiky and memory-bound. Traditional time-window and request-count batching strategies lead to inconsistent GPU utilization, while tensor padding wastes compute on empty tokens. To resolve these bottlenecks, Voyage AI implemented token-count-based batching paired with padding removal in inference engines like vLLM. The architecture uses Redis with Lua scripts to atomically aggregate pending requests until reaching an optimal hardware saturation token threshold. Across production deployments, this approach achieved a 50% reduction in GPU inference latency with 3X fewer GPUs and improved throughput by up to 8×.
Chengcheng Pei, Yuan LinGrab ·
How Grab is accelerating growth with real-time personalization using Customer Data Platform scenarios
Grab previously relied on daily batch attribute updates in its Customer Data Platform, which created engineering bottlenecks and hindered time-sensitive engagement opportunities. To address this limitation, the team introduced Scenarios, a self-serve real-time personalization capability embedded within the platform. The architecture processes real-time event triggers from Grab's Scribe platform using Apache Flink, enriches incoming events with historical context from StarRocks, and evaluates pre-trained machine learning classifiers. Computed outputs sync to Kafka streams or Amphawa, an internal feature store powered by AWS DynamoDB, maintaining end-to-end latencies under fifteen seconds. Across more than a dozen production deployments, including real-time interventions for subscription abandonment within fifteen minutes, the platform achieved over a 3% conversion uplift compared to batch campaigns.
Saubhagya AwaneeshTokenization in Transformers v5: Simpler, Clearer, and More Modular
Transformers v5 overhauls its tokenization framework by separating tokenizer architecture from trained vocabularies. In contrast to v4's dual slow Python and fast Rust files, v5 consolidates each model tokenizer into a single file defaulting to the Rust-backed TokenizersBackend. The pipeline stages—normalizer, pre-tokenizer, model algorithm such as BPE or Unigram, post-processor, and decoder—are now directly exposed and configurable rather than buried in serialized files. Practitioners can instantiate blank tokenizer architectures and train custom vocabularies directly from iterators using native methods like train_new_from_iterator while retaining model-specific formatting rules. The wrapper layer continues to bridge raw tokenization and model requirements by managing chat templates, context limits, and special token insertion.
Ita Zaporozhets, Aritra Roy Gosthipaty, Arthur Zucker, Sergio Paniego, merve, Pedro CuencaThe Open Evaluation Standard: Benchmarking NVIDIA Nemotron 3 Nano with NeMo Evaluator
Assessing whether large language model improvements stem from genuine advances or underspecified evaluation conditions remains a major challenge across the industry. Most published model evaluations omit critical execution parameters, prompt templates, harness versions, and runtime configurations. In response, NVIDIA released Nemotron 3 Nano 30B A3B alongside its complete, reproducible evaluation recipe built with the open-source NeMo Evaluator library. The library serves as an orchestration layer that unifies diverse benchmark harnesses under standard configurations while decoupling evaluation logic from underlying inference backends. Developers can execute the identical evaluation pipeline against local deployments or hosted endpoints using published YAML configurations and structured logging.
Seph Mard, Isabel Hulseman, Besmira Nushi, Piotr Januszewski, Grzegorz Chlebus, VivienneZhang, Wojciech Prazuch, Pablo Ribalta, Nik Spirin, Ferenc GalkoLyft ·
From Python3.8 to Python3.10: Our Journey Through a Memory Leak
During an initiative to upgrade Python services from version 3.8 to 3.10, Lyft engineers encountered severe latency spikes and timeouts in one service within their test environment. Profiling DynamoDB queries revealed that gevent thread joins were taking up to 30 seconds while pod memory consumption climbed steadily. To isolate the issue, engineers used an internal tracemalloc-based profiler triggered via SIGUSR2 signals, temporarily disabling Gunicorn preload to prevent workers from terminating prematurely on signal receipt. Memory traces pointed to botocore and an incompatibility between weakref.finalize and gevent monkey patching in urllib3 version 1.26.16, which prevented connections from returning to the pool. Downgrading urllib3 to version 1.26.15 immediately resolved both the timeouts and the memory leak before a permanent fix arrived in gevent and urllib3 updates.
Jay PatelCUGA on Hugging Face: Democratizing Configurable AI Agents
Many existing AI agent frameworks suffer from brittleness, tool misuse, and failures when executing complex workflows. To address these limitations, the open-source Configurable Generalist Agent (CUGA) introduces structured orchestration that decomposes user goals into programmatic subtasks tracked by a dynamic task ledger. The framework integrates agentic patterns like planner-executor and code-act, delegating subtasks to specialized agents that generate pseudo-code before running execution in a secure sandbox. Released under the Apache 2.0 license, CUGA integrates with Langflow for low-code visual workflow assembly and supports multi-tool environments through OpenAPI specs, MCP servers, and LangChain. Testing on inference platforms like Groq with open models such as gpt-oss-120b demonstrates rapid response times during multi-step planning and validation.
Jim Laredo, Avi Yaeli, Sami Marreed, Ayhan Sebin, Merve UnuvarNew in llama.cpp: Model Management
llama.cpp server now includes a router mode that enables dynamic loading, unloading, and switching between multiple LLMs without restarting the server. The architecture runs each model in an isolated process to ensure a single model crash does not affect other active instances. Running llama-server without specifying a model activates auto-discovery across the cache directory or a designated folder of GGUF files. In addition to on-demand loading and least-recently-used eviction capped by default at four models, the server provides endpoints for manual loading, unloading, and listing model statuses. Models inherit global configuration options or use dedicated configuration presets while also integrating directly into the built-in web UI.
Xuan-Son Nguyen, Victor MustarCodex is Open Sourcing AI models
Hugging Face Skills equips AI coding agents like OpenAI Codex to execute end-to-end machine learning workflows. By reading AGENTS.md files and interfacing via the Model Context Protocol, Codex automates dataset validation, training script updates, and job submissions to Hugging Face Jobs. The workflow supports methods including supervised fine-tuning, direct preference optimization, and reinforcement learning for models ranging from 0.5B to 7B parameters. Throughout execution, Codex tracks live metrics via Trackio, records benchmark evaluations against baselines, and maintains Markdown reports. Once training concludes, Codex merges LoRA adapters, applies GGUF quantization, and publishes the resulting models to the Hugging Face Hub for local deployment.
ben burtenshaw, shaun smithIntroducing swift-huggingface: The Complete Swift Client for Hugging Face
Hugging Face released swift-huggingface, a dedicated Swift package offering complete Hub API integration, reliable file downloads, and inference provider access. The library addresses previous limitations in swift-transformers 1.0, where interrupted multi-gigabyte model downloads could not resume and cache structures differed from Python. To resolve cache duplication, swift-huggingface implements a Python-compatible content-addressed storage layout using symlinks and flock file locking. Authentication is standardized through a TokenProvider pattern supporting auto-detection, Keychain integration, static CI/CD tokens, and OAuth 2.0 sign-in with automatic token refresh. The package also provides URLSession-backed snapshot downloads with granular progress tracking and will soon replace the HubApi implementation inside swift-transformers.
MatttDeepMath: A lightweight math reasoning Agent with smolagents
Mathematical reasoning in large language models often suffers from lengthy chain-of-thought traces and frequent arithmetic mistakes. To address these issues, DeepMath pairs a Qwen3-4B Thinking base model with a sandboxed Python execution environment built using the smolagents library and vLLM backend. The system offloads deterministic calculations by emitting concise Python snippets, executing them safely with restricted module imports and no network access, and folding returned values back into the context. Training utilizes Group Relative Policy Optimization on the Tool-Integrated Reasoning subset of OpenMathReasoning with temperature scheduling and length constraints. Across benchmarks including MATH500, AIME, HMMT, and HLE, DeepMath reduces output token lengths by up to 66% while increasing overall problem-solving accuracy.
Daniel Fleischer, Moshe Berchansky, Moshe WasserblatWe Got Claude to Fine-Tune an Open Source LLM
Hugging Face Skills equips coding agents like Claude Code, OpenAI Codex, and Google's Gemini CLI to manage end-to-end language model fine-tuning. Using the hf-llm-trainer skill, an agent validates dataset formats, selects cloud hardware, configures authentication, and submits jobs to Hugging Face Jobs. Supported techniques include Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Group Relative Policy Optimization (GRPO) for models ranging across various parameter sizes. The integration incorporates Trackio for real-time monitoring and automates LoRA configuration for larger models. Once training completes, the agent pushes artifacts to the Hugging Face Hub and can convert models to GGUF format for local execution.
ben burtenshaw, shaun smithEngineering the right opportunities for Thumbtack Pros.
Ashmann Syngle, a backend software engineer on Thumbtack's Pricing team, focuses on systems that connect local service professionals with homeowners. To improve matching and revenue operations, the team recently deployed new pricing features, launched experiments, and partnered closely with Data Science and Monetization Experience groups. Current engineering efforts center on infrastructure enhancements, specifically upgrading alerting and monitoring across owned services to raise system reliability and operational efficiency. Because the team maintains complex monetization systems within a codebase that has evolved over many years, engineers conduct deep system analyses to evaluate edge cases before implementing foundational platform modifications. Thumbtack supports these initiatives through a virtual-first operational model complemented by regular in-person offsites.
AshmannsyngleSlack ·
Streamlining Security Investigations with Agents
Slack's Security Engineering team handles billions of daily security events and needed a reliable way to streamline on-call alert triage. An initial prototype relying on a single 300-word prompt produced inconsistent results and frequently reached spurious conclusions without properly challenging assumptions. To gain precise control, the team decomposed the workflow into chained model invocations with structured JSON outputs organized across three agent personas: a Director, four domain experts, and a Critic. Domain experts gather raw evidence through tool calls, the Critic evaluates finding quality and synthesizes a timeline, and the Director steers investigation phases using tiered model costs. The multi-agent system enables engineers to supervise investigations via a real-time dashboard while uncovering emergent issues like credential exposures across process ancestry chains.
Dominic Marks