How to Give an Agent a Query Language Instead of a Code Interpreter
A four-step pattern for letting a document agent write validated aggregation pipelines instead of arbitrary code, so results come back shaped and tenancy stays enforced.
Loading…
Topic
Caching
95 posts about Caching. Every summary links to the original.
- 95 posts
- 27 companies
- 6 explainers
See the idea
Caching explainers
Bloom Filters: The Cheap Gate Before the Expensive Question
A Bloom filter is a small bit array plus a few hash functions that answers "has this been seen?" with no false negatives and a false-positive rate you choose up front.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Experimenting with the proposed Cross-Origin Storage API in Transformers.js
Transformers.js browser apps currently redownload and recache identical AI model and WebAssembly resources when they are requested from different origins. Chrome's cache uses a Network Isolation Key composed of the top-level site and current-frame site, so matching CDN URLs do not produce cache hits; the toy example incurs 177 MB of duplicate model storage, while a shared ONNX Runtime Wasm file is 4,733 kB. The proposed Cross-Origin Storage (COS) API addresses this by identifying large files with cryptographic hashes and sharing them across opted-in origins, with a polyfill extension enabling experimentation before native browser support. In the extended Transformers.js flow, a COS lookup can serve resources in milliseconds, while a miss downloads and stores them, and ModelRegistry.is_pipeline_cached() can help select among models already cached.
Sitar-agent: Building a reliable dynamic configuration sidecar at scale
Airbnb modernized sitar-agent, a Kubernetes sidecar that delivers dynamic configuration updates to thousands of polyglot service instances without requiring redeployments. The architecture uses periodic AWS S3 snapshots to enable fast, decoupled pod startups and maintains local file-based storage for the main container to read configs with in-memory caching. Key design decisions include retaining an isolated sidecar over an in-process library and optimizing pull-based polling with server-side caching.
ClickHouse Release 26.5
ClickHouse 26.5 is a monthly release containing 38 new features, 51 performance optimizations, and 224 bug fixes. It reports a record number of performance optimizations and introduces a filesystem table function for running SQL against a local file system. Its main query-planning examples push ORDER BY … LIMIT through eligible LEFT or RIGHT JOINs when the sort key depends only on the preserved side, allowing ClickHouse to limit rows before joining. In a TPC-H benchmark at scale factor 100, enabling this optimization reduced fastest runtime from 1.878 seconds to 0.092 seconds, peak memory from 1.88 GiB to 10.98 MiB, and data read by 1.5×. Additional changes include a GROUP BY … LIMIT shortcut for eligible unordered queries, an experimental in-browser clickhouse-client, and per-subquery query-cache controls.
Modernizing Artifact Storage at Uber
Uber’s legacy artifact platform used two five-node clusters with local disks, asynchronous replication, and GeoDNS failover to support builds across monorepos and microrepos. As artifact volume grew, limited node capacity, inconsistent replication, manual hardware evacuation, and risky multi-terabyte upgrades created operational and data-loss risks. Uber moved to a managed SaaS platform with multi-region availability, cross-region replication, and cloud-native blob storage, then added an internal validation proxy in front of it. The proxy uses MySQL metadata and conditional requests, including If-None-Match and If-Modified-Since, to validate artifacts against the SaaS origin while reusing unchanged bytes; failures can fall back to passthrough, preserving correctness at higher latency and egress cost. The design reduced operational risk, improved observability, and made artifact delivery more cost-efficient at scale.
Accelerating LLM Inference with Prompt Caching for Open‑Source Models on Databricks
Repeatedly processing identical system or instruction prompts for LLM requests wastes compute, increases latency and costs, and limits throughput. Databricks extends its built-in prompt-caching capability from proprietary models to open-weights models used through Foundation Model APIs, including batch inference, pay-per-token, provisioned throughput, and higher-level services such as Agent Bricks, Genie, and AI Functions. Supported models include GPT-OSS 20B and 120B, Gemma 3 12B, fine-tuned Llama 3.1 8B via PEFT serving, and Llama 3.1 8B and 3.3 70B. By implicitly reusing KV caches for identical prompts, the system skips prefill on cache hits; caches are isolated, held only in volatile memory, and never persisted. In a production GPT-OSS batch-inference pipeline, per-replica input-token throughput rose 2.5x and P50 latency fell 3x despite a 30% cache hit ratio, with no customer configuration required.
Optimizing Our Build Times by Migrating from Webpack to Rspack
As Yelp's JavaScript monorepo expanded over time, Webpack build speeds became a significant engineering bottleneck. To resolve this without rewriting complex configurations, the team selected Rspack for its direct compatibility with existing Webpack setups and plugins. Yelp implemented an adapter pattern to generate Rspack configurations dynamically from original Webpack files, enabling a staged opt-in rollout across development tools and CI environments. Following the initial migration, engineers addressed performance issues in barrel files using automated codemods to replace star re-exports and import-then-export patterns with named re-exports. Combined with Rspack's portable persistent caching, the migration reduced integration build times by approximately 52 percent and warm cache builds by up to 80 percent.
From latency to instant: Modernizing GitHub Issues navigation performance
GitHub Issues addressed navigation latency by shifting workloads to the client using a local-first, stale-while-revalidate architecture. To evaluate perceived delays, the team tracked user transitions through Highest Priority Content thresholds, aiming for instant rendering in under 200 milliseconds. The core implementation added a persistent client-side cache using IndexedDB, paired with a synchronous in-memory tier to serve hot issue payloads without asynchronous overhead. A selective preheating mechanism resolves missing cache entries for high-intent links without overburdening backend capacity with redundant requests. Following broad rollout, the proportion of instant React soft navigations increased from 4% to approximately 22%, yielding an overall cache-hit ratio of around 33%.
Frontend Engineering at Palantir: Engineering Multilingual Collaboration
Palantir describes the frontend work behind adding read-only translation mode to Dossier, a collaborative rich-text editor used in multilingual, restrictive environments where public translation services may be unavailable or disallowed. The design lets users choose source and target languages, view a complete document in translation, copy text, and add comments while routing requests through the in-house Language Model Service. Engineers extracted shared translation components, added a Redux state slice and per-element cache, chunked content by paragraphs and other blocks, and used Intersection Observer, batching, debouncing, and caching to translate visible content incrementally; edits invalidate affected translations and trigger retranslation. To preserve sentence-level quality, inline formatting is stripped before translation, so the current translated view is plain text. The resulting mode is available in every Dossier, translating content lazily as users scroll and updating translated blocks when collaborators edit the document.
Reclaiming Terabytes: Optimizing Android image caching with TLRU
Grab’s Android app cached runtime images with DiskLruCache, but its 100 MB size limit did not remove stale content while the cache remained below capacity, contributing substantially to device storage usage. The team developed Time-Aware Least Recently Used (TLRU), which combines LRU size eviction with time-based expiration using last_accessed metadata and a TTL, while a minimum cache-size threshold protects a baseline of cached images. TLRU retains the 100 MB maximum and evicts entries when current_time minus last_accessed exceeds the TTL; the implementation also supports reading existing LRU journal files by ignoring timestamp suffixes, enabling safe rollback. Configuration was tuned through experiments using cache hit ratio, with a maximum acceptable decline of three percentage points. After production rollout, P95 total app size fell approximately 50 MB, while hit ratio stayed within target and infrastructure costs did not increase.
Introducing Waypoint-1: Real-time interactive video diffusion from Overworld
Overworld introduced Waypoint-1, a real-time interactive video diffusion model controllable through text prompts, mouse movements, and keyboard inputs. The architecture uses a frame-causal rectified flow transformer trained on compressed latent representations across 10,000 hours of video game footage paired with control signals. Pre-training uses diffusion forcing with causal attention, while post-training incorporates self forcing via DMD to mitigate autoregressive rollout error accumulation, enable one-pass classifier-free guidance, and support few-step denoising. To achieve low latency on consumer hardware, Overworld developed WorldEngine, a pure Python streaming inference library. Utilizing AdaLN feature caching, static rolling key-value caching with Flex Attention, fused matrix multiplications, and PyTorch compilation, the 2.3B parameter Waypoint-1-Small model reaches up to 60 frames per second on an RTX 5090 GPU.
Lyft’s Feature Store: Architecture, Optimization, and Evolution
Lyft built a centralized Feature Store to optimize feature management, deployment, and data consistency for machine learning workflows across its rideshare platform. The system ingests batch features defined by Spark SQL and JSON metadata into Airflow DAGs, which simultaneously write data to Hive for model training and to an online serving layer for low-latency inference. Real-time streaming features are processed with Apache Flink and Apache Kafka before being ingested into the serving infrastructure. The online layer, dsfeatures, couples DynamoDB with a ValKey write-through LRU cache and OpenSearch to serve embeddings. Infrastructure optimizations reduced standard P95 read latency by a third while supporting over a trillion additional read and write operations.
Tangle: An open-source ML experimentation platform built for scale (2025) - Shopify
Shopify introduces Tangle, an open-source, platform-agnostic machine-learning experimentation platform built for Search and Discovery workflows at commerce scale. It targets six stated failure modes, including repeated data preparation, poor reproducibility, slow deployment, unstructured notebooks, manual query tracking, and limited sharing. Users compose visual directed acyclic graph pipelines from reusable YAML components that wrap containerized or inline command-line programs in multiple languages, while file-based execution moves artifacts through storage. Tangle’s content-based caching reuses results when output content hashes remain identical, including artifacts from still-running executions, rather than relying only on lineage changes. Shopify reports a ten-hour pipeline completing in twenty minutes after one component changes, more than a year of compute time savings, and production use across ranking, semantic search, recommendation, and feature engineering workloads.
Introducing swift-huggingface: The Complete Swift Client for Hugging Face
Hugging Face released swift-huggingface, a dedicated Swift package offering complete Hub API integration, reliable file downloads, and inference provider access. The library addresses previous limitations in swift-transformers 1.0, where interrupted multi-gigabyte model downloads could not resume and cache structures differed from Python. To resolve cache duplication, swift-huggingface implements a Python-compatible content-addressed storage layout using symlinks and flock file locking. Authentication is standardized through a TokenProvider pattern supporting auto-detection, Keychain integration, static CI/CD tokens, and OAuth 2.0 sign-in with automatic token refresh. The package also provides URLSession-backed snapshot downloads with granular progress tracking and will soon replace the HubApi implementation inside swift-transformers.
Building for an Open Future - our new partnership with Google Cloud
Hugging Face and Google Cloud announced an expanded strategic partnership designed to streamline the deployment and management of open models on Google Cloud infrastructure. Prompted by a tenfold increase in Hugging Face usage on Google Cloud over three years, the collaboration introduces a joint CDN Gateway using Hugging Face Xet technology to cache models and datasets directly on Google Cloud. This caching mechanism aims to shorten download times, strengthen model supply chain resilience, and accelerate time-to-first-token across Vertex AI, Google Kubernetes Engine, Cloud Run, and Compute Engine virtual machines. Hugging Face plans to integrate native library support for Google Cloud TPUs, lower instance prices on Inference Endpoints, and enhance Hub security scanning via VirusTotal, Google Threat Intelligence, and Mandiant.
Improving performance by prefetching product pages from Etsy Search
Etsy’s search and web performance teams tested the Speculation Rules API (SRA) to reduce loading time for product listing pages reached from desktop organic search. They added a speculationrules configuration that prefetches eligible listing HTML after a 200-millisecond hover, using URL and data-attribute selectors with moderate eagerness; unlike link prefetch, SRA uses both memory and HTTP caches. The experiment required attention to a two-page memory limit, five-minute retention, non-cacheable listing HTML, video hover behavior, cookies, new-tab targets, and analytics events triggered during prefetches. Results showed 23.6% faster 75th-percentile TTFB, with improvements of 20.7% in FCP, 21.1% in LCP, 20.4% in DOMContentLoaded, and 10.6% in Page Load, while roughly one in 14 prefetches led to navigation.
Medium Android App — Migrating from Apollo Kotlin 3 to 4: Lessons Learned
Migrating the Medium Android app from Apollo Kotlin 3 to 4 exposed cache and exception-handling issues that were hidden by the earlier implementation. Apollo Kotlin 4 uses a new group, plugin, and package identifier, places exceptions in ApolloResponse, and replaces ApolloCompositeException with a primary exception plus suppressed failures. CacheMissException errors appeared in watch() because programmatic cache IDs were incomplete or incorrect; the app addressed this with declarative type and field policies, __typename on every operation, cache-update helpers, and policy-aware safeWatch filtering. The migration also replaced executeV3(), removed fetchThrows = true, added a Currency scalar adapter, and identified HTTP batching and persisted queries as future improvements, while concluding that Apollo 4 produced a more robust cache integration after the underlying issues were fixed.
Highly concurrent in-memory counter in GoLang
An overloaded database faced heavy traffic from campaign usage tracking, motivating an in-memory counter rather than an immediate SQL-to-NoSQL migration. Because exact counts were not required, increments could be cached in the application server and flushed periodically to reduce database queries and CPU use. The implementation uses Go’s Sync.Map, with LoadOrStore and CompareAndSwap for concurrent increments, plus LoadAndDelete to atomically remove values for asynchronous persistence; the source notes a fresh-map swap as an alternative locking optimization. On an Apple M4 Pro, the 2,000-key benchmark reported 53 ns/op and 1 billion operations per 60 seconds, versus 160 ns/op and 466 million operations for a Mutex-protected map, but Sync.Map used 88 B/op and four allocations. In production, update queries fell from 140 QPS to 45 QPS, master CPU from 35% to 18%, and replica CPU from 19% to 12%.
Resilient Import Maps - Better Theme Development and Beyond
Shopify found that JavaScript modules and import maps failed in themes and App Blocks when modules loaded before a map, or when multiple maps collided. These restrictions reflected the requirement that module resolution remain stable, but real pages assembled by themes, apps, Shopify, and extensions could not reliably guarantee ordering or a single map. Working with web-platform contributors, Shopify proposed a reconciliation mechanism with one global import map, a resolved module set, and merging rules that preserve existing resolutions and drop conflicting or unsafe additions. The change reached the HTML specification and shipped in Chromium and WebKit; Chrome supports it from version 133 and Safari from 18.4. Shopify also uses es-modules-shim support for browsers without native support, while Firefox support is hoped for.
Optimizing Stripe API performance in Lambda with caching strategies
The post addresses performance, cost, latency, and rate-limiting challenges that arise when applications make high-volume requests to Stripe's API from AWS Lambda. It presents a two-level cache using ElastiCache for Redis as the first layer and DynamoDB as the durable second layer, with Lambda coordinating cache reads, Stripe retrieval, and writes. On a miss, the function checks Redis, then DynamoDB, fetches the customer from Stripe if necessary, and stores the response in both layers with one-hour Redis and 24-hour DynamoDB TTLs. The DynamoDB design uses composite keys, a GSI for cache-type and timestamp queries, TTL cleanup, and environment-specific capacity settings; the post also discusses Redis connection pooling, Lambda sizing, capacity planning, cost control, monitoring, and webhook-based invalidation.
Evaluating performance impact of removing Redis-cache from a Scylla-backed service
A Rust service at Grab serves tens of thousands of peak QPS for batched count requests, using Scylla, Redis, and MySQL counter configurations. Its 15-minute cache-key truncation and five-minute TTL caused cache-hit declines and recurring Scylla QPS spikes at the end of each quarter-hour, when recent-data requests generated new keys. The experiment progressively disabled Redis, while a daily major compaction job was added to Scylla nodes; the service continued combining minutely, hourly, and daily tables for range calculations. With Redis fully decommissioned, Scylla cache hits rose from about 30% to 70% of reads, cache misses stayed about the same despite doubled traffic, peak CPU moved from roughly 50% to 52%, and overall service performance was maintained while eliminating a component responsible for about 25% of service cost.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.