How to Give an Agent a Query Language Instead of a Code Interpreter
A four-step pattern for letting a document agent write validated aggregation pipelines instead of arbitrary code, so results come back shaped and tenancy stays enforced.
Loading…
Topic
Caching
95 posts about Caching. Every summary links to the original.
- 95 posts
- 27 companies
- 6 explainers
See the idea
Caching explainers
Bloom Filters: The Cheap Gate Before the Expensive Question
A Bloom filter is a small bit array plus a few hash functions that answers "has this been seen?" with no false negatives and a false-positive rate you choose up front.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Inside Hyperdrive: How we solved a rare Hyperdrive MySQL caching bug
Hyperdrive is Cloudflare’s distributed database connection pooler for Workers; its MySQL support became GA after a beta that uncovered over 20 bugs, including a rare caching issue. The most challenging bug caused consistently empty query responses 0.001% of the time on cache-enabled instances, requiring deep protocol analysis. The root cause was that the Client and Endpoint nodes share a data center cache but use different MySQL capability flags, so a cache entry written by one node could be encoded with capabilities that the other node cannot correctly translate. When the reading node encountered such a mismatched entry, it either sent an early EOF or hung, resulting in empty rows for the Worker. The fix was to always write a canonical version of the query result bytes encoded with Hyperdrive’s own capabilities, regardless of which node stored it, which resolved the issue and raised reliability to the 99.99% target.
We just shipped support for the ugliest part of HTTP: Vary
The Vary response header, often called the ugliest part of HTTP, is now supported in Cache Rules on every Cloudflare plan. Vary tells caches which request fields may affect a response, but not which differences actually matter, leading to cache fragmentation or incorrect responses. For example, a server might deliver different image formats to different browsers, and a cache ignoring Vary risks serving the wrong bytes. The origin still names the request headers that may affect a response, but Cloudflare lets users decide how to handle each one. This support addresses the challenge of balancing correct caching with cache efficiency.
Zero downtime patching lakebase part-1 prewarming
Lakebase addresses the workload disruption caused by planned PostgreSQL maintenance, where a restart clears in-memory buffer and local file caches. Although the database can return in about one second at P99, the source reports a roughly 70% pgbench TPS reduction while caches warm, potentially causing timeouts. Lakebase uses stateless Postgres compute replicas and disaggregated shared storage to prewarm a replacement compute before a scheduled restart. The control plane transfers cached-page information, loads pages from shared storage, and follows relevant WAL from Safekeepers, while the current primary continues serving traffic. After prewarming, Lakebase promotes the new compute with PostgreSQL’s pg_promote without restarting the server; tests on 10 GB pgbench workloads found nearly instant throughput recovery, especially for read-only queries.
Halving the Time: How Uber Eats Rebuilt Its Search Pipeline
Uber Eats rebuilt its search pipeline after accumulated architectural decisions made latency increasingly costly across retrieval, hydration, ranking, ads, and presentation. The team changed its primary metric from backend API response time to Above-the-Fold completion, then introduced paginated responses with server-side caching and concurrent template rendering; these changes improved ATF by over 200 milliseconds. Other work removed low-yield retrieval strategies, grouped items by product identity to reduce lookups by over 100 times, split ranking and presentation hydration, eliminated false DAG dependencies, added request hedging, and redesigned ad data as column-oriented in memory. The combined effort cut search latency in half, including approximately 120 milliseconds from retrieval, 100-plus from hydration changes, and 130 from ads, while planned product-based search, zero-pass ranking, and streaming target further gains.
Improving Lakebase Postgres compute cache
Lakebase Postgres uses disaggregated storage backed by object storage, so compute-side caching must deliver low latency and high throughput while avoiding inefficient memory use. Its original design paired a conservatively sized PostgreSQL shared buffer with a local file cache (LFC), but the roughly 1 GB shared-buffer cap pushed many hits to slower NVMe storage and could create double buffering. For fixed-size computes with CU >= 80, the new configuration disables the LFC and sets shared buffers to 75% of DRAM, using explicit 2 MB HugeTLB pages to control page-table and TLB overhead. Benchmark tests found huge pages reduced tail read latency by up to ~40% and CPU use by up to ~30%, while production examples reported doubled throughput, 5× fewer storage reads, or 5× lower CPU use. Work is underway to dynamically resize shared buffers and huge pages for autoscaling computes.
Why You're Getting 429s in Production Using Auth0 and How to Stop Them
Auth0 429 responses in production are tied to rate limits, commonly when applications call the Management API for every request or obtain a new machine-to-machine (M2M) token for every request. Because limits are per tenant and vary by plan, logging x-ratelimit-limit, x-ratelimit-remaining, and x-ratelimit-reset headers exposes approaching exhaustion. User roles, permissions, and metadata that can be treated as access-token snapshots should be added as custom claims by a Post-Login Action, while APIs read them from the JWT. M2M tokens, valid for 24 hours by default, should be cached and refreshed with a 60-second expiry buffer; data requiring freshness should sit behind a server-side cache with a 30-to-60-second TTL. When a 429 occurs, retry at the Unix timestamp in x-ratelimit-reset rather than applying exponential backoff, because the reset time is deterministic.
ZGateway: Learnings from Putting a Proxy in Front of ZippyDB
ZGateway is a stateless proxy tier introduced to unify traffic from ZippyDB’s large, diverse client fleet and address the reliability limits of direct access. By terminating TLS, authorizing requests, applying admission control and shaping, and routing through regional tiers, it replaces a many-to-many connection mesh with bounded client-to-gateway and gateway-to-server hops. Its embedded thick C++ client batches and coalesces shard requests, while a shared pipeline supports pure-proxy and read-through-cache tiers; per-use-case metrics, traces, and quota usage provide operational visibility. The gateway handles more than 1 billion operations per second, carries about 40% of ZippyDB traffic, and adds about 6% computational overhead to an average use case, with traffic projected to exceed 60%; production work also consolidated transaction state handling across 100% of transaction traffic without a reliability regression.
How we could save petabytes of cache storage with Zstandard and Pingora
Cloudflare prototyped Cache Transcoding, a Pingora-based architecture that stores eligible cache responses as Zstandard (zstd) rather than their original representation. On a fill, the proxy compresses qualifying uncompressed text at zstd level 3, records the compressed representation and original content length, keeps it compressed on disk and between Tiered Cache layers, and decodes it on the client-facing path. Eligibility requires a 200 OK response, compressible text, no existing Content-Encoding, known Content-Length of at least 4 KiB, and excludes several request and binary-content cases. In initial tests, eligible assets shrank to roughly one-third of their original on-disk size, with a 2.834x ratio, while encoding cost 4.31 ns per byte and decoding 1.56 ns; the modeled CPU increase stayed within budget, though broader testing remains necessary.
How we saved 100 terabytes of memory by optimizing 1.1.1.1’s DNS cache
Cloudflare's Big Pineapple platform, which powers 1.1.1.1 and other DNS services, stores over 250 billion cache entries simultaneously. To decrease fleet-wide memory consumption, engineers implemented five changes to in-memory cache data structures. Key modifications included swapping growable types like Vec and String for Box slices to eliminate capacity overhead, combining distinct record sections into a single list indexed by offsets, omitting redundant owner domain names, and serializing record bytes into contiguous buffers. In benchmarks, these optimizations decreased per-entry memory footprints by 56% while improving cache lookup latency by 19% and insert throughput by 43%. Across production instances, the rollout reduced working-set memory by approximately 100 terabytes.
Gallup scales real-time coaching for thousands with Amazon Bedrock
Gallup needed to centralize decades of proprietary workplace performance research to provide proactive, real-time coaching for organizational leaders directly within their daily workflows. To solve this challenge, Gallup developed Gallup AI inside its Gallup Access platform using Amazon Bedrock and Anthropic Claude models. The multi-tenant serverless architecture leverages Amazon Bedrock Knowledge Bases and Amazon Kendra for retrieval-augmented generation, while AWS Lambda with FastAPI powers real-time streaming responses. Amazon ElastiCache Serverless caches recent conversation history for sub-millisecond session retrieval, Amazon RDS for MySQL provides durable transactional storage, and Amazon Bedrock Guardrails enforce content safety. Since its June 2024 launch, the system has achieved sub-second time-to-first-byte streaming latency while prompt volume grew approximately sevenfold across thousands of organizations.
The Cloudflare Blog – Brought to you by EmDash
Cloudflare migrated its official blog to EmDash, an Astro-focused content management system, acting as internal Customer Zero to evaluate scaling and usability. To prepare for traffic spikes reaching thousands of requests per second, engineers used k6 to execute ramp, breakpoint, and burst load tests. The resulting production architecture runs EmDash on Cloudflare Workers, pairing it with Workers Cache, an object cache backed by Workers KV, and Hyperdrive connected to PlanetScale. This layered caching design serves 99.5 percent of static files and 70 percent of requests from cache, flattening p95 latency under real-world loads reaching 850 requests per second. The migration also introduced a frontend redesign using the Kumo design system and added Model Context Protocol servers for automated agent workflows.
How We Optimized the Qwen 3.6 Model for Our Junie Agent
Deploying the Junie coding agent locally on Apple MacBook M5 hardware requires overcoming steep performance bottlenecks during model execution. Engineers optimized the entire stack using Qwen3.6-27B rather than Qwen3.8-27B, which severely degrades without token-heavy reasoning enabled. At the agent layer, Junie appends requests directly into a rolling context and caches initial prompt prefixes to maximize KV-cache reuse across tasks. For model inference, the team disabled reasoning, selected 4-bit quantization, and patched MLX-VLM to execute self-attention prefill matrix operations using faster 8-bit instructions on the M5 processor. Combining these prefill improvements with simultaneous multi-token prediction and n-gram speculative decoding delivered up to a 2x generation speedup.
How Generative Recommenders Are Redefining RecSys at Scale
Generative recommenders (GRs) recast recommendation as next-item or next-action sequence prediction conditioned on user histories, replacing traditional embedding-similarity objectives and potentially unifying retrieval and ranking. The post presents HSTU, which uses timestamp-ordered interleaved items and actions, SiLU-based attention weighting, relative attention bias, and gating, alongside Semantic IDs, which derive a smaller vocabulary through hierarchical clustering of item embeddings. For production, recsys-examples combines TorchRec, DynamicEmb, Megatron-Core, recommender-specific KV and decoding components, while nv-embedding-cache provides tiered GPU, CPU, and remote embedding storage. On an H100, the Semantic ID path was reported as 1.85x higher in online throughput and 46% lower median latency than SGLang under the stated long-context, beam-search workload, and DLRM v3 reached 99,997 queries/sec.
Reducing Text2SQL latency with parameterized query templates
In Text2SQL systems, generating SQL on the fly with frontier foundation models can cause response times of 25 to 30 seconds due to 60,000-token prompts and model latency. To solve this without degrading accuracy, an intelligent caching layer was built using parameterized SQL query templates paired with question embeddings. When an incoming question matches a cached template via semantic search above a confidence threshold, named entity recognition extracts values to populate template placeholders, bypassing the initial SQL generation step. In production, this approach reached a 60% cache hit rate after two weeks, cutting per-hit token consumption by roughly 90% and reducing overall latency by 80% to under 5 seconds.
How and Why Netflix Built a Real-Time Distributed Graph: Part 3 — Querying the graph with gRPC…
Netflix designed a query execution serving layer for its Real-Time Distributed Graph to power sub-100ms responses across diverse graph traversal workloads spanning billions of nodes and edges. To prevent compounding network delays in multi-hop queries, the engine uses a breadth-first traversal model that batches entity lookups across entire frontier levels rather than tracing paths depth-first. The entire serving architecture relies on asynchronous composition across small thread pools of 16 to 24 threads, ensuring no thread blocks while waiting on remote storage or enrichment I/O. Selective caching via EVCache targets stable properties with volatility-matched TTLs, producing 70 to 80 percent cache hit rates and reducing backend storage calls by three to four times. A layered filtering hierarchy pushes depth and edge limits directly to the storage tier, eliminating bespoke code changes while isolating system resources against excessive fan-out.
Session revocations at scale
Canva gateways check in-memory session revocations on every request, but pulling over a million revocations from MySQL during deployments created severe database stampedes. To eliminate this bottleneck, the team transitioned to storing a twelve-hour sliding window of revocations in Amazon S3 partitioned into 30-minute chunks. Each revocation is packed into a 16-byte binary structure within a sorted flat array, allowing gateways to query records via direct binary search without object deserialization. Asynchronous workers batch updates into S3 using ZooKeeper leader election and conditional PUT requests for optimistic concurrency control. This architecture reduced the in-memory cache footprint by 87.5% and allowed the team to reduce their MySQL database to just two read replicas.
Scaling Agentic AI Factories Through Extreme Co-Design with NVIDIA BlueField
Agentic artificial intelligence workflows transform data center infrastructure by turning context management, security enforcement, and inter-service communication into active components of the inference data path. To prevent host central processing units and graphics processing units from stalling on these tasks, NVIDIA introduces BlueField-4 data processing units, Vera BlueField-4 STX storage processors, and the DOCA software platform. The BlueField-4 processor offloads Layer 3 routing, encryption, and telemetry using a sixty-four-core Grace processor with up to eight hundred gigabits per second connectivity. Simultaneously, the Vera BlueField-4 STX powers the NVIDIA CMX context memory platform alongside DOCA Memos to store, protect, and share key-value cache across flash storage tiers. This co-designed infrastructure isolates tenant traffic and accelerates context reuse without degrading host-side agent execution.
From weeks to a day: how we made LLM evaluation fast enough to iterate on
Airbnb built a four-layer LLM infrastructure framework to reduce iteration and evaluation turnaround from weeks to a single day. By using per-sample caching for generated references and judge scores, they established a deterministic evaluation foundation that separates genuine model drift from measurement noise. This setup enables rapid, bounded model hotfixes using micro LoRA adapters and comprehensive end-to-end validation across component boundaries.
Scaling Android CI: From 44 → 10 minutes — A Deep Dive into Build Time Optimization and Best…
An Android monorepo’s CI pipeline averaged about 35 minutes in its build phase, with a P90 of roughly 44 minutes, as repeated dependency downloads, environment-driven invalidations, variant bloat, and machine limits increased work. The team reworked caching with branch- and architecture-aware keys, release/develop fallbacks, cached SDK and dependency data, and a Gradle Remote Cache, while filtering repositories and restricting CI to required variants. It also tuned 32GB runners to a 16GB heap and 12GB Metaspace, disabled configuration-cache saves in ephemeral CI, upgraded Kotlin, AGP 8.10.1, and Gradle 8.11.1, and moved critical workflows to Apple Silicon. Gradle tasks fell from 3399 to 2745, while remote-cache hit rates reached P50 44% and P90 56%. After the Apple Silicon refresh, total CI time reached P50 10 and P90 11 minutes, average credits per build dropped from 55 to 43, and a comparable rollout window saved about 5800 credits.
Meta’s AI Storage Blueprint at Scale
Meta describes how its BLOB-storage architecture evolved to address AI workloads' storage bottlenecks, which can stall GPUs and slow research iteration as datasets and GPU deployments expand. Legacy requests traversed multiple stateful metadata layers, sometimes across regions, producing latencies of hundreds of milliseconds even when flash made millisecond access possible. The redesign consolidates metadata into a flat schema backed by ZippyDB for O(1) path resolution, removes the dataplane proxy in favor of a fat client SDK, and deploys regional BLOB-storage stacks beside GPUs. For data loading, Meta combines host memory and flash with regional flash and global HDD-backed storage as tiered caches, using ordinary and deep prefetch, on-demand hydration, and TTL or LRU eviction policies. Production rollout saw rapid adoption, while both data-loading paradigms remain supported.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.