Loading post…
CloudflarePranoy DuttaIncident / postmortem
Inside Hyperdrive: How we solved a rare Hyperdrive MySQL caching bug
Summary
Hyperdrive is Cloudflare’s distributed database connection pooler for Workers; its MySQL support became GA after a beta that uncovered over 20 bugs, including a rare caching issue. The most challenging bug caused consistently empty query responses 0.001% of the time on cache-enabled instances, requiring deep protocol analysis. The root cause was that the Client and Endpoint nodes share a data center cache but use different MySQL capability flags, so a cache entry written by one node could be encoded with capabilities that the other node cannot correctly translate. When the reading node encountered such a mismatched entry, it either sent an early EOF or hung, resulting in empty rows for the Worker. The fix was to always write a canonical version of the query result bytes encoded with Hyperdrive’s own capabilities, regardless of which node stored it, which resolved the issue and raised reliability to the 99.99% target.
Context
Hyperdrive is Cloudflare’s distributed database connection pooler for Workers, now with MySQL support. During the beta, a high-volume customer reported rare empty query responses (0.001% rate) on cache-enabled instances. The Client and Endpoint nodes share a data center cache but use different MySQL capability flags, causing cached response bytes to be encoded with incompatible capabilities and leading to protocol desync.
Approach / What changed
The fix ensures that all cache writes use a canonical response encoding based on Hyperdrive’s own capabilities, regardless of whether the Client or Endpoint stored the entry. This prevents one node from reading cache bytes encoded with a different capability set, eliminating the protocol mismatch that caused empty rows.
Takeaways
Hyperdrive’s Client and Endpoint nodes may have different MySQL capability flags; when they share a cache, entries must be written in a canonical encoding to avoid protocol desync and empty responses.
MySQL’s COM_RESET_CONNECTION resets character set to the server default, not the handshake-negotiated encoding; after a reset, an explicit SET NAMES utf8mb4 is required to restore full UTF-8 support.
By unifying cache write encoding, Cloudflare resolved a 0.001% empty-query bug and raised client connection success rates to 99.99% in Hyperdrive for MySQL.
FAQ
Why did Hyperdrive's MySQL caching cause empty query responses?
The Client and Endpoint nodes shared a cache but used different MySQL capability flags. A cache entry written by one node could be encoded with capabilities incompatible with the reading node, causing it to fail to translate the response and return empty rows instead of data. The fix enforces a canonical encoding for all cache writes.
How does COM_RESET_CONNECTION affect character set encoding?
COM_RESET_CONNECTION resets the connection's character set to the MySQL server's default, not the utf8mb4 encoding negotiated during the initial handshake. Hyperdrive must run a SET NAMES utf8mb4 command after each reset to maintain correct character encoding.
Source and freshness
- Source
- blog.cloudflare.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.