---
title: "How we could save petabytes of cache storage with Zstandard and Pingora"
description: "Cloudflare prototyped Cache Transcoding, a Pingora-based architecture that stores eligible cache responses as Zstandard (zstd) rather than their original representation. On a fill, the proxy compresses qualifying uncompressed text at zstd level 3, records the compressed representation and original content length, keeps it compressed on disk and between Tiered Cache layers, and decodes it on the client-facing path. Eligibility requires a 200 OK response, compressible text, no existing Content-Encoding, known Content-Length of at least 4 KiB, and excludes several request and binary-content cases. In initial tests, eligible assets shrank to roughly one-third of their original on-disk size, with a 2.834x ratio, while encoding cost 4.31 ns per byte and decoding 1.56 ns; the modeled CPU increase stayed within budget, though broader testing remains necessary."
---

# How we could save petabytes of cache storage with Zstandard and Pingora

[Cloudflare](https://yomu.fyi/company/cloudflare) · Aashi Patel · Sep 1, 2026

**Type:** Problem & solution

## Summary

Cloudflare prototyped Cache Transcoding, a Pingora-based architecture that stores eligible cache responses as Zstandard (zstd) rather than their original representation. On a fill, the proxy compresses qualifying uncompressed text at zstd level 3, records the compressed representation and original content length, keeps it compressed on disk and between Tiered Cache layers, and decodes it on the client-facing path. Eligibility requires a 200 OK response, compressible text, no existing Content-Encoding, known Content-Length of at least 4 KiB, and excludes several request and binary-content cases. In initial tests, eligible assets shrank to roughly one-third of their original on-disk size, with a 2.834x ratio, while encoding cost 4.31 ns per byte and decoding 1.56 ns; the modeled CPU increase stayed within budget, though broader testing remains necessary.

## Context

Increasing RAM and hard disk prices are putting pressure on Cloudflare’s massively distributed storage products, including its CDN. The work aims to expand effective cache capacity with existing hardware while reducing the bytes transferred between data centers.

## Approach / What changed

Cache Transcoding uses Pingora to encode eligible, uncompressed text responses with Zstandard before storing them on disk. The compressed representation remains in Tiered Cache transfers, metadata preserves the original content length and marks the encoding, and decoding occurs on the client-facing path. The prototype uses zstd level 3 and a 4 KiB minimum object size.

## Takeaways

- In the controlled corpus, eligible assets compressed by roughly 2.8 times, with a measured 2.834x ratio. Encoding cost 4.31 ns per byte and decoding cost 1.56 ns per byte.
- Transcoding all eligible compressible text at or above 4 KiB captured nearly all measured storage benefit within the CPU budget; restricting compression to only popular content performed worse.
- Compressed objects remain compressed across Tiered Cache transfers, and a storage encoding marker prevents cache layers from encoding an object more than once.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Caching](https://yomu.fyi/topic/caching), [Performance](https://yomu.fyi/topic/performance), [Pingora](https://yomu.fyi/topic/pingora), [Zstandard](https://yomu.fyi/topic/zstandard)

- Source: [Cloudflare](https://blog.cloudflare.com/cache-transcoding)
- Source URL: https://blog.cloudflare.com/cache-transcoding
- Ingested by Yomu: 2026-09-01T16:00:44.822Z

[Read original post](https://blog.cloudflare.com/cache-transcoding)