Topic

Pingora

2 posts about Pingora. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Posts

Modal
Endpoint Candidates: evaluate and promote inference endpoints
Modal introduced Endpoint Candidates to stage alternative inference deployments for Dedicated Endpoints, allowing evaluation of quality, latency, throughput, and cost without affecting production traffic. Users create a Candidate by selecting a model from Modal’s catalog or importing custom weights, then set a percentage of production traffic to mirror. The Pingora-based proxy forks each request stream into separate bounded buffers per Candidate and shares body chunks across mirrors with a per-process memory budget; if a buffer fills or the budget is exceeded, the proxy cancels that mirrored request while production continues. Each Candidate receives its own URL for running existing evaluations on output quality and performance benchmarks under load, and all mirrored compute uses standard Modal compute pricing. When ready, users promote from the dashboard to roll out to the production endpoint, leaving other Candidates available for continued independent testing and enabling Auto Endpoints to validate automated inference optimizations.

Cloudflare
How we could save petabytes of cache storage with Zstandard and Pingora
Cloudflare prototyped Cache Transcoding, a Pingora-based architecture that stores eligible cache responses as Zstandard (zstd) rather than their original representation. On a fill, the proxy compresses qualifying uncompressed text at zstd level 3, records the compressed representation and original content length, keeps it compressed on disk and between Tiered Cache layers, and decodes it on the client-facing path. Eligibility requires a 200 OK response, compressible text, no existing Content-Encoding, known Content-Length of at least 4 KiB, and excludes several request and binary-content cases. In initial tests, eligible assets shrank to roughly one-third of their original on-disk size, with a 2.834x ratio, while encoding cost 4.31 ns per byte and decoding 1.56 ns; the modeled CPU increase stayed within budget, though broader testing remains necessary.

The best engineering reads, once a week.

Topic

2 posts about Pingora. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Modal
Endpoint Candidates: evaluate and promote inference endpoints
Modal introduced Endpoint Candidates to stage alternative inference deployments for Dedicated Endpoints, allowing evaluation of quality, latency, throughput, and cost without affecting production traffic. Users create a Candidate by selecting a model from Modal’s catalog or importing custom weights, then set a percentage of production traffic to mirror. The Pingora-based proxy forks each request stream into separate bounded buffers per Candidate and shares body chunks across mirrors with a per-process memory budget; if a buffer fills or the budget is exceeded, the proxy cancels that mirrored request while production continues. Each Candidate receives its own URL for running existing evaluations on output quality and performance benchmarks under load, and all mirrored compute uses standard Modal compute pricing. When ready, users promote from the dashboard to roll out to the production endpoint, leaving other Candidates available for continued independent testing and enabling Auto Endpoints to validate automated inference optimizations.

Cloudflare
How we could save petabytes of cache storage with Zstandard and Pingora
Cloudflare prototyped Cache Transcoding, a Pingora-based architecture that stores eligible cache responses as Zstandard (zstd) rather than their original representation. On a fill, the proxy compresses qualifying uncompressed text at zstd level 3, records the compressed representation and original content length, keeps it compressed on disk and between Tiered Cache layers, and decodes it on the client-facing path. Eligibility requires a 200 OK response, compressible text, no existing Content-Encoding, known Content-Length of at least 4 KiB, and excludes several request and binary-content cases. In initial tests, eligible assets shrank to roughly one-third of their original on-disk size, with a 2.834x ratio, while encoding cost 4.31 ns per byte and decoding 1.56 ns; the modeled CPU increase stayed within budget, though broader testing remains necessary.

The best engineering reads, once a week.