Topic

Direct I/O

1 post about Direct I/O. Every summary links to the original.

See the idea

Direct I/O explainers

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Clickhouse
What is direct I/O, and why does ClickHouse Managed Postgres use it for backups?
A base backup reads every byte of a live database off the same NVMe drives that serve queries. Buffered reads go through the kernel page cache, evicting warm database pages and adding a kernel copy per byte. Direct I/O skips the cache, leaving Postgres’s warm data in place, but also disables kernel readahead. On a RAID0 array, small direct reads hit only one drive, so ClickHouse Managed Postgres sizes each direct read to span the whole stripe and scales the reader count to the hardware. On a 48 vCPU box with four NVMe drives and a 467 GB database, the direct I/O backup evicted nothing, cut query latency impact by about two thirds, used about 14% less CPU, and finished in 71 seconds.

The best engineering reads, once a week.

Topic

1 post about Direct I/O. Every summary links to the original.

See the idea

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Clickhouse
What is direct I/O, and why does ClickHouse Managed Postgres use it for backups?
A base backup reads every byte of a live database off the same NVMe drives that serve queries. Buffered reads go through the kernel page cache, evicting warm database pages and adding a kernel copy per byte. Direct I/O skips the cache, leaving Postgres’s warm data in place, but also disables kernel readahead. On a RAID0 array, small direct reads hit only one drive, so ClickHouse Managed Postgres sizes each direct read to span the whole stripe and scales the reader count to the hardware. On a 48 vCPU box with four NVMe drives and a 467 GB database, the direct I/O backup evicted nothing, cut query latency impact by about two thirds, used about 14% less CPU, and finished in 71 seconds.

The best engineering reads, once a week.