How Socialpruf built a faster, more reliable data stack by replacing Neon with Postgres managed by ClickHouse

Summary

Socialpruf, a social media performance tracking platform ingesting hundreds of posts per second, experienced analytics latency, connection dropouts, and high network transfer costs while running Postgres on Neon alongside ClickHouse Cloud. To resolve these issues, the team migrated their transactional system of record to NVMe-backed Postgres managed by ClickHouse, colocating transactional and analytical workloads. Although standard Postgres logical replication failed on their 0.5 TB dataset, they achieved continuous synchronization using PeerDB before conducting a production cutover. The migration eliminated connection dropouts, reduced network transfer fees, and improved overall Postgres query performance by approximately 30 percent. To manage thousands of concurrent connections generated by ingestion workers, Socialpruf implemented ClickHouse's bundled, horizontally scalable PgBouncer instances.

Context

Socialpruf ran its transactional data on Neon Postgres, replicating data via CDC to ClickHouse Cloud for analytics. At scale, the platform faced connection dropouts and restarts in Neon that stalled background data processing jobs. Additionally, cross-service network transfer charges between Neon and ClickHouse within the same AWS region exceeded half of their compute expenses.

Approach / What changed

Socialpruf migrated its 0.5 TB transactional database from Neon to Postgres managed by ClickHouse using PeerDB after initial native logical replication failed. The team ran a mirrored setup for a week, validated application behavior using database forks, executed a cutover, and adopted bundled PgBouncer instances to handle thousands of concurrent application and ingestion worker connections.

Takeaways

• Streaming CDC data from Neon to ClickHouse generated network transfer costs exceeding half the compute bill, which collocating managed Postgres in ClickHouse's infrastructure eliminated.
• Initial logical replication failed during a 0.5 TB database migration, prompting the team to use PeerDB to mirror tables and maintain continuous synchronization prior to cutover.
• Migrating to NVMe-backed Postgres managed by ClickHouse improved overall Postgres query latency by around 30% with some queries seeing up to 5x gains, while removing connection drops.

Related reading