Cisco Talos empowers threat researchers while reducing TCO by 75% with ClickHouse Cloud

Summary

Cisco Talos operates a threat-intelligence reputation service that classifies file hashes from Cisco product telemetry, enabling researchers to make security decisions about benign and potentially malicious artifacts. Previously, a custom pipeline moved data from Databricks Delta and S3 through EKS, where staging and aggregated tables supported an API; the self-managed design retained about a petabyte of compressed, replicated data on 32 high-compute nodes. The team rebuilt the system in four phases, using Terraform, SSO/SAML, sharded ClickPipes for a 12-month backfill, shadow traffic, and a validated cutover to ClickHouse Cloud on AWS. The resulting architecture uses EventBridge, SQS, ClickPipes, SharedMergeTree, object storage, and AWS PrivateLink, delivering less than a minute of queue lag, zero data loss, a 100% response match across more than 1,000 hashes, zero downtime, roughly 90% lower storage costs, and about 75% lower TCO while reclaiming over 300 engineer hours annually.

Takeaways

1. 1

   The migration preserved service continuity: production ingestion handled roughly 3,000 files per day with zero data loss, less than a minute of queue lag, and zero downtime or consumer impact.

2. 2

   Comparisons of API responses for more than 1,000 randomly selected hashes produced a 100% match between the old and new backends.

3. 3

   Separating storage from compute and removing custom ingestion and related SOC 2 maintenance reduced storage costs by roughly 90%, lowered TCO by about 75%, and saved over 300 engineer hours annually.