---
title: "Cisco Talos empowers threat researchers while reducing TCO by 75% with ClickHouse Cloud"
description: "Cisco Talos operates a threat-intelligence reputation service that classifies file hashes from Cisco product telemetry, enabling researchers to make security decisions about benign and potentially malicious artifacts. Previously, a custom pipeline moved data from Databricks Delta and S3 through EKS, where staging and aggregated tables supported an API; the self-managed design retained about a petabyte of compressed, replicated data on 32 high-compute nodes. The team rebuilt the system in four phases, using Terraform, SSO/SAML, sharded ClickPipes for a 12-month backfill, shadow traffic, and a validated cutover to ClickHouse Cloud on AWS. The resulting architecture uses EventBridge, SQS, ClickPipes, SharedMergeTree, object storage, and AWS PrivateLink, delivering less than a minute of queue lag, zero data loss, a 100% response match across more than 1,000 hashes, zero downtime, roughly 90% lower storage costs, and about 75% lower TCO while reclaiming over 300 engineer hours annually."
---

# Cisco Talos empowers threat researchers while reducing TCO by 75% with ClickHouse Cloud

[Clickhouse](https://yomu.fyi/company/clickhouse) · Jun 15, 2026

**Type:** Problem & solution

## Summary

Cisco Talos operates a threat-intelligence reputation service that classifies file hashes from Cisco product telemetry, enabling researchers to make security decisions about benign and potentially malicious artifacts. Previously, a custom pipeline moved data from Databricks Delta and S3 through EKS, where staging and aggregated tables supported an API; the self-managed design retained about a petabyte of compressed, replicated data on 32 high-compute nodes. The team rebuilt the system in four phases, using Terraform, SSO/SAML, sharded ClickPipes for a 12-month backfill, shadow traffic, and a validated cutover to ClickHouse Cloud on AWS. The resulting architecture uses EventBridge, SQS, ClickPipes, SharedMergeTree, object storage, and AWS PrivateLink, delivering less than a minute of queue lag, zero data loss, a 100% response match across more than 1,000 hashes, zero downtime, roughly 90% lower storage costs, and about 75% lower TCO while reclaiming over 300 engineer hours annually.

## Context

A small engineering team supporting Cisco Talos's threat intelligence services was maintaining a nearly 2-trillion-row, 2 PB dataset on self-managed ClickHouse and EKS. The architecture required custom ingestion, aggregation, upgrades, backups, troubleshooting, vulnerability management for SOC 2 compliance, and over-provisioned compute tied to growing storage needs. These operational and cost burdens diverted engineering time from building products for threat researchers and risked service disruption during scaling.

## Approach / What changed

The team migrated from OSS ClickHouse to ClickHouse Cloud on AWS in four phases: creating Terraform modules and SSO/SAML integration; building the ingestion pipeline and backfilling 12 months with sharded ClickPipes; validating performance and data through shadow traffic; and completing a source comparison before cutover. The resulting design uses EventBridge and SQS for object notifications, ClickPipes for ingestion, SharedMergeTree with object storage, query-time aggregation enabled by smart partitioning, and AWS PrivateLink for private API connectivity.

## Takeaways

- The migration preserved service continuity: production ingestion handled roughly 3,000 files per day with zero data loss, less than a minute of queue lag, and zero downtime or consumer impact.
- Comparisons of API responses for more than 1,000 randomly selected hashes produced a 100% match between the old and new backends.
- Separating storage from compute and removing custom ingestion and related SOC 2 maintenance reduced storage costs by roughly 90%, lowered TCO by about 75%, and saved over 300 engineer hours annually.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [AWS](https://yomu.fyi/topic/aws), [Kubernetes](https://yomu.fyi/topic/kubernetes), [Migrations](https://yomu.fyi/topic/migration), [Performance](https://yomu.fyi/topic/performance)

- Source: [Clickhouse](https://clickhouse.com/blog/cisco)
- Source URL: https://clickhouse.com/blog/cisco
- Ingested by Yomu: 2026-08-29T01:35:10.008Z

[Read original post](https://clickhouse.com/blog/cisco)