# Treating Data as Code at Two Sigma

[Two Sigma](https://yomu.fyi/company/two-sigma) · Emily Majewski · Nov 13, 2025

**Type:** Problem & solution

## Summary

Scaling research and trading platforms across thousands of data sources created bottlenecks at Two Sigma, where reliance on database snapshots and fragmented infrastructure slowed dataset delivery to data scientists. To address operational costs and architectural complexity, the data engineering team adopted software development principles by treating data as code. The organization migrated to Google BigQuery's serverless architecture, standardized SQL transformations using dbt, and defined declarative pipelines with Terraform under continuous integration workflows. Internal tooling was introduced to track directed acyclic graph dependencies, automate anomaly detection, and streamline data discovery. These shifts eliminated manual data movement routines, reduced operational overhead, and enabled formal data contracts to safeguard downstream consumers and emerging language model integrations.

## Context

As Two Sigma scaled research and trading platforms across thousands of data sources, traditional data handling methods created bottlenecks and maintenance overhead. Data scientists experienced long waits for datasets due to landscape complexity. Relying on database snapshots to reproduce and move data failed to keep pace with business demands, while fragmented infrastructure and scattered SQL transformations increased operational costs.

## Approach / What changed

Two Sigma adopted software engineering practices for data management by migrating to Google BigQuery's serverless architecture to centralize analytics-ready datasets. The team integrated dbt to version-control SQL transformations and automate data quality testing, alongside Terraform to define pipelines declaratively. They also built internal tools for automated anomaly detection, data discovery and documentation, and computational workflow orchestration.

## Takeaways

- Migrating fragmented infrastructure to BigQuery enabled automatic scaling and shifted engineering focus from infrastructure management to building centralized, analytics-ready datasets.
- Applying dbt and Terraform brings software engineering practices like version control, declarative definitions, automated SQL testing, and continuous integration workflows to data pipelines.
- Formal data contracts establish explicit agreements between teams to prevent breaking schema changes, ensuring dataset reliability and enabling downstream use by large language models.

**Tags:** [CI/CD](https://yomu.fyi/topic/ci-cd), [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Migrations](https://yomu.fyi/topic/migration), [Observability](https://yomu.fyi/topic/observability), [Testing](https://yomu.fyi/topic/testing)

[Read original post](https://www.twosigma.com/articles/treating-data-as-code-at-two-sigma)