Treating Data as Code at Two Sigma

Summary

Scaling research and trading platforms across thousands of data sources created bottlenecks at Two Sigma, where reliance on database snapshots and fragmented infrastructure slowed dataset delivery to data scientists. To address operational costs and architectural complexity, the data engineering team adopted software development principles by treating data as code. The organization migrated to Google BigQuery's serverless architecture, standardized SQL transformations using dbt, and defined declarative pipelines with Terraform under continuous integration workflows. Internal tooling was introduced to track directed acyclic graph dependencies, automate anomaly detection, and streamline data discovery. These shifts eliminated manual data movement routines, reduced operational overhead, and enabled formal data contracts to safeguard downstream consumers and emerging language model integrations.

Context

As Two Sigma scaled research and trading platforms across thousands of data sources, traditional data handling methods created bottlenecks and maintenance overhead. Data scientists experienced long waits for datasets due to landscape complexity. Relying on database snapshots to reproduce and move data failed to keep pace with business demands, while fragmented infrastructure and scattered SQL transformations increased operational costs.

Approach / What changed

Two Sigma adopted software engineering practices for data management by migrating to Google BigQuery's serverless architecture to centralize analytics-ready datasets. The team integrated dbt to version-control SQL transformations and automate data quality testing, alongside Terraform to define pipelines declaratively. They also built internal tools for automated anomaly detection, data discovery and documentation, and computational workflow orchestration.

Takeaways

• Migrating fragmented infrastructure to BigQuery enabled automatic scaling and shifted engineering focus from infrastructure management to building centralized, analytics-ready datasets.
• Applying dbt and Terraform brings software engineering practices like version control, declarative definitions, automated SQL testing, and continuous integration workflows to data pipelines.
• Formal data contracts establish explicit agreements between teams to prevent breaking schema changes, ensuring dataset reliability and enabling downstream use by large language models.

Related reading

• Two Sigma · Feb 4, 2025

  AI Core Team Lead Mike Schuster on How to Get the Most From LLMs

  Mike Schuster, Head of the AI Core Team at Two Sigma, advocates for grounding large language model adoption in practical tasks rather than speculative industry hype. Realistic enterprise applications focus on accelerating data processing, running faster experiments, and extracting domain-specific features from transcripts such as earnings calls and Federal Reserve speeches via prompt engineering. Because financial data faces inherent volume limits across trading days, successful deployments require multidisciplinary human teams to balance rapid technical experimentation with rigorous domain expertise and analytical reasoning. Schuster also dismisses predictions that programming will become obsolete, comparing coding to learning a musical instrument that cultivates structured thinking, problem decomposition, and scientific common sense essential for building complex predictive models.

  Joy Looney

• Two Sigma · Oct 23, 2025