Reducing the noise: How Atlassian built an OCSF security lakehouse on Databricks

Summary

Atlassian faced a security data pipeline ingesting 44 billion events daily from 20 years of growth and acquisitions, each with proprietary logging formats. To unify this fragmented telemetry, the team standardized on the Open Cybersecurity Schema Framework (OCSF) and built a security lakehouse on Databricks. They embedded the entire OCSF schema into Go structs (TypeGo), guaranteeing compile-time correctness and reducing AI hallucinations during automated transformation. Table optimization with partitioning and Z-ORDER cut query times by 95%, and Graviton clusters with Photon reduced cost and runtime by two-thirds. The result was a self-service platform where SOC analysts can onboard new data sources in days, and detection teams moved from one-to-one alerts to signal-based clustering using standardized OCSF values.

Takeaways

1. 1

   Embedding the OCSF schema into Go structs (TypeGo) guarantees compile-time correctness and reduces AI hallucinations during automated transformation.

2. 2

   Table optimization with partitioning and Z-ORDER on structured OCSF data improved query performance by 95%, and Graviton clusters with Photon cut cost and runtime by roughly two-thirds.

3. 3

   Standardization on OCSF enabled detection teams to move from one-to-one alerts to signal-based clustering, where multiple findings are grouped into a single alert using standardized values like actor.user.

FAQ