How the FDA is building a secure, AI-ready data foundation on Databricks for Government

Summary

The FDA built HALO (Harmonized AI and Lifecycle Operations for Data) as a secure, governed, AI-ready enterprise data platform for modernizing siloed systems without interrupting regulatory work. Its move to Databricks on AWS GovCloud, following FedRAMP High authorization sponsorship, added Unity Catalog as a governance layer across a multi-tenant architecture, while Terraform-based security patterns, PrivateLink, customer-managed keys, and the compliance security profile support regulated workloads. The agency migrated more than 5,000 users and 8,000 jobs and pipelines with zero downtime, refactoring over 1,000 pipelines and 4,000 notebooks. After onboarding eight centers and 30 programs, FDA reported query responses improving over 30%, compute costs falling over 20%, and provisioning and sharing time dropping over 75%. HALO also supports responsible AI use cases such as MARS, with humans retaining decision authority.

Context

The FDA faced siloed data, duplicated effort, inconsistent pipelines, and high operational overhead across multiple centers and programs. Analysts and scientists spent too much time finding and reconciling data, while existing infrastructure was increasingly misaligned with the agency’s scientific and regulatory mission. The platform also needed to meet stringent FedRAMP High, DoD IL5, and other sensitive-workload requirements.

Approach / What changed

The FDA created HALO as a secure, governed, AI-ready enterprise data platform and moved to Databricks on AWS GovCloud. It adopted Unity Catalog for shared governance in a multi-tenant architecture, alongside Terraform-based deployment, PrivateLink, customer-managed keys, and the compliance security profile. The agency used wave-based migration practices, refactored legacy pipelines and notebooks, and integrated HALO with its Elsa enterprise AI platform.

Takeaways

• The FDA migrated more than 5,000 users and more than 8,000 jobs and pipelines with zero downtime, while refactoring over 1,000 data pipelines and 4,000 notebooks for Unity Catalog.
• The agency reported query response improvements of more than 30%, compute cost reductions of more than 20%, and a decrease of more than 75% in provisioning, permissioning, and data-sharing time.
• MARS uses Databricks and Elsa to help reviewers analyze structured and unstructured regulatory submission data, while a human-in-the-loop model keeps scientists and reviewers responsible for decisions.

Related reading