Public health intelligence shouldn't require a data scientist

Summary

State, tribal, local, and territorial (STLT) health agencies manage data across surveillance, vital records, Medicaid, WIC, and emergency preparedness systems, but those systems are fragmented and difficult to query together. That limits rapid public health intelligence: questions linking emergency-department visits with pharmacy dispensing, school absenteeism, vaccination, demographic, or geographic data can require epidemiologists to assemble manual queries over weeks, even when decisions require answers within hours. The post presents Databricks Genie as a natural-language interface for querying this environment, backed by a Databricks engine that handles petabyte-scale datasets across real-time streams and historical records. It describes cross-program synthesis, Unity Catalog row- and column-level access controls, HIPAA-compliant governance, traceability to the underlying query, and validation controlled by health experts. Examples include county-level influenza-like illness trends overlaid with vaccination coverage and identifying counties with high opioid overdose rates and low treatment utilization; Genie is described as available today.

Context

STLT health agencies have substantial data infrastructure, but information is distributed across independently built systems and often accessible only to users with specialized technical skills. Cross-system analysis for outbreak response, resource deployment, and intervention targeting can take weeks, although public health decisions may require answers within hours.

Approach / What changed

Databricks Genie provides a natural-language interface for querying monitoring and program data across real-time streams and historical records. The described implementation uses Databricks for scalable processing and Unity Catalog for row- and column-level access control, with query traceability and health-expert control over access and accuracy validation.

Takeaways

• Genie can combine reportable disease data, syndromic monitoring feeds, lab results, and demographic data in one conversational environment.
• Unity Catalog enforces data access at row and column level, while each answer is traceable to a specific query.
• A single query can compare current outbreak signals with historical baselines and combine Medicaid, WIC, behavioral health, and emergency response data.

Related reading