Clinical operations intelligence belongs on the Lakehouse

Summary

The Site Feasibility Workbench addresses fragmented clinical operations data by placing a site-selection decision-support application, its models, and data on Databricks. Released as a fully open-source Databricks App, it combines a FastAPI backend, React frontend, Unity Catalog-governed data, Lakebase operational state, and AI/BI Genie within the workspace. TA-segmented LightGBM models use the organization’s CTMS, EDC, and IRT history alongside real-world evidence, patient access, qualification, Open Payments, and protocol-execution signals to rank sites. Each prediction stores SHAP attributions in a governed Unity Catalog Delta table, while MLflow tracks training runs and model versions; saved shortlists persist to Lakebase and Genie answers governed-data questions. The six-step workflow supports protocol selection through final shortlisting, including diversity considerations; the release is a decision-support layer rather than a source of record, and deployment reportedly takes about 30 minutes before security review and validation.

Context

Clinical operations data is distributed across warehouses, CTMS, EDC, BI layers, and separate application databases that do not support connected decision-making. The resulting synchronization lag, integration overhead, and reliance on spreadsheets weaken trust in site-feasibility decisions.

Approach / What changed

The workbench runs as an open-source Databricks App with FastAPI, React, Unity Catalog, Lakebase, AI/BI Genie, LightGBM models, SHAP explanations, and MLflow tracking. It uses organization-specific clinical history and related signals to guide site selection, preserve shortlists, and provide governed prediction lineage.

Takeaways

• SHAP attributions for each site prediction are stored in a governed Unity Catalog Delta table, allowing recommendations to be examined for systematic under-weighting of community sites, minority-serving institutions, or first-time investigators.
• The six-step workflow covers protocol selection, scoring constraints, geographic overview, site ranking, SHAP-based site analysis, and final shortlist creation, with diversity included as a scoring dimension.
• The workbench is explicitly a decision-support layer; CTMS, EDC, and IRT remain the authoritative source-of-record systems, while saved shortlists are persisted in Lakebase for team sharing.

Related reading