Databricks for Good and Virtue Foundation: Partnering to Connect Medical Volunteers to Critical Health Services in 72 Countries

Summary

Virtue Foundation’s VF Match platform connects medical professionals with volunteer opportunities across 72 low and low-middle income countries, but scaling its initial LLM proof of concept into production required a robust data pipeline. The Databricks-based Foundational Data Refresh combines Overture Maps and Bright Data, processes more than 25 million web pages with targeted GPT extraction steps, and uses Spark, Photon, and Lakeflow Jobs to orchestrate checkpointed, modular workloads. Splink resolves duplicate facilities and nonprofits across messy sources; Photon reduced the slowest entity-resolution partition from 30 minutes to approximately two minutes, a 15x improvement. A prototype VF Agent adds natural-language access through LangGraph, Databricks Model Serving, AI Search, and Genie, while the resulting system provides an up-to-date global view of healthcare infrastructure and volunteer needs.

Context

Virtue Foundation needed to scale an initial LLM proof of concept into a production-grade system that could aggregate healthcare facility and nonprofit data across 72 low and low-middle income countries, identify gaps in services, and help match medical volunteers with critical needs.

Approach / What changed

The platform uses Overture Maps and Bright Data as source inputs, targeted GPT extraction stages, star-schema storage, status-based checkpointing, a configurable extraction registry, Spark and Photon for distributed processing, and Lakeflow Jobs for orchestration. Splink performs probabilistic entity resolution, while a prototype LangGraph multi-agent system routes natural-language requests to AI Search or Genie.

Takeaways

• The Foundational Data Refresh processed more than 25 million web pages by separating medical relevance classification, organization-type identification, and extraction of specialties, equipment, and procedures into targeted LLM steps.
• Status-based checkpointing lets records resume from their processing state without repeating expensive LLM calls, while the extraction registry keeps prompts and methods modular, reproducible, and extensible.
• Photon reduced the slowest entity-resolution partition from 30 minutes to approximately two minutes, addressing skewed pairwise workloads and delivering a reported 15x improvement.

Related reading