Unlocking the Archives: Turning Unstructured Documents into a Searchable Database for Groundwater Discovery

Summary

MapAid and the Sudan Association for Archiving Knowledge (SUDAAK) needed to make nearly 700 scanned geological reports searchable and recover well data for MapAid’s WellMapr groundwater models. Databricks built a Databricks-only pipeline that renders pages as images, uses multimodal AI Functions for page classification and schema-constrained JSON, and applies targeted sampling to cut classification processing by more than 70%. Water-relevant documents receive page-by-page OCR through the Foundation Model API, with well and borehole identifiers linking information across pages before structured records capture locations, depths, static water levels, and yields. In its first full run, 654 documents and 5,570 pages were classified in under three hours; 95% received excellent or good evaluations, about half contained water data, and 299 well or borehole records were extracted.

Context

Decades of Sudanese geological surveys, well-drilling reports, and field studies were digitized but scattered across nearly 700 scanned files and lacked a retrieval system. This made borehole and aquifer information difficult to find and limited its use in MapAid’s WellMapr groundwater models.

Approach / What changed

The pipeline stores rendered page images in Unity Catalog Volumes, samples informative pages for document classification, and uses multimodal AI Functions to produce geographic, subject, and water-relevance metadata. Water-related documents undergo full page-by-page OCR through the Foundation Model API, entity linking, and schema-constrained extraction of well and borehole records. Automated AI evaluation flags low-confidence classifications for manual review.

Takeaways

• Sampling shorter documents fully and longer documents from informative sections reduced AI processing volume by more than 70% while preserving classification quality.
• Well and borehole identifiers serve as anchor entities during OCR, allowing locations, depths, water levels, and yields distributed across multiple pages to be linked into structured records.
• An AI evaluator rates classifications for accuracy, completeness, and consistency; documents below a confidence threshold are routed for manual review, and 95% of first-run classifications were rated excellent or good.

Related reading