DatabricksSean Fischer, Chris Mantz, Andy LaunchburyAnnouncement
HL7v2 is the dominant healthcare messaging standard but is difficult to parse due to its nested, delimiter-encoded structure. Existing workarounds like converting to FHIR or using third-party engines add cost and latency. Funke, the successor to the open-source Smolder library, is a new open-source Python and PySpark library under the Databricks License that parses HL7v2 messages directly into native Spark types, preserving the full hierarchy losslessly. It deploys as a Databricks Asset Bundle with a Declarative Pipeline that uses Auto Loader to ingest raw messages into a Unity Catalog volume, then parses them into structured silver tables following the medallion pattern. The library supports all HL7 message types and versions, includes accessor helpers for extracting fields by segment and component, and ships with a demo using synthetic ADT events to illustrate end-to-end ingestion and gold table creation, including a live census and bed utilization dashboard.
HL7v2 is the dominant healthcare messaging standard but is difficult to parse due to its nested, delimiter-encoded structure. Existing workarounds like converting to FHIR or using third-party engines add cost and latency. The open-source Smolder library was a previous solution but predates modern Databricks platform features.
Funke parses HL7v2 messages directly into native Spark types, preserving the full hierarchy. It deploys as a Databricks Asset Bundle with a Declarative Pipeline that ingests raw messages via Auto Loader into a Unity Catalog volume, then parses them into structured silver tables. The library includes accessor helpers for extracting fields by segment and component, and supports all HL7 message types and versions.
Funke parses HL7v2 messages losslessly into native Spark types, preserving the full hierarchy as a map of segment names to repetitions, with fields addressable by number, repetition, component, and subcomponent.
Funke deploys as a Databricks Asset Bundle with a Declarative Pipeline following the medallion pattern, using Auto Loader for ingestion and Unity Catalog for governance.
Funke includes accessor helpers like get_value, get_segment, get_field, get_component, and a parse_hl7_version helper, and ships with a demo using synthetic ADT events to illustrate end-to-end ingestion and gold table creation.
Funke reads the delimiter characters declared in the MSH header of each message, including field, component, repetition, and subcomponent separators, and uses them to parse the message correctly.
Funke is deployed as a Databricks Asset Bundle. Users clone the repository, open it in the DAB editor, and click Deploy, which builds the library and provisions the pipeline, Unity Catalog schema, and landing volume. Alternatively, deployment can be done via the Databricks CLI.
No, Funke is a parser and ingestion accelerator. It does not replace an interface engine or the clinical and HL7 expertise required to interpret messages correctly. The mapping from segments to business concepts is left to the user's domain knowledge.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
A four-step pattern for letting a document agent write validated aggregation pipelines instead of arbitrary code, so results come back shaped and tenancy stays enforced.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
