How Deutsche Börse built a generative AI tool to tackle the large-scale migration of Zeppelin notebooks to Databricks

Summary

Deutsche Börse Group is migrating a high volume of Zeppelin notebooks used by more than 2,000 users from Cloudera to Databricks, after Cloudera’s 2027 Zeppelin decommissioning and the group’s move to cloud analytics. Because the notebooks combine SQL, Python, custom interpreters, data-source references, visualizations, widgets, and scheduling logic, the team rejected fully rule-based rewriting. Its Zeppelin to Databricks Notebook Converter, delivered as a Databricks App, deterministically maps paragraphs to cells, translates interpreter syntax, reformats metadata as .ipynb JSON, and preserves original content. A generated, environment-specific prompt then gives Genie context about interpreters, data sources, and configuration patterns so it can reconstruct logic, ask clarifying questions, and handle the variable work. The hybrid workflow reduced redevelopment to 15–20 minutes per notebook, while keeping SQL, Python, visualizations, widgets, Oracle and HDFS references, scheduling, and custom business logic untouched; large-scale testing is next.

Context

Cloudera is fully decommissioning Zeppelin in 2027, while Deutsche Börse Group is moving analytics workloads to the cloud and adopting Databricks as its unified analytics platform. The migration covers a high volume of notebooks used by more than 2,000 people, many of which contain heterogeneous, business-critical logic and are embedded in daily workflows.

Approach / What changed

A Databricks App performs deterministic structural conversion: it maps Zeppelin paragraphs to Databricks cells, translates interpreter syntax, reformats metadata into valid .ipynb JSON, and preserves original content. It also generates environment-specific prompts for Genie, which reconstructs variable notebook logic, asks clarifying questions, and handles elements the converter intentionally does not rewrite.

Takeaways

• The converter automates notebook structure but leaves SQL, Python, visualizations, widgets, Oracle and HDFS references, scheduling logic, and custom business code untouched.
• Environment-specific Genie prompts include custom interpreters, data sources, and configuration patterns to improve logic reconstruction and produce more usable results.
• The hybrid workflow reduced manual notebook redevelopment to 15–20 minutes per notebook, and business users can perform migrations without a dedicated engineering team.

Related reading