---
title: "A Decision Framework for ETL Migration to Databricks"
description: "An ETL migration with hundreds of stored procedures, schedulers, scattered permissions, and a warehouse renewal deadline needs workload-by-workload decisions rather than a single rewrite strategy. The framework assigns work among Databricks SQL, Spark Declarative Pipelines (SDP), and PySpark or Spark SQL notebooks. SQL tasks suit single statements, Unity Catalog-governed stored procedures handle procedural logic, SDP manages dependencies, retries, quality constraints, and batch-plus-streaming, while notebooks cover complex logic, ML feature engineering, integrations, and large or tightly controlled Spark workloads. It recommends four phases—assessment, quick wins, modernization, and optimization—using profiling, side-by-side validation, and parallel runs before retiring legacy systems. Migration tools can automate 60–80% of initial conversion, but architecture choices remain essential: the goal is consolidating orchestration, metadata, lineage, permissions, and validation rather than reproducing technical debt."
---

# A Decision Framework for ETL Migration to Databricks

[Databricks](https://yomu.fyi/company/databricks) · Rafael Aielo · Jun 26, 2026

**Type:** Explainer

## Summary

An ETL migration with hundreds of stored procedures, schedulers, scattered permissions, and a warehouse renewal deadline needs workload-by-workload decisions rather than a single rewrite strategy. The framework assigns work among Databricks SQL, Spark Declarative Pipelines (SDP), and PySpark or Spark SQL notebooks. SQL tasks suit single statements, Unity Catalog-governed stored procedures handle procedural logic, SDP manages dependencies, retries, quality constraints, and batch-plus-streaming, while notebooks cover complex logic, ML feature engineering, integrations, and large or tightly controlled Spark workloads. It recommends four phases—assessment, quick wins, modernization, and optimization—using profiling, side-by-side validation, and parallel runs before retiring legacy systems. Migration tools can automate 60–80% of initial conversion, but architecture choices remain essential: the goal is consolidating orchestration, metadata, lineage, permissions, and validation rather than reproducing technical debt.

## Context

The migration involves hundreds of stored procedures, multiple schedulers, permissions distributed across roles and schemas, and an approaching cloud data warehouse renewal deadline. Teams disagree between rewriting everything in PySpark and moving SQL unchanged, while metadata, lineage, permissions, and opportunities to consolidate them also require attention.

## Approach / What changed

Classify each workload by logic, team skills, complexity, resource use, and dependencies, then select Databricks SQL, Spark Declarative Pipelines, or PySpark and Spark SQL notebooks. Execute the migration in assessment, quick-win, modernization, and optimization phases, using migration tooling for profiling and conversion, parallel runs for validation, and Unity Catalog, Workflows, and declarative pipelines to consolidate platform capabilities.

## Takeaways

- Single-statement SQL should become a SQL task; procedural logic involving variables, loops, parameters, or error handling belongs in a Unity Catalog-governed stored procedure callable from Workflows.
- Spark Declarative Pipelines provide automatic dependency management, retries, scaling, data-quality constraints, and unified batch-plus-streaming definitions, while Enzyme chooses incremental updates or full recomputation.
- Migration tooling can handle roughly 60–80% of initial conversion, but teams should reserve engineering effort for modernization and validate systems side by side with schemas, row counts, checksums, and aggregates.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Databricks](https://yomu.fyi/topic/databricks), [Migrations](https://yomu.fyi/topic/migration), [Spark](https://yomu.fyi/topic/spark), [Unity Catalog](https://yomu.fyi/topic/unity-catalog)

- Source: [Databricks](https://www.databricks.com/blog/decision-framework-etl-migration-databricks)
- Source URL: https://www.databricks.com/blog/decision-framework-etl-migration-databricks
- Ingested by Yomu: 2026-08-30T16:59:49.779Z

[Read original post](https://www.databricks.com/blog/decision-framework-etl-migration-databricks)