---
title: "What is Data Transformation?"
description: "Data transformation turns raw data extracted from source systems into usable, trusted datasets for operational needs, analytics and decision-making. The process addresses inconsistencies, missing values and duplicates through techniques such as cleansing, de-duplication, integration, aggregation, validation and format revision, often mapping data to a target schema during ETL. Performance considerations include compaction, partitioning, file-size tuning, data skipping and pruning, and caching to reduce storage, scanning, processing time and latency. The article also compares on-premises, cloud, open-source, proprietary, no-code, low-code and codable ETL tools, then describes Spark Declarative Pipelines and Lakeflow Jobs on Databricks for quality checks, pipeline development, monitoring and orchestration. A Navy Federal Credit Union proof of concept illustrates a rapid deployment, with a proof of concept completed in a week and a dashboard released days after production deployment."
---

# What is Data Transformation?

[Databricks](https://yomu.fyi/company/databricks) · Databricks Staff · Sep 3, 2026

**Type:** Explainer

## Summary

Data transformation turns raw data extracted from source systems into usable, trusted datasets for operational needs, analytics and decision-making. The process addresses inconsistencies, missing values and duplicates through techniques such as cleansing, de-duplication, integration, aggregation, validation and format revision, often mapping data to a target schema during ETL. Performance considerations include compaction, partitioning, file-size tuning, data skipping and pruning, and caching to reduce storage, scanning, processing time and latency. The article also compares on-premises, cloud, open-source, proprietary, no-code, low-code and codable ETL tools, then describes Spark Declarative Pipelines and Lakeflow Jobs on Databricks for quality checks, pipeline development, monitoring and orchestration. A Navy Federal Credit Union proof of concept illustrates a rapid deployment, with a proof of concept completed in a week and a dashboard released days after production deployment.

## Context

Organizations collect large volumes of data that can contain inconsistencies, missing values, duplicates, disparate formats or structures, making it difficult to use for trustworthy analysis, integration, reporting and decision-making.

## Approach / What changed

Data is cleansed, mapped, standardized and transformed, often to a specific schema, before being loaded through ETL. The text describes transformation techniques, performance optimizations, ETL tool categories and Databricks capabilities for quality checks, pipeline development, monitoring and orchestration.

## Takeaways

- In ETL, transformation is the stage where data is cleansed, mapped and converted into a usable resource, often according to a specific schema, before loading.
- Partitioning can improve query performance by enabling parallel processing and reducing the amount of data scanned; data skipping and pruning further limit unnecessary reads.
- Spark Declarative Pipelines includes Expectations for applying data quality checks to every record passing through a query, while Lakeflow Jobs manages and monitors multitask workflows.

**Tags:** [Data Engineering](https://yomu.fyi/topic/data-engineering), [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Data Quality](https://yomu.fyi/topic/data-quality), [ETL](https://yomu.fyi/topic/etl)

- Source: [Databricks](https://www.databricks.com/blog/what-is-data-transformation)
- Source URL: https://www.databricks.com/blog/what-is-data-transformation
- Ingested by Yomu: 2026-09-04T00:01:01.965Z

[Read original post](https://www.databricks.com/blog/what-is-data-transformation)