Loading…
Topic
2 posts about ETL. Every summary links to the original.
See the idea
AWS Glue runs serverless Spark jobs that transform data in batches; engine-native pipelines transform during ingestion. This comparison shows how to pick between them by freshness, cost, and data fidelity.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Repointing a dbt project to Databricks involves swapping the adapter and profile, handling SQL dialect differences, and keeping transformation logic unchanged. The dbt-databricks adapter bridges the project to Databricks Lakehouse, which uses a three-level namespace (catalog.schema.table) and supports open formats like Delta Lake and Apache Iceberg. Common dialect gaps include REGEXP_SUBSTR, DIV0, and OBJECT_CONSTRUCT, which can be fixed with Databricks-native functions or dbt_utils macros; the post walks through each error and its fix. After compilation and validation using audit-helper to compare outputs model by model, the project can be deployed as a dbt task in Lakeflow Jobs with built-in governance via Unity Catalog. The process is incremental: one-time setup costs apply, then each additional model requires only dialect fixes, making it a low-risk starting point for warehouse migration.
Data transformation turns raw data extracted from source systems into usable, trusted datasets for operational needs, analytics and decision-making. The process addresses inconsistencies, missing values and duplicates through techniques such as cleansing, de-duplication, integration, aggregation, validation and format revision, often mapping data to a target schema during ETL. Performance considerations include compaction, partitioning, file-size tuning, data skipping and pruning, and caching to reduce storage, scanning, processing time and latency. The article also compares on-premises, cloud, open-source, proprietary, no-code, low-code and codable ETL tools, then describes Spark Declarative Pipelines and Lakeflow Jobs on Databricks for quality checks, pipeline development, monitoring and orchestration. A Navy Federal Credit Union proof of concept illustrates a rapid deployment, with a proof of concept completed in a week and a dashboard released days after production deployment.
The best engineering reads, once a week.
