---
title: "AWS Glue vs. Transform-at-Ingest: Where Should the Work Happen?"
description: "AWS Glue runs serverless Spark jobs that transform data in batches; engine-native pipelines transform during ingestion. This comparison shows how to pick between them by freshness, cost, and data fidelity."
---

# AWS Glue vs. Transform-at-Ingest: Where Should the Work Happen?

**Type:** [Comparison](https://yomu.fyi/visual-explainers/category/comparison.md)

![A two-column comparison diagram: the left column shows AWS Glue as a serverless batch transform layer with Spark workers and connectors, the right shows engine-native transform during ingestion, and a bottom decision band maps freshness, cost, and type fidelity to a choice.](https://yomufyi.s3.us-east-1.amazonaws.com/visual-explainers/39543e5e-3dc0-428a-820d-c01d58ce7562/image-d36a41d1-a614-4661-9841-a41bca93a6c8.webp)

> AWS Glue runs serverless Spark jobs that transform data in batches; engine-native pipelines transform during ingestion. This comparison shows how to pick between them by freshness, cost, and data fidelity.

## Explanation

Transformation can happen in two places: in a batch job that prepares data before it is queried, or inside the engine that serves the query. AWS Glue sits in the first camp, offering serverless Spark workers, built-in connectors, and pay-per-DPU-hour billing, but its connectors can flatten types and both cost estimation and job startup need planning. Engine-native approaches push the work into ingestion, where asynchronous inserts and pre-aggregations make data query-ready as it lands, instead of relying on separate background sorting and refresh steps. The right choice depends on how fresh the data must be, what each prepared byte costs, and whether the source's type fidelity matters downstream.

## FAQ

### What does the Glue DynamoDB connector cost you in type fidelity?

The connector simplifies data types, so distinctions such as string set versus list of strings can be lost before your transform even runs. Downstream logic and round-trip writes may therefore not preserve the original shape of the data. Validate type mapping early, and keep point-in-time recovery enabled for mutating commands.

### Does upgrading the AWS Glue version actually change job cost and startup?

Yes. Teams migrating from Glue 4 to 5 and 6 reported faster startup and roughly 30 percent lower DPU costs, and since billing is per DPU-hour both startup time and worker count feed the bill. Test the upgrade against your own jobs, because connector behavior and bundled library versions can shift.

**Tags:** [Apache Spark](https://yomu.fyi/topic/apache-spark.md), [AWS](https://yomu.fyi/topic/aws.md), [AWS Glue](https://yomu.fyi/topic/aws-glue.md), [Cost Optimization](https://yomu.fyi/topic/cost-optimization.md), [Data Pipelines](https://yomu.fyi/topic/data-pipelines.md), [ETL](https://yomu.fyi/topic/etl.md)

## Related reading

### [OLTP vs. OLAP: Choosing Between Transactional and Analytical Databases](https://yomu.fyi/post/oltp-vs-olap-choosing-between-transactional-and-analytical-databases.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Oct 14, 2026
- Type: Explainer

OLTP handles high-volume, low-latency transactions on current operational data for online banking, e-commerce, and inventory, while OLAP handles complex analytical queries on large historical datasets for financial analysis and business intelligence. OLTP databases ensure data integrity through ACID compliance and use normalized row-oriented tables; OLAP systems use denormalized dimensional models and columnar storage for efficient scans and aggregations. Data pipelines such as ETL or CDC tools like Debezium connect OLTP and OLAP systems, but latency and pipeline complexity rise with analytical freshness requirements. For mixed workloads, HTAP or lakehouse architectures can reduce synchronization overhead and analytical lag. The choice between OLTP and OLAP depends on the workload’s read-write pattern, latency needs, and data volume, with TPC-C and TPC-H as reference benchmarks.


### [What is HTAP? A plain-English guide to hybrid transactional/analytical processing](https://yomu.fyi/post/what-is-htap-a-plain-english-guide-to-hybrid-transactional-analytical.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Oct 14, 2026
- Type: Explainer

HTAP (Hybrid Transactional/Analytical Processing) brings transaction processing and analytics together so applications can run analytical queries on current operational data without waiting for batch data pipelines. This architecture addresses the delay caused by traditional separation of OLTP and OLAP systems, where data had to be copied into a data warehouse before analysis. HTAP designs vary: converged systems use a single database engine for both workloads, while composed systems use separate engines connected through shared storage or replication. The central challenge is providing analytical access to fresh operational data without compromising transactional performance and reliability. For workloads needing independent scaling, Lake Transactional/Analytical Processing (LTAP) offers a unified storage layer under one governance model, reducing the need for change data capture and replication pipelines.


### [How Ridgeline unified Iceberg and ClickHouse into one analytics layer for investment management](https://yomu.fyi/post/how-ridgeline-unified-iceberg-and-clickhouse-into-one-analytics-layer.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Oct 14, 2026
- Type: Problem & solution

Ridgeline, a cloud-native investment management platform, rebuilt its performance analytics layer after Postgres could not handle the scale of larger customers. The team evaluated Redshift, SingleStore, Trino over Iceberg, and a custom in-memory engine before choosing ClickHouse for its analytical performance and lower cost. They adopted ClickHouse Cloud on AWS, using separate organizations for staging and production and per-customer services, with assistance from ClickHouse engineers. Historical data lands in Apache Iceberg via a Spark pipeline and is queried through ClickHouse, while recent data is written directly into ClickHouse. The architecture enabled Ridgeline to serve customers 10x larger than before and more than double its live customer count.


### [Why Redshift Serverless can’t match ClickHouse Cloud for real-time analytics](https://yomu.fyi/post/why-redshift-serverless-can-t-match-clickhouse-cloud-for-real-time-ana.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Oct 14, 2026
- Type: Benchmark

CostBench streamed 113.2 billion stock-market quotes into both systems while four aggregate queries ran every ten minutes and two drill-down queries ran every hour. ClickHouse asynchronous inserts order raw data and build pre-aggregations during ingestion, while Redshift relies on separate background sorting and materialized view refreshes. ClickHouse Cloud led in preparation cost ($28.69 versus 522.03), normalized query cost ( 0.05 versus $24.97), and accumulated query runtime (58.12 seconds versus 1.69 hours). The combined score shows ClickHouse Cloud delivered 1,996× better end-to-end performance per dollar in this continuous-ingestion comparison. The difference originates in the preparation path: ClickHouse makes data query-ready during inserts, whereas Redshift’s streaming, sorting, and refresh steps advance independently.


### [How to Rewrite Apache Spark UDFs for GPU Acceleration with AI Agents](https://yomu.fyi/post/how-to-rewrite-apache-spark-udfs-for-gpu-acceleration-with-ai-agents.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Elizabeth Goodman
- Published: Oct 13, 2026
- Type: Tutorial

User-defined functions (UDFs) in Apache Spark break the declarative abstraction, forcing GPU-accelerated pipelines to fall back to CPU execution and incur data transfer overhead. NVIDIA cuDF addresses this with GPU parallelism, but UDFs remain a bottleneck due to their opaque code. The post describes three approaches to rewrite UDFs into GPU-accelerable forms: SQL expressions, the RapidsUDF interface using cuDF columnar operations, or custom CUDA kernels via JNI. The cuDF skills for Apache Spark automate this process with four agent-guided feedback loops for testing, conversion, benchmarking, and optimization, providing a project template and best practices. A production deployment converted over 30 UDFs, achieving a 22x speedup on a 600-billion-row workload, reducing operator time from 274.55 to 12.78 hours.


### [Bulk Executor for Amazon DynamoDB, Part 2: Lessons we learned the hard way](https://yomu.fyi/post/bulk-executor-for-amazon-dynamodb-part-2-lessons-we-learned-the-hard-w.md)
- Company: [AWS](https://yomu.fyi/company/aws.md)
- Author: Jason Hunter
- Published: Oct 13, 2026
- Type: Problem & solution

The post describes design tradeoffs and workarounds encountered while building the open-source Bulk Executor tool for Amazon DynamoDB on AWS Glue. Core requirements included real-time local-script feel, scaling to hundreds of parallel workers without user infrastructure management, and easy command extensibility. AWS Glue was chosen for serverless parallelism and a built-in DynamoDB connector, but the connector simplified data types, losing distinctions like string set versus list of strings. Challenges included estimating costs for provisioned tables, handling job termination with ^C, and streaming output via CloudWatch Live Tail which samples events and delivers out-of-order. Solutions included a two-phase cost estimate for deletes, programmatic interrupt handling, writing large output to S3 with a preview, mandating point-in-time recovery for mutating commands, and using AI for testing and linting; the team also migrated from AWS Glue 4 to 5 and 6 for faster startup and 30 percent lower DPU costs.