# Processing ETL tasks with Ratchet

[Grab](https://yomu.fyi/company/grab) · Amar Prakash · Jul 19, 2021

**Type:** Explainer

## Summary

Grab's Lending team relies on multiple microservices to finance various user segments, requiring automated pipelines for operational tasks like loan rescheduling and merchant whitelisting. To execute these workflows, the team uses Ratchet, a Go library that organizes data pipelines into concurrent stages connected by Go channels and JSON payloads. In the Azkaban service, each ETL task is structured as a Job Processor that configures custom Data Processors for extraction, transformation, and loading. For merchant whitelisting, a custom processor pulls uploaded CSV files from Amazon S3, validates the rows to prevent manual errors, calls microservice APIs, and passes the output to an empty load stage. This pipeline architecture allows the team to isolate failures quickly, eliminate manual data validation, and automate routine operational tasks.

## Context

Grab's Lending team needed to handle recurring operational data workflows, such as batch processing, profile creation, and merchant whitelisting, while preventing manual human error from propagating across microservices.

## Approach / What changed

The team implemented ETL pipelines within their Azkaban service using Ratchet, a Go-based data pipeline library that executes concurrent Data Processors for extraction, transformation, and loading linked by Go channels.

## Takeaways

- Ratchet uses Go channels to pass JSON data between stages, running each Data Processor concurrently in its own goroutine.
- The Azkaban service models pipelines using a Job Processor interface with SetSource, SetTransformer, SetDestination, and Execute methods.
- For workloads like merchant whitelisting where transformed data does not require long-term storage, the pipeline completes using an Empty Data Processor for the load stage.

**Tags:** [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Go](https://yomu.fyi/topic/go), [Microservices](https://yomu.fyi/topic/microservices)

[Read original post](https://engineering.grab.com/processing-etl-tasks-with-ratchet)