# PII masking for privacy-grade machine learning

[Grab](https://yomu.fyi/company/grab) · Fabrice Harbulot · Jun 1, 2023

**Type:** Problem & solution

## Summary

Data engineers at Grab require real-world streaming data to tune machine learning models, but user personal data cannot be exposed in non-production environments. To solve this, the data streaming team implemented an automated masking pipeline between production and staging Kafka clusters. Developers explicitly tag Personally Identifiable Information (PII) types in Protocol Buffers schemas, which a CI script validates against a keyword list before packaging the schemas into Scala JARs. An in-house Apache Flink application running in production consumes the unmasked streams, applies dynamic and consistent transformations like keyed HMACs or seeded random values, and produces sanitised records to staging. Untagged new fields in schema updates are automatically dropped by the deserialiser until the masking application is redeployed, preventing accidental data leaks.

## Context

Data engineers need access to realistic production streaming data to train and tune machine learning models, but privacy regulations and security policies forbid granting them direct access to unmasked user PII in production.

## Approach / What changed

Developers tag PII fields within Protobuf schemas, which are verified via CI keyword-scanning scripts and compiled into Scala JARs for an Apache Flink application that consumes production Kafka streams, dynamically applies consistent HMAC or seeded masking, and outputs sanitised data to a staging Kafka cluster.

## Takeaways

- CI pipelines validate Protobuf schema changes against an extensive PII keyword list, failing merge requests unless matching fields are tagged with specific PII types or added to a verified whitelist.
- Data masking maintains consistency for ML model training by using keyed HMACs for strings and unmasked values as seeds for numerical and date randomisation.
- If a schema introduces a new PII-tagged field before the Flink masking application is recompiled and redeployed, the deserialiser drops the field, preventing unmasked data from reaching staging.

**Tags:** [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Kafka](https://yomu.fyi/topic/kafka), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Privacy](https://yomu.fyi/topic/privacy), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://engineering.grab.com/pii-masking)