Automate custom PII detection at scale with Amazon Macie and Step Functions

Summary

Regulated organizations ingesting large data volumes must detect and classify standard and domain-specific personally identifiable information to satisfy compliance mandates like GDPR, HIPAA, CCPA, and PCI DSS. To eliminate manual inspection, an event-driven architecture orchestrates Amazon Macie scans for files landing in Amazon S3. Amazon EventBridge captures upload events and invokes AWS Step Functions, which manages Lambda functions to stage objects, trigger Macie classification jobs with custom regex identifiers, and poll for results. The pipeline separates data across a three-bucket pattern representing raw, staged, and scanned states to prevent mixing unprocessed data with validated assets. Finally, the workflow exports timestamped compliance reports in CSV and JSON formats while alerting security teams via Amazon SNS on high-severity findings.

Context

Organizations in regulated industries ingest large volumes of data containing standard PII and custom identifiers, creating a compliance requirement under regulations like GDPR, HIPAA, CCPA, and PCI DSS to locate and protect sensitive data without relying on manual classification.

Approach / What changed

Deploy an event-driven AWS architecture where EventBridge catches S3 uploads, Step Functions orchestrates Lambda and Macie using custom data identifiers across a three-bucket lifecycle (raw, staged, scanned), and SNS alerts on high-severity findings while generating CSV and JSON compliance reports.

Takeaways

• Amazon Macie accounts support up to 10,000 custom data identifiers, but individual classification jobs are limited to a maximum of 30 custom identifiers.
• Macie imposes a CreateClassificationJob throttle limit of 0.1 requests per second (one job every 10 seconds), requiring batching for high-volume workloads.
• The architecture enforces a three-bucket pattern (raw, staged, and scanned) to isolate objects by processing state and ensure unscanned files never mix with validated data.

Related reading

• AWS · Aug 19, 2026

  How Clario technology detects PHI/PII in DICOM images using Amazon Bedrock

  Clario developed an automated PHI and PII detection solution on AWS to process thousands of DICOM image slices and PDF files in clinical trials. Medical imaging workflows require strict privacy compliance under HIPAA, GDPR, and ICH E6 guidelines across standard tags, custom vendor metadata, and pixel-burned text. To address this, the pipeline retrieves files from Amazon S3 and runs memory-intensive detection workloads on Amazon EKS backed by Amazon RDS for PostgreSQL. Amazon Textract handles optical character recognition, while Anthropic's Claude Sonnet on Amazon Bedrock classifies sensitive data using domain-optimized prompt engineering. The system identifies precise coordinates and sensitive entity types rather than directly altering files, enabling human-in-the-loop review before downstream redaction occurs.

  Alex Boudreau

• AWS · Jul 22, 2026

  Building multi-Region resiliency for AWS CloudFormation custom resource deployment