AWSRichard WhitworthTutorial
The post presents a resilience-testing tutorial for applications that depend on Amazon Simple Queue Service (Amazon SQS), where failed queue operations can stall downstream processing and trigger cascading outages. It uses AWS Fault Injection Service (AWS FIS) and AWS Systems Manager (SSM) Automation to apply a scoped deny resource policy that blocks data-plane actions, including sending, receiving, deleting, changing visibility, and purging, while preserving queue management. The experiment increases disruption across four phases, then observes producer-side and consumer-side behavior through application and Amazon CloudWatch metrics, including circuit-breaker state, failed sends, backlog, message age, duplicates, and dead-letter-queue effects. The guidance emphasizes hypotheses with measurable success criteria, non-production guardrails, tagged target queues, and a stop condition based on customer impact. It also describes consumer-only, partial, and compound-failure extensions, warning that concurrent experiments on one queue can leave a stale deny policy.
Applications may respond similarly when SQS operations fail because of an IAM misconfiguration, network partition, bad deployment, or transient service event. The post focuses on discovering whether failure-handling and observability mechanisms work when access to SQS disappears, rather than verifying SQS itself.
AWS FIS runs an experiment through SSM Automation, applying a scoped deny resource policy to tagged SQS queues, increasing disruption across four phases, restoring access, and using application and CloudWatch metrics to assess producer and consumer behavior and recovery. The policy must deny data-plane actions only, not queue-management operations.
The deny policy should cover only SQS data-plane actions; denying sqs:* can also block the automation’s ability to remove the policy and lock the queue.
Producer and consumer failures expose different behaviors: failed sends test buffering and circuit breakers, while denied receives and deletes reveal backlog growth, redelivery, and DLQ pressure after recovery.
Concurrent impairment experiments on the same queue are unsafe because read-modify-write policy updates can overwrite one another and leave a stale deny policy behind.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
Amazon SQS connects producers and workers, but its real design test is failure. Compare the processing path with the signals and controls needed when queue operations break.
Learn the step-by-step process for deploying .NET 10 applications to Heroku, from buildpack detection to SDK pinning, and understand the lifecycle of .NET LTS releases.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
