# The Curious Case of the Phantom Instance

[Grab](https://yomu.fyi/company/grab) · Lian Yuanlin · Dec 28, 2015

**Type:** Problem & solution

## Summary

Datadog dashboards for the grab\_attention cluster displayed periodic 1.5X step increases in Elastic Load Balancer (ELB) health check requests and ElastiCache Redis connections, creating the illusion of an untracked instance running outside Auto Scaling Group records. Inspecting instance hostname tags revealed that the existing two instances were simply receiving elevated ping counts from the load balancers. AWS Support clarified that ELB scaling events provision new nodes while keeping old nodes running for roughly 90 minutes to handle cached DNS clients. Investigation also revealed that two separate ELBs were attached to the cluster, altering expected request baselines. Furthermore, the application's health check endpoint initiated a non-pooled Redis connection on every request, directly translating load balancer pings into database connection spikes.

## Context

Datadog and CloudWatch metrics showed 1.5X step-waveform spikes in ELB health check counts and ElastiCache Redis connections, but Auto Scaling Group history showed no matching scaling events or new server IP addresses.

## Approach / What changed

The team analyzed Datadog metrics by instance hostname tags, audited the ELB configuration and health check code, and consulted AWS Support to determine why health check and connection counts elevated.

## Takeaways

- ELB scaling events can temporarily double health check counts because new nodes are deployed and old nodes remain active for approximately 90 minutes to handle cached DNS queries.
- The presence of multiple load balancers—in this case, separate internal and public ELBs configured with 5-second check intervals—cumulatively increases total incoming health check requests.
- Executing unpooled database pings inside an application's health check endpoint causes external load balancer health check spikes to propagate directly as new connection spikes to underlying services like Redis.

**Tags:** [AWS](https://yomu.fyi/topic/aws), [Monitoring](https://yomu.fyi/topic/monitoring), [Observability](https://yomu.fyi/topic/observability), [Redis](https://yomu.fyi/topic/redis)

[Read original post](https://engineering.grab.com/curious-case-of-the-phantom-instance)