# Catching Anomalies Early in Mobile App Releases

[Nextdoor](https://yomu.fyi/company/nextdoor) · Walt Leung · Jan 11, 2023

**Type:** Problem & solution

## Summary

Nextdoor deploys weekly mobile updates across iOS and Android using phased rollouts starting at 1% adoption to minimize blast radius. Standard aggregate observability cannot reliably detect early regressions because early adopters skew significantly more active than average users, obscuring silent drops in overall metric noise. To overcome this selection bias, Nextdoor built App Release Anomaly Detection using difference-in-differences causal inference. The methodology verifies pre-adoption trends with standard deviation bounds and fits a linear regression model to estimate the effect against an unobserved counterfactual. During a rollout of iOS version v1.234.5, this model identified a statistically significant app session decline at 1% adoption, isolating the regression 10 days earlier than week-over-week metrics.

## Context

Nextdoor runs weekly iOS and Android releases to tens of millions of weekly active users using phased rollouts starting at 1% adoption. Standard aggregate observability fails to catch silent regressions early because small percentage drops get lost in aggregate noise, and early adopters inherently skew more active than the median user base.

## Approach / What changed

Nextdoor developed App Release Anomaly Detection using difference-in-differences causal inference analysis. After verifying that adopters and non-adopters exhibit similar pre-adoption trends using standard deviation bounds over the preceding days, the system fits a linear regression model (y = β0 + β1\*Time\_Period + β2\*Treated + β3\*(Time\_Period\*Treated) + e) to estimate release impact against an unobserved counterfactual.

## Takeaways

- Phased mobile rollouts skew early metrics because daily users adopt new releases faster than weekly or monthly users, preventing direct baseline comparisons.
- Difference-in-differences analysis models the unobserved counterfactual by measuring pre- and post-release metric changes in adopters relative to non-adopters.
- Nextdoor's causal inference model caught an iOS session regression at 1% rollout on March 5th, detecting it 10 days earlier than traditional week-over-week aggregate monitoring.

**Tags:** [Android](https://yomu.fyi/topic/android), [Deployment](https://yomu.fyi/topic/deployment), [iOS](https://yomu.fyi/topic/ios), [Observability](https://yomu.fyi/topic/observability), [Reliability](https://yomu.fyi/topic/reliability)

[Read original post](https://engblog.nextdoor.com/catching-anomalies-early-in-mobile-app-releases-ac95adf9da81)