---
title: "That Old Certificate Expired and Started an Outage. This is What Happened Next"
description: "On September 30th, 2021, Shopify nearly suffered a platform-wide outage after Let’s Encrypt’s old root certificate expired, exposing stale certificate data in internal clients rather than at the public edge. At 10:15 a.m., monitoring detected certificate errors after deployment-related connection changes, and investigation traced them to Ruby services that depended directly or indirectly on httpclient, whose embedded cacert.pem was six years old and did not trust the applicable certificate chain. Shopify forked the library, replaced cacert.pem with a current root certificate bundle, deployed it to canaries, and automated pull requests for affected repositories. The temporary fix restored the monolith, while systems were added to track root-certificate changes and update the fork automatically. Deployment locking kept most monolith pods from cycling, limiting the incident to canaries and causing secondary effects including delayed order confirmations, throttled marketing campaigns, and missing product recommendations."
---

# That Old Certificate Expired and Started an Outage. This is What Happened Next

[Shopify](https://yomu.fyi/company/shopify) · 2023-10-18 · Jan 12, 2022

**Type:** Incident / postmortem

## Summary

On September 30th, 2021, Shopify nearly suffered a platform-wide outage after Let’s Encrypt’s old root certificate expired, exposing stale certificate data in internal clients rather than at the public edge. At 10:15 a.m., monitoring detected certificate errors after deployment-related connection changes, and investigation traced them to Ruby services that depended directly or indirectly on httpclient, whose embedded cacert.pem was six years old and did not trust the applicable certificate chain. Shopify forked the library, replaced cacert.pem with a current root certificate bundle, deployed it to canaries, and automated pull requests for affected repositories. The temporary fix restored the monolith, while systems were added to track root-certificate changes and update the fork automatically. Deployment locking kept most monolith pods from cycling, limiting the incident to canaries and causing secondary effects including delayed order confirmations, throttled marketing campaigns, and missing product recommendations.

## Context

Shopify used Let’s Encrypt as its public certificate provider, but several internal services relied on httpclient, which embedded a six-year-old root certificate store. When the old Let’s Encrypt root certificate expired, newly initialized internal TLS connections began failing after deployments recreated Kubernetes pods.

## Approach / What changed

Shopify locked monolith deployments, identified httpclient as the shared dependency, forked its repository, replaced cacert.pem with a current root certificate bundle, validated the change in canaries, and used automation to open pull requests for affected repositories. It also added tracking and automatic updates for root-certificate changes, while identifying the system certificate store as a possible longer-term approach.

## Takeaways

- httpclient embedded its own root certificate store, and its six-year-old cacert.pem caused internal TLS validation failures after the Let’s Encrypt root certificate expired.
- HTTP keep-alive delayed the incident’s visible effects because TLS validation occurred only when connections were initialized; Kubernetes pod recreation during deployment forced new connections.
- Locking deployments limited the Shopify monolith impact to canaries, while dependent systems experienced missing product recommendations, throttled marketing campaigns, and delayed order confirmations.

**Tags:** [Incident Response](https://yomu.fyi/topic/incident-response), [Kubernetes](https://yomu.fyi/topic/kubernetes), [Monitoring](https://yomu.fyi/topic/monitoring), [Reliability](https://yomu.fyi/topic/reliability)

- Source: [Shopify](https://shopify.engineering/that-old-certificate-expired-and-started-an-outage-this-is-what-happened-next)
- Source URL: https://shopify.engineering/that-old-certificate-expired-and-started-an-outage-this-is-what-happened-next
- Ingested by Yomu: 2026-08-30T15:28:07.323Z

[Read original post](https://shopify.engineering/that-old-certificate-expired-and-started-an-outage-this-is-what-happened-next)