---
title: "Erlang"
description: "4 posts about Erlang, summarised, each linking to the original."
---

# Erlang
> 4 posts about Erlang, summarised, each linking to the original.

## Articles

### [Latency Incident Report: Socket Bind Before Connect](https://yomu.fyi/post/latency-incident-report-socket-bind-before-connect.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Lex Neva
- Published: Mar 30, 2017
- Type: Incident / postmortem

An August 11, 2016 incident caused increased routing latency in Heroku’s EU Common Runtime, even though CPU, packet, memory, request, and socket metrics remained normal. The routing layer received EADDRINUSE while connecting to dynos, leading engineers to investigate Linux TCP port allocation, TIME\_WAIT, and the Erlang VM’s system calls with strace. They found that the VM called bind() with 0.0.0.0:0 before connect(), forcing Linux to allocate an ephemeral source port without knowing the destination; this made the ephemeral port range an effective per-instance traffic limit and could fail before 65,536 connections. Testing reproduced the failures and showed that setting SO\_REUSEADDR before bind(), exposed in Erlang as {reuseaddr, true}, removed the bind errors; the corresponding Vegur patch was deployed a few days later.


### [How We Sped up SNI TLS Handshakes by 5x](https://yomu.fyi/post/how-we-sped-up-sni-tls-handshakes-by-5x.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Fred Hebert
- Published: Dec 22, 2016
- Type: Incident / postmortem

Heroku’s SNI TLS termination work exposed severe handshake performance problems under benchmarks designed to avoid session resumption, caching, and keep-alive reuse. Initial tests made nodes slow and unresponsive 7–8x earlier than comparable HTTP tests, while perf top showed Erlang garbage collection and ETS hash operations consuming unexpectedly large amounts of CPU. Tracing linked repeated ets.select\_delete calls to the centralized ssl\_manager process, where ssl\_pkix\_db’s certificate cache path was still used even for DER certificates supplied directly in memory. Engineers patched the SSL library to bypass that cache wholesale. Each instance then handled three times the benchmark capacity, while SNI latencies fell from 100–120 milliseconds to 20–25 milliseconds for the reported app subset; the patch was later shipped in Erlang/OTP-19.1.


### [Heroku Proxying becomes Free Software](https://yomu.fyi/post/heroku-proxying-becomes-free-software.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Fred Hebert
- Published: Oct 21, 2015
- Type: Announcement

Heroku’s HTTP routing stack has three layers: state synchronization, request routing, and HTTP proxying. The platform team open-sourced the proxying layer as Vegur, an Erlang library that validates, normalizes, and forwards requests between clients and dynos. Unlike a conventional nginx reverse proxy, Vegur’s hosts and configuration are fully programmable, allowing state from hardcoded values, files, databases, or direct updates, while routing algorithms and debugging headers can also remain dynamic. Its delegated capabilities include domain lookup, backend selection, interface choice, resource checkout and checkin, customizable failure pages, per-application or per-backend controls for tunneling 100-continue requests, and upstream or downstream header insertion. Vegur had run in production on Heroku for over a year and powered the Heroku Private Spaces router.


### [Stuff Goes Bad](https://yomu.fyi/post/stuff-goes-bad.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Fred Hebert
- Published: Sep 17, 2014
- Type: Announcement

The Heroku Routing team describes a manual created to preserve hard-earned operational knowledge from developing and maintaining Erlang systems at scale. That knowledge had typically been passed informally through incident simulations, code reviews, and similar practices, leaving no persistent record for new team members. The manual presents itself as a guide to being an “Erlang medic” during failures, combining tips, tricks, code snippets, and practices for understanding failure sources and debugging running production systems. It is available as a free PDF under a Creative Commons License at erlang-in-anger.com, and was released in time for the Chicago Erlang conference, where Routing team members and Erlang community participants could discuss real-world applications.