Loading…
Topic
Go
89 posts about Go. Every summary links to the original.
- 89 posts
- 15 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Building ClickCannon - a tool for benchmarking ClickHouse
ClickCannon was created to answer how much CPU, memory, and disk are needed for ClickStack observability workloads, while accounting for schema, ingestion rate, hardware, and concurrent queries. It replays about 2 TiB of real OpenTelemetry trace data, converted to ClickHouse Native format with ZSTD compression, rather than relying only on synthetic datasets. A custom multithreaded Go reader/inserter provides control over uncompressed throughput and exposes stream metrics such as compressed and uncompressed bytes, row counts, and block counts, while the framework simulates users and query workloads during ingestion. The project evolved into an open-source, generic ClickHouse benchmarking tool for custom insert and query workloads, with fine-grained control over throughput, concurrency, and user behavior; the post reports that it can push substantial throughput from a single machine.
Zero-Growth Stack, Real Gains: How Stack Allocation Can Save 10% CPU in Go
At Uber, repeated Go goroutine stack expansion consumed substantial CPU in a service whose stack-memory usage remained low, making memory a viable trade-off. Go starts goroutines with 2KB stacks and doubles them when needed, copying the previous stack; the post also notes that adaptive stack sizing and stack shrinking can add runtime cost. Uber patched the runtime, disabled adaptive sizing and shrinking, and used profile-guided analysis of pprof traces plus assembly-derived function stack usage to choose a static power-of-two starting size. For the profiled service, the analyzer identified about 19KB of required stack usage, leading to a 32KB setting instead of 2KB. After deployment, stack-growth CPU fell from close to 10% to below 1%, while most instances used about 50MB of stack and a few reached 200MB.
How GitHub uses eBPF to improve deployment safety
Deployment scripts can introduce dangerous circular dependencies when they rely on services or assets from platforms that are currently experiencing outages. Blocking network access at the host level is impractical because stateful nodes continue serving live traffic during rolling deployments. To solve this, GitHub isolates deploy scripts into dedicated Linux cGroups and attaches custom eBPF programs via the cilium/ebpf Go library. The system uses socket-address hooks to redirect DNS queries to a userspace proxy that checks a domain blocklist, while egress packet hooks map DNS transaction IDs to process IDs. This approach successfully prevents deploy-time circular dependencies, provides full command-line audit logs for blocked requests, and speeds up incident recovery.
From Custom to Open: Scalable Network Probing and HTTP/3 Readiness with Prometheus
Slack encountered a lack of client-side observability when introducing HTTP/3 at its edge. Because HTTP/3 runs over QUIC on UDP instead of TCP, neither existing commercial SaaS tools nor the internal Prometheus Blackbox Exporter could probe the new endpoints. To address this gap, an engineering intern integrated the open-source quic-go library into Prometheus Blackbox Exporter to build a native, configurable HTTP/3 probe. Slack deployed an internal implementation using the upstream functionality while waiting for pull request reviews, ultimately landing the contribution upstream. The resulting setup provides a unified view of HTTP/1.1, HTTP/2, and HTTP/3 metrics within Grafana, enabling reliable alerting and faster correlation with other telemetry.
Highly concurrent in-memory counter in GoLang
An overloaded database faced heavy traffic from campaign usage tracking, motivating an in-memory counter rather than an immediate SQL-to-NoSQL migration. Because exact counts were not required, increments could be cached in the application server and flushed periodically to reduce database queries and CPU use. The implementation uses Go’s Sync.Map, with LoadOrStore and CompareAndSwap for concurrent increments, plus LoadAndDelete to atomically remove values for asynchronous persistence; the source notes a fresh-map swap as an alternative locking optimization. On an Apple M4 Pro, the 2,000-key benchmark reported 53 ns/op and 1 billion operations per 60 seconds, versus 160 ns/op and 466 million operations for a Mutex-protected map, but Sync.Map used 88 B/op and four allocations. In production, update queries fell from 140 QPS to 45 QPS, master CPU from 35% to 18%, and replica CPU from 19% to 12%.
Taming the monorepo beast: Our journey to a leaner, faster GitLab repo
Grab’s decade-old Go monorepo grew to 12.7 million commits, 73GB of Git-tree metadata, 176GB of blobs, 12 million references, and 444,000 files, causing replication lag, slow developer workflows, and GitLab operational bottlenecks. A staging proof of concept compared full and single-commit histories under production-like load: replication delay fell from 160–240 seconds to 1–2.5 seconds, nearly a 100x improvement. Because custom migration tooling processed about 100 commits per hour, the team retained commits reachable from more than 2,000 critical tags, one month of recent history, and simplified merge commits, while archiving older history. The resulting prune reduced commits by 99.9%, improved replication performance by 99%, preserved code and tags without data loss, and left teams with minor migration adjustments.
My AI Skeptic Friends Are All Nuts
The essay argues that skepticism about LLMs for software development often reflects outdated experience with chat-based code generation rather than current agent workflows. Modern coding agents inspect repositories, edit files, invoke Unix tools and MCP calls, use Git, and run linters, compilers, tests, and model checkers, turning feedback into iterative repairs. The author says these systems can automate tedious typing, lookup, bookkeeping, test refactoring, and edit-compile-test-debug cycles while developers retain responsibility for curation, judgment, direction, and reviewing merged code. Hallucinations are presented as substantially mitigated by tool-based validation, although the author notes that agents remain unsuitable for some trusted operations and that effectiveness depends on language, tooling, prompts, indices, and guardrails. The conclusion is that AI-assisted programming is already materially changing developer productivity, despite its noise and unresolved limitations.
Deploying a Simple Go/Gin Application on Heroku
Deploying a Go/Gin application can be simple locally but requires explicit preparation for cloud deployment, including dependency management, environment configuration, and a process definition. The guide initializes a Go module, adds Gin with go get, runs main.go locally, and verifies the API with curl, using Go 1.24.0 in its example. It then creates a Heroku app, adds a Procfile with web: go run main.go, runs go mod tidy, commits the project, and pushes the main branch to the Heroku remote. Heroku detects the Go app through go.mod, installs Go 1.24.0 and dependencies, builds ./bin/go-gin, and launches the web process; the deployed /quote endpoint returns JSON, while heroku logs --tail confirms a successful request.
30 Minutes With MCP and flyctl
Kurt Mackey describes building a minimal MCP server for flyctl in 30 minutes, motivated by testing how an LLM could inspect deployed applications. MCP connects an LLM client to server-exported tools through JSON exchanges; this implementation uses stdio mode, a Go MCP library, and exposes only fly logs and fly status, producing roughly 90 lines of code. Existing JSON modes in flyctl commands made the tools easier to drive from automation. When pointed at unpkg through Claude, the server surfaced a globally distributed deployment, identified two critical machines, and connected repeated restarts and oom_killed events to Bun using about 3.7 GB of a machine’s 4 GB allocation. The post calls the result surprisingly useful while warning that local MCP servers give cloud-hosted Claude access to native programs and would be safer in isolation.
How Tinder Eased Development Pain With Ignis
Managing hundreds of microservices with disparate configurations forced Tinder engineers to deploy code to staging environments to verify every change, slowing development velocity. To shorten feedback loops, the team created Ignis, a Go-based CLI tool that lets engineers run an isolated service locally while proxying ingress and egress traffic through a multi-tenant development Kubernetes cluster. Ignis pairs with an API gateway for client testing and uses a custom Smart Router proxy to dynamically intercept downstream gRPC and HTTP requests via Kubernetes informers and header propagation. Since launching in 2019, the tool enabled Tinder to eliminate a dozen staging environments, decrease cloud spend, and accelerate onboarding, followed by a V2 release adding remote IDE support and monitoring.
How we improved translation experience with cost efficiency
Grab faced translation quality and cost challenges in its booking chat system when tourism resumed after COVID restrictions eased. Device language settings proved unreliable for determining message language, prompting the team to use Lingua alongside booking context heuristics and third-party fallbacks for real-time detection. To replace expensive and inaccurate general translation APIs, Grab distilled a large open-source Hugging Face model into lightweight, language-specific models trained on ten million synthetic chat examples. A post-translation validation step verifies non-translatable entities such as numbers and emojis before falling back to external services if mutations occur. Caching layers were also introduced across translation paths to curb redundant on-the-fly execution.
Profile-guided optimisation (PGO) on Grab services
Profile-guided optimisation (PGO) improves Go application performance by feeding production CPU profile data into subsequent compiler builds. Teams evaluated PGO across multiple Grab services, including TalariaDB and an orchestrator within the Catwalk service, using Go 1.20 and Go 1.21 builds. TalariaDB saw substantial resource savings after compiling Docker images with a 360-second pprof profile, resulting in at least a 10% drop in CPU usage, a 30% drop in memory consumption, and a 38% reduction in persistent volume usage for event ingestion queues. Conversely, testing on Catwalk showed only a 5% performance gain after adjusting the profiling duration from 59 seconds to 6 minutes, an outcome deemed insufficient to justify pipeline modifications. Broader adoption across monorepo services remains constrained until the central build process supports PGO arguments and automated profile retrieval.
Router 2.0 Common Runtime Router: The Road to Beta
Heroku’s Router 2.0 is a beta replacement for the Common Runtime’s Erlang-based “Hermes” router, which is planned to be replaced in 2024. The project addresses the difficulty of safely delivering upgrades from a legacy custom HTTP server and aims to share routing code with the Go-based Private Spaces router; common functionality has been extracted into an internal HTTP library. Architecturally, Router 2.0 is an L7 router running in memory behind the existing L4 proxy, alongside Hermes, allowing app-level traffic to move from the old path without a big-bang cutover. The team added continuous load testing that exercises scaling policies and blocks deployments missing internal service-level objectives, and introduced Network Error Logging to measure browser-observed failures. Feature parity is still incomplete, with HTTP/2 planned next; apps can opt into the beta using a Heroku Labs command.
Streamlining Grab's Segmentation Platform with faster creation and lower latency
Grab's Segmentation Platform previously stored user-to-segment mappings across individual rows in ScyllaDB, causing write bottlenecks during segment creation and read latencies too high for downstream consumers. To resolve these performance limitations, the team transitioned to storing segments as Roaring Bitmaps saved as single blobs in object storage. This compression strategy splits 32-bit integer user IDs into chunks across array, bitmap, and run containers based on data density, reducing a one-million-member segment to under one megabyte. An accompanying client SDK manages segment retrieval, decoding, update notifications, and least-recently-used in-memory caching. Consequently, consumers such as Grab's communications platform achieved peak throughput of 15,000 queries per second with sub-millisecond p99 read latencies.
Tokenized Tokens
Fly.io describes a security problem in its Rails-based platform: as the application accumulated secrets, Rails became the part of the attack surface least trusted, yet still handled many operations involving bearer tokens. It compares model encryption, Vault or KMS, and narrowly scoped secret-handling services, noting that each leaves gaps or creates operational overhead when secrets must be read for arbitrary third-party APIs. The proposed Tokenizer is a stateless HTTP proxy holding a Curve25519 private key; encrypted secrets travel in Proxy-Tokenizer headers, and the proxy injects plaintext or computes HMACs while host allowlists constrain misuse. A companion SSOkenizer performs OAuth2 flows outside Rails, tokenizes resulting bearer tokens, and returns credentials usable only through Tokenizer. The tools are open source, and the design provides a path for third-party integrations and SSO without adding raw secrets to Rails.
Go module proxy at Grab
Grab's 69.3 GiB multi-module Go monorepo caused commands like go get to take over 18 minutes as Git repeatedly traversed commit history, downloaded large worktrees, and overloaded their GitLab VCS infrastructure. To bypass direct VCS queries without losing automatic updates for external repositories, the team deployed the Athens Go module proxy configured in fallback network mode. They used the GOVCS environment variable to disable Git access specifically for the monorepo path, forcing Athens to fall back to its internal object storage when resolving monorepo modules. A dedicated CI pipeline pre-populates and refreshes the Athens cache whenever new monorepo modules are released. This setup reduced monorepo go get execution times to approximately 12 seconds and allowed a 70% scale-down of the Athens proxy cluster.
Performance bottlenecks of Go application on Kubernetes with non-integer (floating) CPU allocation
Grab's real-time stream processing platform encountered severe consumer lag and CPU throttling when running Go-based Kafka consumer pipelines on Kubernetes. The issue originated when the Vertical Pod Autoscaler (VPA) scaled pod CPU allocations down to floating-point values such as 1.94 cores. Because AUTO-GOMAXPROCS rounds non-integer CPU limits down to integers, Go runtime thread allocation dropped to 1 core, significantly throttling pipeline throughput despite available pod capacity. Setting a minimum floor of 2 cores instantly restored CPU utilization to 95% and cleared the message backlog. To prevent similar throttling, the team utilized integer CPU scaling recommendations available in VPA v0.13 on Kubernetes 1.25 and above.
2.3x faster using the Go plugin to replace Lua virtual machine
Talaria, an open-source distributed time-series database developed at Grab, previously allowed users to run custom data transformation scripts during ingestion using a Lua virtual machine. Launching and executing Lua scripts caused significant performance overhead when processing large volumes of events. To resolve this bottleneck, the team replaced the Lua VM with Go plugins compiled as Linux shared libraries (.so files). Benchmarks revealed that calling Go plugins achieves performance on par with native Go functions, executing roughly 2.3 times faster and consuming 2.3 times less memory than cached Lua VMs. Both execution methods conform to a unified Handler interface to load and run custom transformations.
Playing Traffic Cop with Fly-Replay
Fly.io’s fly-replay response header lets the Fly Proxy replay an HTTP request on another app, region, or VM within the same organization. The post explains how this can send writes near a leader database, preserve request locality after uploads, wake Machines stopped on exit, or route customer traffic across apps. It then builds a small Go reverse-proxy application that looks up the request hostname in SQLite and returns either app=our-default-app or the registered customer’s app in the header. Local request handling and lookup took about 4 ms, while the real-world proxy-and-replay path added about 100 ms; private customer apps remained reachable because Fly services can be defined without public IP addresses. The post also mentions LiteFS and internal DNS as possible discovery mechanisms.
Using Server Sent Events to Simplify Real-time Streaming at Scale
Shopify’s 2022 BFCM Live Map needed to deliver real-time sales and product data to thousands of concurrent users while processing data from millions of merchants. Its 2021 architecture used WebSocket delivery to a presentation layer, which stored messages in a mailbox that clients polled at least every 10 seconds, creating latency and bottlenecks. Shopify replaced that path with a streamlined Flink-based pipeline and a Golang SSE server that subscribes to Kafka topics and pushes JSON data to registered, authenticated clients as soon as it is available. The SSE server ran horizontally behind Nginx load balancers, with a configurable Java client used to load-test concurrent connections. The 2022 system maintained 100 percent uptime, delivered SSE data within milliseconds of availability, and visualized data within 21 seconds of creation, including pipeline processing.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.