Loading…
Topic
Go
89 posts about Go. Every summary links to the original.
- 89 posts
- 15 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Leveraging Go Worker Pools to Scale Server-side Data Sharing
Shopify’s Server Pixels service validates, processes, augments, and produces more than one billion customer events daily, but rising traffic caused Kafka input-topic consumption lag when downstream processing slowed. Its original design could spawn an unlimited number of goroutines while processing or producing event batches, consuming CPU and memory and threatening the five-minute event-processing SLO. The service was redesigned around fixed-size Go worker pools: workers receive Job values through channels, process them via reusable work functions, and use wait groups and context cancellation for orderly shutdown. Load testing showed 15 workers in both processor and producer raised throughput to 12.9 thousand events per second per pod, while 50 workers reached 19.3 thousand; subsequent improvements reached 21 thousand, a 170% increase over 7.75 thousand, and the service handled a maximum of 46 thousand events per second during BFCM 2021.
Using Terraform to Manage Infrastructure
Shopify’s Conversations team used Twilio TaskRouter to route voice and chat tasks to agents according to routing rules and skills. As routing needs grew, configuration through Twilio’s GUI became difficult to understand and lacked clear change history, tests, and rollback support. Terraform provided an infrastructure-as-code workflow, backed by a reliable API, a Go client library, and a provider implementing CRUD operations; the team built a provider because Twilio did not have one at the project’s start. Resource definitions included schemas, dependencies, and an importer, while Terraform’s plan and apply commands previewed and executed changes. Integrated with Atlantis and GitHub pull requests, the workflow added plans, reviews, revision history, and easier rollbacks, while enabling business and support teams to submit infrastructure changes themselves.
Adding the V8 CPU Profiler to v8go
v8go adds support for V8’s CPU Profiler, allowing Go users to measure JavaScript execution in a V8 isolate and inspect sampled top-down call trees with function and engine details. The API work aimed to stay idiomatic for Go, remain close to V8’s API, support future profiler features, and minimize Cgo overhead. The first design loaded profile data lazily, making separate Cgo calls for profile and node properties; the alternative built the complete graph in C++ before returning it to Go. Benchmarks showed lazy loading was faster for a small tree, averaging about 20 microseconds versus 25 for eager loading, while a larger tree reversed the result at roughly 90 versus 60 microseconds. The comparison demonstrates that Cgo costs depend on workload size and that performance assumptions require benchmarks using representative inputs.
Our User-Mode WireGuard Year
Fly.io’s flyctl needed to let users SSH into Firecracker VMs and reach private 6PN networks without requiring root access or system-level WireGuard configuration. The solution runs wireguard-go, a userland TCP/IP stack derived from gVisor’s Go implementation, and golang.org/x/crypto/ssh inside flyctl, later coordinating sessions through a cross-platform background agent. The design initially exposed serious operational and usability problems: concurrent sessions displaced one another, agent behavior was fragile, and remote peer creation was slow, while deploys and remote builds expanded the mechanism’s role. The team improved concurrency, Windows support, DNS resolution, configuration handling, and peer-creation latency, and built an unmerged WebSocket path for networks blocking WireGuard’s UDP transport. The author concludes that the complexity was costly early in 2021 but enabled capabilities such as private DNS lookup and ping.
Processing ETL tasks with Ratchet
Grab's Lending team relies on multiple microservices to finance various user segments, requiring automated pipelines for operational tasks like loan rescheduling and merchant whitelisting. To execute these workflows, the team uses Ratchet, a Go library that organizes data pipelines into concurrent stages connected by Go channels and JSON payloads. In the Azkaban service, each ETL task is structured as a Job Processor that configures custom Data Processors for extraction, transformation, and loading. For merchant whitelisting, a custom processor pulls uploaded CSV files from Amazon S3, validates the rows to prevent manual errors, calls microservice APIs, and passes the output to an empty load stage. This pipeline architecture allows the team to isolate failures quickly, eliminate manual data validation, and automate routine operational tasks.
Debugging High Latency Due to Context Leaks
Market-Store, Grab's feature store for real-time machine learning features, experienced latency spikes from under 200 milliseconds to 2 seconds as traffic grew. Metrics and logs showed no direct correlation to API issues, but heap profiling with PPROF revealed continuously increasing memory held by child contexts. Further analysis tracked the leak to an update in Grab's open-source Async Library, which switched background contexts to uncancelled task contexts for worker runners. Because parent contexts maintained references to these uncancelled child contexts, the garbage collector could not reclaim their memory. This progressive memory exhaustion directly degraded API latency.
Updating Illustrations at Scale
Shopify’s Polaris team needed to update an illustration style across a product whose assets had become fragmented across more than 6,000 repositories. Illustrators and developers could not reliably identify an image’s repository, code references, reuse, or file type, while GitHub API limits and search behavior prevented dependable large-scale discovery. An initial Node.js monorepo cloned repositories, used globby to find PNG, SVG, and WebP files, searched source lines, and served JSON results through a Preact interface, but rebuilding 24 repositories took more than 35 minutes and produced inaccurate data. The team replaced it with @shopify/get-repo-images, a Go executable that parallelizes file operations, bundles crawling and site generation, and exposes a searchable Next.js interface through NPM. Search and build time fell to eight minutes, with repository cloning as the remaining bottleneck, and the tool was open-sourced.
Hooking Up Fly Metrics
Fly apps include built-in Prometheus instrumentation, exposing automatically collected metrics through a public Prometheus endpoint and allowing applications to export custom metrics. The walkthrough connects an organization’s Fly Prometheus URL to Grafana with a Fly token, then explains dashboard variables and PromQL queries using counters, gauges, histograms, labels, irate(), rate(), increase(), sum(), and regional grouping. Examples cover network traffic, TCP connections, HTTP response rates by region, and response-time heatmaps built from histogram buckets ranging from 5ms to 120s. Application instrumentation is shown for Go, Python Flask, and Node.js, with scraping configured in fly.toml through a metrics port and path. The post concludes that metrics are currently free, with some tier free forever, although a future pricing plan is expected for ambitious usage.
SSH and User-mode IP WireGuard
Fly.io’s SSH feature addresses the need to access applications running in Firecracker micro-VMs without requiring users to build containers with an SSH service or install WireGuard. The first part uses 6PN, a private IPv6-based network routed across Fly’s WireGuard mesh, and Hallpass, a small Go SSH server bound to each instance’s 6PN address. Authentication relies on an organization’s root SSH certificate published in private DNS, while the API signs short-lived user certificates. To support users without WireGuard installed, flyctl combines wireguard-go’s userland protocol with gVisor’s user-mode TCP/IP stack, allowing ordinary Go networking code and DNS resolution to operate over the private network. The resulting certificate-based SSH path is built into flyctl and reaches internal DNS names such as dogmatic-potato-342.internal, although the user-mode implementation is substantially slower than kernel TCP/IP.
One Small Step Closer to Containerising Service Binaries
Engineering teams at Grab initiated a transition to containerized microservices to standardize environments, enhance security, and decouple services from internal runtime tooling. During this migration, developers noticed that statically-linked Go service binaries were reaching bloated sizes over 100 MB. By analyzing the binaries using the open-source tool go-binsize-viz alongside the Go nm toolchain, the team visualized compiled symbols as interactive treemaps. This analysis revealed that 11 MB of unused message format symbols were being pulled in because a generic interface shared a directory with auto-generated streaming code. Restructuring the packages to isolate interfaces from generated code successfully decreased the binary size down to 78 MB.
Customer Support Workforce Routing
Grab replaced its third-party customer support routing software with an in-house workforce routing system for Livechat to gain better priority controls, bespoke configurations, and deeper analytics. The platform separates requests into distinct priority and business queues, using parallel workers that spend varied time slices dequeuing higher-priority issues like safety concerns. To prevent request starvation, workers operate out of sync across queue priority levels while dynamic queue limits cap incoming volume based on agent availability and performance. The system routes requests through an intermediate Agent Group layer, calculating eligibility scores from proficiency and concurrency metrics while managing per-agent locks to prevent over-allocation.
Managing Google Cloud Platform Project-Wide SSH Keys
Google Cloud project-wide SSH keys can remain in project metadata indefinitely after a user connects to a VM, potentially allowing matching private-key holders to access Linux VMs with root access unless project-wide keys are blocked. Shopify developed SSH-Pruner because OS Login did not work with Google Kubernetes Engine, while deleting all keys also removed Google’s keys, disrupted scripts, and created slow metadata updates. The new Go application uses Google Cloud APIs to find organization projects with Compute Engine enabled, reads the “ssh-keys” metadata, parses entries, retains unexpired keys, writes pruned common instance metadata, and records readable logs; Google-managed “sshKeys” entries are skipped. SSH-Pruner was already running as a cron job on some projects, with a slow rollout planned to limit workflow disruption.
Trident - Real-time Event Processing at Scale
Trident serves as Grab's internal real-time event-processing and workflow automation engine, driving user campaigns, rewards, and notifications across multiple business lines. To handle peak loads exceeding 2,000 events per second without duplicate execution, the system consumes decoupled Kafka streams and enforces exactly-once semantics using Redis and MySQL deduplication checks. Processing efficiency relies on server autoscaling aligned with Kafka partition counts, combined with dynamic goroutine allocation per consumer. To minimize rule evaluation overhead, Trident indexes active campaigns into an in-memory hash map by event type, cutting processing time by at least 90%. Furthermore, condition evaluation is optimized through lazy loading and a weighted sorting algorithm that checks low-cost in-memory data prior to executing expensive database queries or external service calls.
Pummelling the Platform–Performance Testing Shopify
Shopify prepares for high-traffic sales events such as BFCM by repeatedly simulating customer traffic on representative test shops, aiming to expose bottlenecks before merchants experience them. Its Platform Conditioning team supports service-owning teams with approachable performance-testing tools and processes, distinguishing load tests for known traffic levels from stress tests that push services toward failure. A Go load generator runs Lua flows through lightweight VMs, models realistic traffic shapes and costly endpoints, and can issue tens of millions of requests per minute or more. Tests are initiated through Slack using the internal ChatOps tool Spy, which returns links to dashboards, logs, and results. Cronograma provides formal experiments with hypotheses, observations, conclusions, alert detection, Slack timelines, and comparable metrics such as response times, 5xx errors, and requests per minute; the tools have led to performance improvements and give Shopify confidence in future sales events.
Democratising Fare Storage at Scale Using Event Sourcing
Grab's legacy system stored booking and fare details in a single relational table, creating a bloated booking entity that tracked only the latest fare state and hindered rapid feature iteration. To resolve scalability, stability, and debugging challenges across millions of daily bookings, the team developed Fare Storage using the Event Sourcing pattern. The new architecture persists all fare modification events chronologically in DynamoDB, backed by a cache for eventually consistent reads and message streaming for downstream processing. The platform employs optimistic locking with versioning to manage concurrent updates, enforces idempotency through client-generated transaction UUIDs, and delegates metadata serialization to an SDK to prevent storage API changes.
Always Be Connecting (with HTTPS)
The post explains how to ensure users reach an HTTPS version of an application running behind Fly’s edge proxies, particularly when a custom domain and certificate are attached. Fly’s default fly.dev behavior redirects HTTP automatically, but custom certificates do not activate that upgrade because connection behavior remains under the application owner’s control. The recommended method checks the X-Forwarded-Proto header, which records whether the original client connection used http or https, and returns a 301 redirect to the same URL with an https scheme when needed. Examples implement the check in Go middleware, Express middleware for Node, and Nginx configuration; the post also notes that non-Fly deployments can inspect the request and URL scheme instead, while Fly was considering future automation.
Go Modules- A Guide for monorepos (Part 2)
Managing dependencies in a multi-module monorepo created developer friction at Grab due to unexpected changes from previous vendoring attempts and accidental imports. Because Go modules were not yet enabled directly for builds, the team implemented a continuous integration check that executes go mod vendor and rejects merge requests if any diffs exist in go.mod or the vendor directory. Adopting this CI check required configuring SSH deploy keys for private repositories, adding retry logic for network-related false positives, and standardizing on a single Go version to prevent checksum discrepancies. To streamline ongoing maintenance across hundreds of dependencies, the team developed an automated tool named AutoVend Bot. The bot runs go list -m -u all to detect updates and opens a scheduled batch of merge requests each day for human review.
Go Modules- A Guide for monorepos (Part 1)
Grab transitioned its large Go monorepo dependency management from Glide to Go modules while retaining an existing vendor directory structure. The team generated root go.mod configurations from glide.yaml and used go mod vendor without directly enabling module-mode builds. Incompatible nested sub-vendor paths were excluded by placing empty go.mod files, relying on the rule that modules cannot contain other modules. Post-migration maintenance revealed challenges with dependency inheritance and implicit go.mod updates during builds, which engineers investigated using go mod graph and digraph to trace dependency paths.
Continuous Deployment with GitLab
The guide shows how to set up continuous deployment from GitLab CI/CD to a Fly application, using the small Go hellofly example. GitLab supplies the repository contents to a Runner defined by the golang:1.13 image, while a before_script installs curl and downloads flyctl through Fly’s installation script. A deploy job then runs flyctl deploy, which builds and deploys the application from a committed fly.toml file. Authentication is configured by obtaining a token with flyctl auth token and storing it as a protected, masked FLY_API_TOKEN CI/CD variable in the repository settings. After .gitlab-ci.yml is committed and pushed, GitLab starts the deployment job, while the guide notes that more advanced pipeline structures and triggers remain available.
Simpler Fly deployments for NodeJS, Rails, Go, and Java
flyctl now supports building and deploying NodeJS, Rails, Go, and Java applications to Fly with two commands. The feature uses Cloud Native Buildpacks from buildpacks.io, turning an application into a Docker image during deployment without requiring users to understand Buildpacks. Cloud Native Buildpacks are described as an evolution of Heroku and Cloud Foundry systems, but are not compatible with their older buildpacks. After installing the latest flyctl, users can initialize an app with flyctl init --builder cloudfoundry/cnb:bionic and run flyctl deploy, then open it with flyctl open. Buildpack support complements Dockerfile-based image builds, while listed builders provide different language coverage and image sizes; the post recommends heroku/buildpacks:18 for its wider coverage.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.