Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,739 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Heroku Proxying becomes Free Software
Heroku’s HTTP routing stack has three layers: state synchronization, request routing, and HTTP proxying. The platform team open-sourced the proxying layer as Vegur, an Erlang library that validates, normalizes, and forwards requests between clients and dynos. Unlike a conventional nginx reverse proxy, Vegur’s hosts and configuration are fully programmable, allowing state from hardcoded values, files, databases, or direct updates, while routing algorithms and debugging headers can also remain dynamic. Its delegated capabilities include domain lookup, backend selection, interface choice, resource checkout and checkin, customizable failure pages, per-application or per-backend controls for tunneling 100-continue requests, and upstream or downstream header insertion. Vegur had run in production on Heroku for over a year and powered the Heroku Private Spaces router.
We Patched Ruby On Rails for a 10% Performance Bump
Rails response time improved by more than 10% in a patch that also reduced allocated objects by 29%, motivated by memory and performance problems observed in production Heroku applications. The work began with customer reports of R14 out-of-memory errors, swap usage, and suspected memory leaks, then used a customer app and reusable benchmarks to reproduce issues. The derailed benchmarks perf:objects task and memory_profiler exposed allocation hot spots, including arrays joined into strings and duplicated hashes; refactoring reduced unnecessary objects, while routing changes produced a 31% speed increase for constrained URL generation. The resulting savings required no application changes and were planned for the next Rails version, with Rails 5 pre-release testing mentioned.
Preparing Our Response to the OpenSSL Security Flaw
Heroku prepared for an announced high-severity OpenSSL vulnerability whose details were embargoed until patch release. The team began by mapping exposure and system criticality, considering SSL clients, possible man-in-the-middle attacks, encrypted database connections, and worst-case remote code execution, then created stack-image update actions and priorities. Engineers were assigned responsibilities under the existing incident-response process, with communication rules, escalation criteria, paging, and availability planned in advance. When the OpenSSL update and vulnerability details arrived on July 9 at about 8am Central, triage showed it was neither an RCE nor server-side issue, and Ubuntu reported that neither LTS version was affected; because Heroku production used LTS, no remediation or all-hands response was needed. The preparation still exercised exposure review and response coordination for future incidents.
Improved production stability with circuit breakers
Heroku’s API consumes more service endpoints than it serves, making its availability dependent on public, internal, and third-party services that can produce tail latency, 500 errors, broken sockets, or bad deploys. To contain these failures, the team wrapped external calls with Redis-backed circuit breakers that count errors over a time window, trip at an error-rate threshold, and support fallback responses or user-friendly exceptions. CB2 provides this implementation, including individually identified breakers, while percentage-based thresholds avoid the scaling problem of fixed error counts. Deployment starts with logging-only breakers so parameters and unexpected internal-API behavior can be reviewed before traffic is affected, followed by monitoring state changes with safeguards against duplicate notifications caused by race conditions. The post also recommends explicit, low network-level timeouts, arguing that slow services can damage stability more than fast failures.
Incremental Garbage Collection in Ruby 2.2
Ruby 2.2 introduces RincGC, an incremental garbage collector intended to reduce the long pause times that Ruby 2.1’s generational collector could produce during major GC. The approach splits marking into fine-grained steps interleaved with Ruby execution, while lazy sweeping remains incremental; because CRuby lacks sufficient write barriers, unprotected objects are rescanned before white objects are reclaimed. The article explains the white, grey, and black object model, the write-barrier safety problem, and uses GC::Tracer events with a Rails benchmark simulating 30,000 requests to measure pauses. In the reported measurement, generational GC produced seven large pauses with a maximum of about 15 ms, while incremental GC reduced the maximum to around 2 ms, without reducing total GC time or improving throughput.
Debugging Super Methods with Ruby 2.2
Ruby exposes method metadata and caller-stack information, but before Ruby 2.2 it was difficult to identify the method ultimately reached by super, especially in metaprogrammed code. The example defines Dog#bark and SchneemsDog#bark, then extends an instance with DoubleBark, whose bark method calls super twice; inspecting the class hierarchy incorrectly reports Dog#bark because the module is not an ancestor of the instance's class. Ruby 2.2 adds Method#super_method, a feature proposed in request #9781 and accepted into the Ruby trunk, which returns the method Ruby will invoke for super, including the correct SchneemsDog#bark in this case. Calling source_location on that returned Method provides the location of the actual implementation and avoids the custom metaprogramming previously needed to trace it.
Short Time Outs and Asynchronous Requests on Heroku
Heroku’s short request timeout and H12 errors reflect a scaling problem: standard web servers process queued requests serially, so one long operation can block others. The post separates slow server-side actions, such as video processing or PDF generation, from file uploads, and recommends background processes for the former and direct browser uploads to Amazon S3 or Google Cloud Storage for the latter. When a client must wait for completion, it describes polling or a WebSocket, noting that WebSocket connections close after 55 seconds of inactivity and ordinary pages still need to respond within 30 seconds. Avoiding lengthy web-process work improves performance and scalability; rack-timeout for Ruby or expressjs’ timeout for Node can terminate requests after a few seconds, with a recommended limit no higher than 10 seconds.
Benchmarking Rack Middleware and Improved Speed
ActionDispatch::Static, enabled by config.serve_static_assets = true, checks incoming Rails requests to decide whether to serve a disk file or pass the request onward. To measure the middleware in isolation, the author loads it directly, wraps it around a no-op Proc, and uses Rack::MockRequest with benchmark-ips to compare requests against the no-op baseline. The original middleware achieved 15,891.2 iterations per second, versus 28,660.9 for the no-op, while the modified version reached 24,490.9 iterations per second. That change made non-asset middleware requests roughly 54% faster, bringing the gap to the no-op case to about 10%, although asset requests became slightly slower and the difference was not measurable. Across the full Rails application stack, the reported overall speedup was approximately 2.6%.
Django and Node together on Heroku
Django-based Heroku Connect needed realtime dashboard support through socket.io and Node.js, but Heroku apps could route traffic through only one dyno type. The team chose to run both stacks in one dyno using a runit buildpack and its sub-Procfile format: Node binds to PORT, while node-http-proxy forwards normal web requests to Gunicorn-hosted Django on 127.0.0.1:$DJANGO_PORT. This arrangement increases contention because processes share memory, CPU, and a filesystem, and it makes dyno logs less directly attributable; the team also flattened Procfiles into Procfile.local for Foreman-based local development. After running the architecture in production for a few months without issue, the team reported minimal proxy overhead and retained single-git-push deployment while establishing a path for websocket communication.
Instrumentation by Composition
Instruments is a Go library for collecting application metrics over discrete time intervals, including request rates, latencies, data-structure sizes, and event durations. It supplies Counter, Rate, Reservoir, Gauge, Derive, and Timer instruments; values represent only the latest time-window and are read through a single-reader Snapshot operation. Composition is central: instruments implement either Discrete, returning one int64, or Sample, returning a slice, so custom instruments can wrap existing ones; the example embeds a Reservoir in an http.Handler to record upload request sizes. Derived instruments build on this model, with Derive using Rate and Timer using Reservoir, while Quantile, Mean, and reporting utilities process snapshots. An optional registry and logfmt reporter support periodic output, but applications can also send selected values to systems such as Graphite or use them for control signals.
Getting started with the Force.com APIs for the Hackathon
With a Salesforce hackathon approaching, the guide introduces the Force.com APIs as a way to build applications using Salesforce1’s authentication, data storage, and business rule services. It compares native Salesforce1 applications, connected web applications using SOAP or REST, and three mobile patterns, including hybrid extensions, native iOS or Android clients, and a “force.com proxy” model. For hands-on setup, it recommends a free Developer Edition account and Salesforce Workbench, then explains OAuth2 web flow and password-plus-token login before showing Ruby with Restforce and Node.js with Nforce. The examples configure a Connected App, keep credentials in environment variables, query records with SOQL, and create Account records, providing a practical starting path for API integration.
The Heroku Mobile App Template
The Heroku Mobile Template is an open-sourced full-stack starting point for hybrid mobile apps that can be deployed to Heroku. It addresses the early choice among native iOS or Android, mobile web, hybrid clients, and backend technologies. The sample, “Quiz Live,” combines an AngularJS client with a Node.js and Express server, SocketIO real-time communication, BookshelfJS, and Postgres persistence for authentication, data storage, and notifications. Apache Cordova can package the HTML and JavaScript assets as a native app, while Ionic supplies cross-platform mobile UI components. Developers can fork the GitHub repository, deploy with the Heroku Button, and run it locally with Node.js and Postgres; the template also includes a Force.com example that creates Lead records for registering Quiz users.
Stuff Goes Bad
The Heroku Routing team describes a manual created to preserve hard-earned operational knowledge from developing and maintaining Erlang systems at scale. That knowledge had typically been passed informally through incident simulations, code reviews, and similar practices, leaving no persistent record for new team members. The manual presents itself as a guide to being an “Erlang medic” during failures, combining tips, tricks, code snippets, and practices for understanding failure sources and debugging running production systems. It is available as a free PDF under a Creative Commons License at erlang-in-anger.com, and was released in time for the Chicago Erlang conference, where Routing team members and Erlang community participants could discuss real-world applications.
How to Secure the Celery Task Queue with Fernet
Celery jobs sent through a Redis broker can expose their contents because Redis does not currently support SSL and Heroku add-ons communicate over the public web. The proposed solution encrypts jobs before they travel between dynos and Redis by wrapping Kombu’s default serializers with Fernet symmetric encryption from Cryptography. The open-source kombu-fernet-serializers library supplies these encrypted serializers through Setuptools entry points; setup requires adding the package, storing a Fernet key in KOMBU_FERNET_KEY, and configuring Celery to use fernet_json with MIMETYPE in CELERY_ACCEPT_CONTENT. The example creates a Celery app with REDIS_URL as its broker and uses the serializer settings, retaining Redis’s straightforward configuration while addressing sensitive information in transit.
Hutils – Explore your structured log data
Heroku’s internal components use logfmt to capture production activity, enabling retrospective request analysis, flexible trace queries, historical metrics, and incident investigation. The format can also become an unreadable 51-page wall of text, motivating hutils, a small suite of tools installable through Rubygems for exploring structured log data. Its ltap command extracts message streams from Splunk or Papertrail using a search query, while lviz parses messages into a tree that groups common key/value nodes and reduces duplicate data; the tools are designed to chain together. Additional commands include lcut for selecting fields and lfmt for colorized, whitespace-enhanced formatting, with both also supporting heroku logs --tail.
Retrospectives
Retrospectives are presented as a way for software engineering teams to review operational incidents, investigate root causes, create remediation tasks, and improve teamwork. Beyond technical review, the process can focus on emotional and human aspects of projects, including interactions and ways to work better together. The suggested timing includes post-incident or post-project sessions, mid-project course corrections, and weekly meetings; a 90-minute Friday session one week after a project ends is described as effective, while next-day and three-week delays are discouraged. An active, neutral facilitator should prepare an agenda and guest list, state a clear purpose, involve everyone, and use one or two activities such as a timeline, five whys, or Start Doing / Stop Doing / Keep Doing brainstorming.
Heroku Connect: Event-driven Data Sync For Force.com
Heroku Connect synchronizes data in both directions between Salesforce (Force.com) and Heroku Postgres, but its original polling design trades simplicity and reliability for synchronization latency, wasted API calls, and unnecessary database queries. The post presents an event-driven alternative using the Force.com Streaming API and PostgreSQL’s LISTEN/NOTIFY system. For Salesforce, Heroku Connect automatically creates a PushTopic per synchronized table, subscribes with Node.js and NForce, and dispatches Redis tasks when notifications arrive. Those events act as change signals rather than complete data transfers: workers query the Force.com SOAP API because streaming has limitations and subscribers may miss events. On Postgres, PL/pgSQL trigger functions call pg_notify after DML, while node-postgres listens on named channels; the system supports added UPDATE and DELETE triggers, though triggers are still required.
Using force.com from your Heroku apps
Force.com and Heroku, both part of the Salesforce1 platform, can be connected from Heroku applications through data synchronization or direct API access. Heroku Connect provides point-and-click, two-way synchronization between Force.com and the Postgres database attached to an app, allowing reads and writes in Postgres while changes are copied asynchronously; that latency makes it unsuitable when an immediate Force.com response is required. For synchronous access, open-source client libraries cover Force.com’s SOAP, REST, Streaming, Bulk, and Metadata APIs across languages including Java, Ruby, Node.js, PHP, Python, Objective-C, and Go. The article maps libraries and packages to supported APIs, including restforce and salesforce_bulk for Ruby, nforce for Node.js, simple-salesforce for Python, mobile SDKs, and ForceCLI for scripting SOAP and Metadata operations.
Introducing the Heroku HTTP API Toolkit
Heroku is open sourcing the toolchain it uses to design, document, and consume HTTP APIs, covering a design guide, JSON Schema tooling, and generated Ruby and Go clients. The guide addresses versioning, resource structure, naming, serialization, errors, request IDs, pagination, and caching, while prmd can bootstrap schemas, verify completeness, and generate documentation. JSON Schemas also drive client generation through Heroics for Ruby and Schematic for Go, with examples showing access to the Heroku Platform API and generation from its schema. Heroku says the toolkit supports both user-facing and internal services, where shared schemas and generated documentation provide a common language for proposing APIs or changes and help align design practices.
Top topics
- Architecture834
- Performance817
- LLMs673
- Machine Learning622
- Open Source568
- AI Agents564
- AI483
- Developer Experience448
Weekly digest
The best engineering reads, once a week.