---
title: "Latest reads"
description: "The engineering internet, summarised so you can actually read it."
---

# Latest reads
> The engineering internet, summarised so you can actually read it.

## Articles

### [Bug Bounty Year in Review 2019](https://yomu.fyi/post/bug-bounty-year-in-review-2019.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Feb 6, 2020
- Type: Announcement

Shopify’s 2019 bug bounty review describes experiments and process improvements intended to increase program speed, while reporting changes in response, resolution, disclosure, and bounty metrics. Shopify-Experiments, a private program launched in mid-2019 for high-signal, high-impact hackers, tested expanded scope, full payment after triage, disclosure requirements, self-closing false positives, and collaboration with third-party developers. Using registration data, HackerOne API exports, and a Slack chatbot, the team improved program analytics and automated report-state changes, assignments, comments, bounty suggestions, and common invalid-report closures. Average first response fell to 16 hours, triage to 2 days and 13 hours, bounty payment to 7 days and 1 hour, and post-triage resolution to 20 days and 3 hours; disclosures rose to 74 bugs. For 2020, Shopify announced full bounty payment within seven days of triage, a $50,000 maximum bounty, higher payouts for several vulnerability classes, and added visibility for duplicate reports.


### [Turboku - The Art Of Faster Heroku Apps](https://yomu.fyi/post/turboku-the-art-of-faster-heroku-apps.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Feb 5, 2020
- Type: Benchmark

Turboku is a migration path that automatically turns Heroku web dynos into Fly applications, addressing how existing Heroku apps can move to Fly’s globally distributed edge network. Fly routes requests to the closest datacenter through one global IP address, or two with IPv6, while TLS terminates closer to users and modern ECDSA certificates are automatic for Fly domains. A benchmark around the world used an existing application to compare Heroku and Fly connection stages, full requests, and cached requests. Fly reduced DNS, TCP, TLS, network, and full-request times, with the largest reported improvement appearing when caching was used: 180.16 ms versus Heroku’s 817.75 ms, a 77.91% reduction; uncached full requests fell 35.39%, while TTFB was 1.24% higher.


### [Chrome’s Changes Could Break Your App: Prepare for SameSite Cookie Updates](https://yomu.fyi/post/chrome-s-changes-could-break-your-app-prepare-for-samesite-cookie-upda.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Lenora Porter
- Published: Feb 4, 2020
- Type: Tutorial

Chrome's SameSite changes alter when third-party cookies are sent in cross-site requests, affecting sites that rely on embedded services, analytics, advertising, or other cross-domain functionality. Starting February 4, 2020, Chrome was to reject such cookies unless they were secured and assigned an appropriate SameSite value, as part of a secure-by-default model intended to reduce CSRF attacks, other vulnerabilities, and privacy leaks. The post explains first- versus third-party cookies, contrasts Strict, Lax, and None, and gives scenarios for selecting each setting; SameSite=None must be paired with Secure and HTTPS. It also recommends enabling Chrome 80 flags to test for errors, then updating application cookies or coordinating with third-party providers. Server-side support is noted for PHP, NodeJS, Python, Flask, and Ruby on Rails, with direct JavaScript cookie handling also mentioned.


### [Command Lines, Flyctl and Fly](https://yomu.fyi/post/command-lines-flyctl-and-fly.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Jan 29, 2020
- Type: Announcement

Fly presents the Fly Global Application Platform as a command-line-first service and introduces flyctl as the way to sign up for free and manage an application's entire lifecycle. The post argues that command-line interfaces integrate directly with modern workflows, are simple and expressive, and keep developers in their terminal and current directory. To replace its Node-based command-line technology, Fly built flyctl in Go as self-contained binaries for macOS, Linux, and Windows, and adopted Cobra for familiar command/subcommand behavior and contextual help. Web interfaces remain intended for rich dashboards and metric displays, alongside interactive terminal use. The command-line approach also makes instructions self-contained, scriptable, and repeatable, which the post identifies as valuable for support and operations, while the open GitHub repository invites feedback.


### [React Native is the Future of Mobile at Shopify](https://yomu.fyi/post/react-native-is-the-future-of-mobile-at-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jan 29, 2020
- Type: Announcement

Shopify is moving forward with React Native for new mobile apps after years of native development, driven by mobile purchasing volumes and a desire to consolidate iOS and Android work. It describes React Native as a framework that uses JavaScript and React’s declarative model to translate virtual DOM structures into native platform views through native bindings. Experiments with Arrive, Point of Sale, and Compass reported twice the productivity on Arrive, an estimated 80% code sharing target, and observed sharing of 95% for Arrive and 99% for Compass, while testing lowered the considered Android CPU threshold from 2GHz to 1.5GHz. The policy does not require rewriting existing native apps, leaves low-level libraries native and native hiring in place, and calls for dedicated React Native tooling, foundations, and open-source partnerships.


### [Zoom Developer Relations — Meet the Team](https://yomu.fyi/post/zoom-developer-relations-meet-the-team.md)
- Company: [Zoom](https://yomu.fyi/company/zoom.md)
- Author: Abraham Queen
- Published: Jan 24, 2020
- Type: Explainer

Zoom's Developer Relations team supports third-party developers, manages the App Marketplace, and maintains platform tools including APIs and Web SDKs. The team comprises technical writers, developer advocates, developer educators, security engineers, and marketplace specialists. Specific responsibilities include authoring integration documentation, reviewing early API designs, testing webhooks, scheduling security evaluations for Marketplace applications, and building sample code. Developer interactions take place across multiple channels, including the Zoom Developer Forum, Zendesk support tickets, Zoom meetings, app submission review notes, and partnership calls. Specialists also guide creators through the formal app submission lifecycle, reviewing functional workflows, privacy statements, and regional ISV opportunities across global markets like APAC.


### [Behind the Curtain: App Publishing Requests](https://yomu.fyi/post/behind-the-curtain-app-publishing-requests.md)
- Company: [Zoom](https://yomu.fyi/company/zoom.md)
- Author: Abraham Queen
- Published: Jan 22, 2020
- Type: Explainer

Zoom App Marketplace reviews App Publishing Requests to enable broad distribution of applications across all Zoom accounts. During the initial review stage, marketplace specialists evaluate submission metadata, including descriptive text, media assets, functional documentation, and requested scopes to verify that applications do not access unnecessary user information. Reviewers then perform a functional assessment by executing the developer-provided test plan using supplied test accounts and credentials. When issues occur, reviewers deliver feedback through review notes, requiring developers to resolve the problems and resubmit their app to trigger a re-review cycle. Developers can validate production readiness prior to public submission through account-level local testing or by creating a Publishable URL sharing request for external testing.


### [Scaling Mobile Development by Treating Apps as Services](https://yomu.fyi/post/scaling-mobile-development-by-treating-apps-as-services.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jan 16, 2020
- Type: Problem & solution

Shopify describes the challenge of scaling mobile development without slowing feature delivery as teams and projects adopt divergent tools, making knowledge sharing, cross-project contribution, and incident response harder. It extends ServicesDB, previously used to track production services, to treat mobile apps as services with owners and explicit expectations encoded in configuration files. Automatic checks cover ownership, contact information, CI and Shipit Mobile deployment configuration, internal-tool versioning, and production error and crash monitoring. Failed checks generate GitHub issues with remediation instructions, while the contract helps the Mobile Tooling team identify compatible projects and helps developers understand why tooling may not work. The implementation currently has basic checks, with planned work including mobile-app service-level objectives, easier project creation, and shorter build times.


### [Terrier: Open-Source Tool to Identify and Analyze Container and Image Components](https://yomu.fyi/post/terrier-open-source-tool-to-identify-and-analyze-container-and-image-c.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: chris le roy
- Published: Jan 15, 2020
- Type: Tutorial

Terrier is an open-source tool for identifying and verifying files and other components in OCI images and running containers, introduced alongside a Blackhat Europe talk on exploiting builds in the cloud. It targets supply-chain and forensic use cases, such as checking whether an image contains a known file or malicious package. YAML configuration selects image or container mode, supplies SHA256 hashes and, for container checks, paths; Terrier scans image layers or a container’s merged folder while keeping processing in memory for performance. Examples locate /usr/local/go/bin/go, report that the Jeilyfish package is absent, and verify curl and Go files in a running container; CircleCI and GitHub Actions examples show how checks can gate image-build pipelines.


### [How We Prevented App Performance Degradation from Sudden Ride Demand Spikes](https://yomu.fyi/post/how-we-prevented-app-performance-degradation-from-sudden-ride-demand-s.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Corey Scott
- Published: Jan 8, 2020
- Type: Problem & solution

Grab experienced severe system strain when sudden localized spikes in ride demand, triggered by events like heavy rain or concert dismissals, coincided with driver shortages. These localized bursts overloaded the platform and degraded the experience for users outside the affected areas. To mitigate this, engineers created the Spampede filter, a circuit-breaker mechanism placed at the start of the booking pipeline. The filter converts pickup locations into Geohash Integer buckets and partitions time using Unix timestamps, tracking unfulfilled requests in Redis with atomic increments and time-to-live expirations. When unallocated requests exceed configured thresholds within a specific bucket, the system immediately short-circuits new incoming bookings to protect overall platform stability.


### [Plumbing At Scale](https://yomu.fyi/post/plumbing-at-scale.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Karan Kamath
- Published: Jan 6, 2020
- Type: Problem & solution

Grab's backend services process terabytes of data ingress per hour, generating recurring needs for stream transformations, joins, and time-windowed aggregations across diverse workloads. To support these asynchronous processing patterns across their Go ecosystem, the Coban team developed a managed, NoOps event sourcing and stream processing platform. The architecture packages stateless processing pipelines as Kubernetes deployments on AWS, polling Kafka event logs and using ScyllaDB as a shared metastore for stateful needs like deduplication and windowing. Stream processing pods combine ingestion triggers, a worker pool runtime, and user-provided domain logic plugins with customizable failure handling. This infrastructure scales to handle over 300 billion events weekly while maintaining workload isolation and elastic autoscaling.


### [Journey to a Faster Everyday Superapp Where Every Millisecond Counts](https://yomu.fyi/post/journey-to-a-faster-everyday-superapp-where-every-millisecond-counts.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Renu Yadav
- Published: Dec 26, 2019
- Type: Problem & solution

Grab undertook an initiative to reduce startup time and improve time to interactive (TTI) on its passenger mobile app. Because local benchmarks failed to simulate real device and network conditions, the team instrumented code in production across 8–9 million daily users to capture p50 and p95 metrics. Initial gains came from caching service tiles between sessions and removing a startup animation, saving four seconds. Architectural changes followed, including converting iOS dynamic frameworks to static linking and merging others, while Android initialisation was refactored with Kotlin coroutines. Replacing a heavy third-party analytics library with an internal experimentation platform yielded further startup reductions.


### [How to Implement a Secure Central Authentication Service in Six Steps](https://yomu.fyi/post/how-to-implement-a-secure-central-authentication-service-in-six-steps.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 26, 2019
- Type: Tutorial

Shopify built Identity to reduce the overhead of managing separate staff accounts across multiple stores, where each Shop isolated user data and lacked single sign-on. The service uses OpenID Connect (OIDC) to centralize authentication while retaining destination-specific authorization, and it models legacy one-store accounts separately from Identity accounts that can access multiple destinations. Migration begins by synchronizing profile and security data, routing authentication through Identity, and prompting eligible users to verify email ownership, combine accounts, and provide required 2FA credentials. Single legacy accounts can be auto-upgraded without changing their UUID, while multi-account merges use a MergeSession and operations implemented with ActiveOperation; WebAuthn also supports physical security keys as a second factor. Finally, Shopify changed store signup and staff invitations to prevent new legacy accounts, reporting that more than 75% of active accounts had been upgraded or combined at the time of writing.


### [Marionette - Enabling E2E User-scenario Simulation](https://yomu.fyi/post/marionette-enabling-e2e-user-scenario-simulation.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Anish Jha
- Published: Dec 23, 2019
- Type: Problem & solution

Conducting end-to-end testing across Grab's transport microservices became difficult due to service availability, environment construction, cross-service authentication, and complex data setups for real-world user accounts. To address these challenges without relying on physical mobile devices or emulators, Grab built Marionette, an internal simulation platform for passenger and driver interactions. The platform provisions required test data, coordinates booking lifecycles, and isolates test executions across distinct user groups using localized cohorts. Engineers can configure driver and passenger behaviors, execute workflows, and run load or integration tests through a dedicated user interface, a Go SDK, and RESTful APIs.


### [Slow Database Queries? Are Database Types to Blame?](https://yomu.fyi/post/slow-database-queries-are-database-types-to-blame.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Ben Fritsch
- Published: Dec 19, 2019
- Type: Problem & solution

Slow database queries can stem from schema choices as well as inefficient SQL. The example uses PostgreSQL with a json column holding large payloads, then selects a single url value for an app\_uuid and orders the results. EXPLAIN ANALYZE with BUFFERS showed 7,106 shared buffer hits, representing about 60MB of JSON data loaded and discarded, while execution reached 1,141.296 milliseconds in the reported run. The solution split the JSON into separate text fields for data and url, bringing query time down from about 1,200 milliseconds to 10 milliseconds. The broader conclusion is that schemas should be revisited as application usage becomes better understood, but changes are not necessary when the design isn't causing a problem.


### [The Curious Case of the Table-Locking UPDATE Query](https://yomu.fyi/post/the-curious-case-of-the-table-locking-update-query.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Richard Schneeman
- Published: Dec 19, 2019
- Type: Incident / postmortem

The post recounts a PostgreSQL performance incident in which metadata processing stopped as jobs accumulated and an UPDATE targeting a single UUID remained active for days. Diagnostics showed the query consuming more than 80% of database time, while manual execution did not reproduce the delay. A pg\_stat\_activity and pg\_locks query exposed many waiting statements and an AccessExclusiveLock entry whose locktype was tuple, leading the investigation toward PostgreSQL row locking and contention. The eventual cause was threaded code that issued concurrent updates to the same row for each job; removing that concurrency, aborting old backends with pg\_cancel\_backend, and deploying the change restored processing and began churning through the backlog. A correction notes that the reported lock was on a tuple rather than the table, and that PostgreSQL does not have lock promotion as suggested in the debugging discussion.


### [Let It Crash: Best Practices for Handling Node.js Errors on Shutdown](https://yomu.fyi/post/let-it-crash-best-practices-for-handling-node-js-errors-on-shutdown.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Julián Duque
- Published: Dec 18, 2019
- Type: Tutorial

Node.js applications can lose visibility into failures and experience downtime when processes crash or shut down without useful logging and recovery practices. The post presents lessons from consulting and a talk on the Node.js process lifecycle, focusing on graceful shutdown, error visibility, and fast restart after catastrophic errors. It recommends keeping startup lean by prebuilding data or compiling assets, then contrasts process.exit(), which accepts an exit code, with process.abort(), which terminates immediately and may generate a core dump for llnode-based postmortem debugging. It also explains beforeExit and exit, including their differing asynchronous-work behavior, and introduces SIGTERM and SIGINT as operating-system signals that applications can handle. The intended result is a restartable process strategy that reduces downtime, while preserving evidence needed to investigate crashes.


### [How Shopify Manages API Versioning and Breaking Changes](https://yomu.fyi/post/how-shopify-manages-api-versioning-and-breaking-changes.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 17, 2019
- Type: Explainer

Shopify describes how it versions APIs and manages breaking changes for an ecosystem of tens of thousands of partners. It releases date-based API versions every three months, with changes introduced in an unstable version and represented in the monolith through frozen records and a change file. Teams classify changes by whether third-party developers must migrate, use impact analysis and forward-compatibility testing, and protect uncertain changes with beta flags. For necessary breaking changes, tooling marks breaking or possibly breaking request paths, sends metadata through Monorail to a data warehouse, and produces reports on affected calls. Runtime conditionals then help teams manage old and new behavior, measure adoption, identify remaining risk, and communicate with developers.


### [How We Implemented Domain-Driven Development in Golang](https://yomu.fyi/post/how-we-implemented-domain-driven-development-in-golang.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Kapil Chaurasia
- Published: Nov 21, 2019
- Type: Problem & solution

Building GrabPlatform's partner integration self-service portal initially resulted in an unstructured codebase where individual files exceeded 500 lines and lacked proper segregation. Modifying existing functions carried high risks of breaking functionality across imported source collections. To resolve this, the team restructured the Go application using Domain-Driven Design principles in coordination with product domain experts. They mapped business rules into bounded contexts, identified entities and aggregate roots, introduced repository interfaces, and utilized domain events for cross-context communication. The refactoring distributed core functionality evenly, simplified onboarding, and aligned technical terminology with business concepts.


### [Successfully Merging the Work of 1000+ Developers](https://yomu.fyi/post/successfully-merging-the-work-of-1000-developers.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 14, 2019
- Type: Problem & solution

Shopify’s single monolith supports changes 40 times a day and roughly 400 commits to master daily, making safe trunk-based development difficult at the scale of 1,000+ developers. Its first Merge Queue limited deployment drift and batch size, but queued pull requests did not receive CI, soft conflicts could break master, and a browser extension enabled accidental direct merges. Merge Queue v2 replaces that workflow with GitHub comments and webhooks, programmatic branch protection, an emergency /shipit --emergency path, and a predictive branch where queued changes receive CI before merging. A reconciliation algorithm synchronizes that branch through GitHub’s GraphQL API, while failure-tolerance thresholds handle flaky tests and remove repeatedly failing pull requests. CI starts when requests enter the queue; batches of eight and three concurrently tested batches balance throughput, risk, and resource use, while the changes improved user experience, deployment safety, and deploy throughput.


[Newer posts](https://yomu.fyi/posts/page/235.md) · [Older posts](https://yomu.fyi/posts/page/237.md)