Loading…
Browse by type
Explainer reads
Explainer posts on Yomu, summarised newest first.
- 1,318 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Explainer posts
Five Common Data Stores and When to Use Them
The article compares five common data stores—relational databases, NoSQL databases, key-value stores, full-text search engines, and message queues—and connects each to appropriate design needs. Relational databases use schema-bound tables, primary and foreign keys, SQL, and ACID transactions for durable business-critical data, while NoSQL databases trade strict schemas for flexibility with document and wide-column models. In-memory stores such as Redis and Memcached provide fast access and configurable eviction for caching; Redis additionally supports richer data structures and disk persistence. Full-text search engines target word and substring searches, whereas Kafka uses a distributed append-only log and producer-consumer model to move data reliably between services. The article concludes that the right choice depends on whether data must be permanent, flexible, quickly accessed, searched, or transferred.
How Shopify Scales Up Its Development Teams
The article argues that scaling development is primarily about creating more effective teams, not simply increasing headcount. It first recommends checking whether an existing team prioritizes impactful work, minimizes supporting activity, and can ship code quickly while finding bugs; otherwise hiring may be premature. Shopify organizes product work into problem spaces and teams of five to nine, uses a multi-step process combining technical exercises with cultural assessment, and aims for at least as many onboarded members as onboarding members. New teams receive a baseline for work practices and technology, while transparent technical design, shared tooling with dashboards and embedded quality and testing standards, optional pairing, and frequent retrospectives support consistency and learning. The conclusion is that these practices, rather than hiring alone, enable development teams to scale more effectively.
Being a Principal Engineer at Grab
Grab's rapid growth resulted in roughly 350 microservices powering its superapp, creating the need for defined individual contributor career milestones. At Grab, a principal engineer oversees the architecture of an entire Tech Family, a sub-organisation containing over 50 engineers and 20 or more microservices. Responsibilities include translating broad, ambiguous problems into concrete projects, managing technical debt, and aligning multiple engineering teams across global R&D centres. The role demands continuous technical leadership through RFC design reviews, cross-functional communication, mentorship, and self-directed prioritization alongside engineering leadership. Ultimately, principal engineers amplify engineering quality and operational stability without directly managing people.
Up to 75% Faster Maintenances with Heroku Postgres and Redis Premium Plans
Heroku describes how it minimizes downtime during routine security and patching maintenances for high-availability Heroku Postgres and Heroku Redis plans. These plans run a writable primary alongside a hidden read-only standby, and maintenance requires moving application writes safely to the standby before promoting it. Previously, Heroku changed DATABASE_URL and restarted app dynos, then waited for the old primary to shut down and replicate remaining writes, which could extend downtime during long-running transactions. The new method redirects new connections to the new primary, kills existing connections to the old primary, and relies on reconnections plus a rolling restart to prevent further writes there. Heroku reports HA maintenances now complete in 15–40 seconds instead of 60 seconds or more, and says the behavior is default for Premium, Private, and Shield plans.
Living on the Edge of Rails
Shopify describes running its Core monolith on the unreleased Rails 6 master branch, or “Edge of Rails,” since February 2019, with the goal of continuously using the framework’s latest revision. Instead of annual upgrades, weekly changes reduce diff size and let developers use upstream features quickly, while a 130,000-plus-test suite finds edge cases and informs patches to Rails and related projects such as Zeitwerk. Solid Track automates weekly Rails bumps, opens GitHub pull requests, and triggers git bisect when CI fails to identify the breaking commit. Green builds move through ten-minute canary deployments before reaching the remaining servers. The approach trades the stability of released dependencies for faster fixes and features, relying on monitoring, tests, and controlled deployment to manage risk.
Pagination with Relative Cursors
Incremental page-number pagination becomes increasingly expensive for large offsets, because the database scans and discards earlier records before returning a page. Shopify describes relative cursor pagination, which remembers the last record seen and continues from that point, using an indexed id to keep query time nearly constant as pages increase. In tests on a shop with 14 million products, a 100,000 offset took 2,221.60 ms, compared with 5.24 ms using the last id, while offsets of 1,000,000 and above consistently timed out. Sorting by another field requires retaining both its last value and a unique secondary key such as id, with a matching compound index to avoid skipped records and maintain performance. Shopify introduced Link-header URLs for next and previous pages and began migrating REST endpoints from page numbers, reporting early /admin/products.json usage as about 11 times faster, though adoption remained low and full results were pending.
Lessons from Leading a Remote Engineering Team
Managing remote engineering teams can expand hiring beyond office locations, support a more diverse workforce, retain employees through relocations, and enable distributed coverage such as 24/7 support. The guidance describes building such teams around relationships, trust, and communication, with “Default to open internally” and “Charge your trust battery” as supporting cultural values. It recommends asynchronous documentation and discussion in email, GitHub issues, Google Docs, and project systems, supplemented by a fixed weekly synchronous meeting and shared core hours. Managers should also support healthy home workspaces, maintain regular one-on-ones and timely contact, arrange selective in-person sessions, and actively champion remote employees’ visibility, recognition, and career progression. The approach requires deliberate effort but can provide hiring, composition, retention, and strategic benefits.
Dataclips Power Insights at Heroku
Heroku Dataclips let employees create SQL queries through a web GUI against Heroku Postgres databases, then share results through unique URLs or HTTP endpoints returning CSV or JSON. At Heroku, Dataclips reduce dependence on engineers or DBAs by letting departments retrieve fresh production metrics, build dashboards, track product usage, identify deprecated-feature users, and investigate suspicious signup patterns. To make querying safer, the organization runs dataclips against a read-only Postgres follower rather than the leader; writes propagate from leader to followers in close to real-time. Business Operations also teaches SQL and the internal schema through Office Hours, aiming to replace recurring one-off requests with self-service query skills.
How Heroku Operates its Multi-Tenant Apache Kafka Services
Heroku’s Apache Kafka service supports thousands of developers and millions of transactions, including customers sharing multi-tenant clusters. The post explains how those clusters provide security, predictable performance, availability, and production-like topology at lower cost than single-tenant plans. Multi-tenant and single-tenant offerings use the same numbers of brokers and ZooKeeper nodes, while Kafka ACLs, per-tenant resource namespaces, and producer/consumer byte quotas isolate customers and limit resource contention. Heroku provisions purchased resources, expands storage at 80% of quota, and uses higher defaults for partitions, replicas, and in-sync replicas; exceeding capacity can eventually throttle throughput without stopping the cluster. Automated tests model 50 users with varied traffic, exercise throughput and partition configurations, and simulate broker failures and restarts to identify operational limits and adjust the infrastructure before customer impact.
Making Grab’s Everyday App Super
Grab manages an expanding superapp ecosystem comprising ride-hailing, food delivery, payments, and partner content surfaced through the Grab Feed. As content volume grows, the platform risks overwhelming users with irrelevant information. To address this, Grab built a recommendation engine that ranks cards using signals across user profiles, content metadata, and contextual factors such as time and location. The system employs multiple recommendation strategies—including popularity metrics, user favorites, collaborative filtering, habitual patterns, and cross-platform deep embeddings—which are selected or aggregated. Recommendation quality is evaluated via offline metrics like Recall@K and NDCG alongside online engagement experiments.
Six Strategies for Deploying to Heroku
Heroku offers six deployment strategies: Git push, GitHub integration, Review Apps, Docker, Terraform, and the “Deploy to Heroku” button. Each has different requirements and trade-offs, and strategies can be combined in a workflow. Git push is simple but requires a developer with access to both repository and app, while GitHub integration can automate deployments and block them until CI tests pass. Review Apps provide disposable instances for pull requests; Docker supplies self-contained environments but requires maintaining a stack and does not support pipeline promotions. Terraform uses declarative HCL to manage Heroku and external infrastructure despite lacking official Heroku support, and buttons offer one-click deployments from GitHub repositories but do not auto-update or connect deployed apps back to their repositories.
Implementing Android POS Receipt Printing on Shopify
Shopify’s Android POS lacked printed-receipt support until March 2019, leaving merchants who needed it to use iPads. The design targets a pragmatic, adaptive, composable, maintainable, and highly testable API, while handling more than 8 billion possible receipt content combinations across discounts, tips, transactions, gift cards, refunds, payment methods, and regional requirements. Its six-step pipeline aggregates models from SQLite or GraphQL, transforms them into a PrintableReceipt, defines view-based sections, creates printer-agnostic PrintRequest objects, renders section bitmaps at the required resolution, and passes them through the Hardware SDK to vendor-specific integrations. The PrintableReceiptComposer contains most business logic, with unit tests covering features and edge cases, while screenshot tests compare rendered output pixel by pixel in CI. This componentized separation isolates printer support from receipt generation and allows new printer models to be added in a couple of hours.
Connecting the Invisibles to Design Seamless Experiences
Service design operates as connective tissue within complex product ecosystems by bridging digital touchpoints, physical operations, and backstage technical workflows. At Grab, placing an order on GrabFood requires coordinating driver allocation, customer support paths, and long-term data storage rather than simply transmitting information to merchants. Focusing exclusively on singular features risks breaking broader network dependencies when modifications ripple into other operational systems. Grab addresses these interdependencies through participatory design processes and visual mapping across cross-functional teams. This holistic framework evaluates whether systemic issues, such as inaccurate restaurant operating hours, are best resolved through in-app feature changes or operational adjustments.
Tourists on GrabChat!
Grab examined more than 3.7 million tourist messages across Singapore, Malaysia, and Indonesia sent between December 2018 and March 2019 to evaluate passenger communication patterns. The platform deployed in-house translation and prewritten, auto-translated chat templates to bridge language barriers between international riders and local drivers. Analysis showed that bookings utilizing chat templates experienced a 10% higher ride completion rate than those without. Image-sharing features were most heavily used in high-traffic hubs such as airports, shopping malls, and major tourist centers to aid driver location. Passengers also consistently used messaging to clarify luggage capacity, provide identifiable passenger descriptions, and check pet policies.
Bubble Tea Craze on GrabFood!
GrabFood recorded a regional average order growth rate of 3,000% for bubble tea across Southeast Asia in 2018. Individual country growth rates ranged from over 250% in Malaysia to more than 8,500% in Indonesia over their respective tracking periods. The customer base for bubble tea expanded by over 12,000%, supported by a 200% increase in merchant outlets to nearly 4,000 locations representing over 1,500 brands. Southeast Asian consumers ordered an average of four cups per person per month, led by Thailand at six cups and the Philippines at five cups. Order timing concentrated primarily around lunchtime meals and midday afternoon breaks.
Pair Programming Explained
Pair Programming Explained presents pair programming as two people writing, debugging, or exploring code together, while explaining why Shopify uses it for knowledge sharing, collective ownership, learning, quality, and onboarding. It defines terms including problem, solution, driver, navigator, expert, novice, and disengagement, and stresses that the driver handles implementation while the navigator guides broader strategy and spots errors. Guidance covers when to pair, the trade-offs among expert-expert, expert-novice, and novice-novice combinations, and the risks of “watch the master” or disengagement. Suggested practices include switching roles with the Pomodoro Technique, using Strong Style or Digging for Gold, and holding a retrospective to discuss flow, learning, and tension. The article presents pairing as a focused collaborative practice that benefits from shared attention, regular breaks, and deliberate communication rather than an unstructured coding session.
Getting to Know Python 3.7: Data Classes, async/await and More!
Python 3.7 is presented as an upgrade for developers seeking new language features, library improvements, and performance gains over earlier Python 3 releases. The article focuses on data classes, where the @dataclass decorator inspects typed attributes and generates an init method, reducing boilerplate for data-oriented classes. It also covers asyncio.run for simpler asynchronous execution, breakpoint() for entering pdb, module-level getattr for lazy loading, and context variables that keep values separate across execution contexts. Dictionary insertion order becomes part of the language specification in 3.7, while OrderedDict remains available for advanced behavior; reported improvements include 10–30% faster startup, 45–70% faster common sorting cases, 5.5-times-faster dict.copy(), and 4–6-times-faster namedtuple creation.
Building Shopify’s Application Security Program
Shopify's application security program is designed to preserve safety while products and services scale quickly across a platform serving more than 800,000 merchants. It organizes the work around scaling secure applications, scaling security teams, and scaling security interactions, with a culture that treats security as a non-negotiable priority. Standardizing on Ruby on Rails and other shared technologies lets tooling go deep, while service tracking provides visibility into applications, dependencies, code, and automated dependency updates. The security organization grew from one employee into specialized teams, supplemented by penetration testing, bug bounties, and audits, while incident response remains shared across the company. At the code level, tripwires detect dangerous patterns early; Shopify renamed html_safe to dangerously_output_as_html, links flagged pull requests to XSS training, and alerts Application Security, shifting review toward automation.
Turn Your Code into Docker Images with Cloud Native Buildpacks
The post presents Cloud Native Buildpacks (CNB) as an alternative to hand-written Dockerfiles for turning application source code into Docker images. It argues that Dockerfiles make developers manage language runtimes, system packages, dependency caching, cleanup, and repeated maintenance, creating a leaky abstraction between application and deployment concerns. CNB uses app-aware buildpacks to detect a project’s language from files such as Gemfile or pom.xml, reuse work from previous builds, download dependencies, prepare the application, and export an OCI-compliant image. The beta buildpacks.io tooling provides pack build and a Heroku builder supporting Ruby, Node.js, Java, Python, PHP, and Go; pack rebase can update an image’s toolchain without rebuilding it. The stated conclusion is that this approach reduces operational complexity while retaining compatibility with Docker tooling and the broader container ecosystem.
Designing Resilient Systems Beyond Retries (Part 3): Architecture Patterns and Chaos Engineering
Building resilient systems requires architectural safeguards and proactive testing beyond basic retries and circuit breakers. Architectural patterns such as idempotency keys enable safe retries without creating inconsistent state during failures. Asynchronous responses and deferrable work isolate services from downstream dependency latency and errors, though they can conflict with the fail-fast principle. To validate system behavior under stress, chaos engineering introduces intentional failures in production to test hypotheses against a defined steady state. Selectively adopting complementary patterns reduces failure points while avoiding unnecessary architectural complexity.
Top topics
- Architecture830
- Performance812
- LLMs648
- Machine Learning617
- Open Source561
- AI Agents542
- AI454
- Developer Experience446
Weekly digest
The best engineering reads, once a week.