Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,535 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Caching Without Marshal Part 1: Marshal from the Inside Out
Rails caching commonly relies on Ruby’s Marshal serialization, which can encode complex objects but also embeds their class names in cache entries. Shopify describes an incident in which a beta-flag refactor changed classes while old and new code overlapped during deployment; old code then failed on cached instances containing unfamiliar class names and methods, despite passing CI. The article examines Marshal through Ruby’s marshal.c implementation, covering atomic and composite types, instance variables, object references, circularity via TYPE_LINK, and core-type subclasses via TYPE_UCLASS. It presents MessagePack as a more compact, stricter, and controllable alternative intended to make caching safer by default, while deferring the cache migration and implementation details to the series’ next part.
Apollo Cache is Your Friend, If You Get To Know It
Shopify’s migration from Apollo GraphQL client 2 to client 3 prompted a closer examination of the Apollo InMemoryCache after past bugs were linked to misunderstanding or misuse. The cache stores an in-memory representation of queried data for the current browser session, and its lifecycle covers fetching, normalization, updating and merging, then garbage collection and eviction. Fetch policies determine whether data comes from the cache, the network, or both; under the default cache-first policy, incomplete data triggers a network request. Normalization breaks responses into objects, assigns cache identifiers usually from __typename and id or configured key fields, and stores them in a flattened structure. Automatic UI updates depend on matching identifiers and suitable query or mutation responses, while objects made unreachable by changed identifiers can remain until garbage collection removes them.
Authentication in Ionic Angular with Supabase
The tutorial builds a simple realtime messaging app with Ionic Angular and Supabase, covering account authentication and database protection. It defines Postgres tables for users, groups, and messages, enables Row Level Security with policies for public reads and authenticated operations, and adds a trigger to copy newly registered users into a public users table. Angular services, route guards, and Supabase client logic handle email/password registration and login, session state, protected navigation, and magic links. For native apps, Capacitor URL events extract access and refresh tokens from a custom-scheme callback before establishing the session and navigating to the groups page. The tutorial concludes that Supabase can support Ionic Angular authentication with magic links, while phone authentication through Twilio is another possible option.
Logbook: October 29 to November 6, 2022
This weekly logbook covering October 29 through November 6, 2022 reports updates across Elixir, Postgres, Laravel Livewire, Redis, and Fly Machines. Chris McCord describes how React inspired Elixir LiveView, while revamped Elixir documentation and Livewire tutorials cover client-side pagination, grouping, sorting, and browser notifications from the server. Fly’s revised Postgres documentation distinguishes the tools it provides for provisioning and managing a cluster from the responsibilities customers retain, and points readers to managed Postgres services. The Fly Redis dashboard command opens the Upstash Redis console with instance statistics and connection instructions, while a Fly Machines preview supports defining multiple processes to run inside one container; the log also notes updated documentation and a required CLI version update.
How to Categorize and Prevent Risks of Sensitive Links in URLScan
Security integrations with URLScan can accidentally index sensitive links when API calls are executed with public visibility. Tinder Security Labs constructed targeted queries to uncover exposed links across file-sharing services, enterprise ticketing platforms, and single sign-on portals. Investigations revealed that exposed links included document magic links containing contracts and bank details, as well as active new-hire registration links that allowed attackers to complete registration workflows and configure two-factor authentication. In one SSO query alone, researchers discovered 1,300 indexed invitation links, with 20 to 50 new links appearing daily. The researchers collaborated with URLScan to purge the exposed indexes from public searches and advised organizations to configure security scanning tools to use private or unlisted visibility settings.
Reducing BigQuery Costs: How We Fixed A $1 Million Query
During the infrastructure work for a marketing tool, Shopify's team found a BigQuery query that would have processed about 75,462,743,846 bytes per request and cost nearly $1 million monthly at an estimated 60 requests per minute. The query served a pipeline ingesting one billion rows through Apache Flink, with state managed by RocksDB and streaming requests from Apache Kafka; scaling beyond the release made ingestion unsustainable. To support general availability, the team evaluated an external SQL warehouse that could load Parquet atomically, handle 60 requests per minute, and export results to Google Cloud Storage, then clustered a dataset on two feature columns used in WHERE clauses. Running the same query on the clustered table reduced billed data to 508.1 MB and identified 108.3 MB scanned, lowering estimated monthly cost to about $1,370.67; the post also recommends selecting needed columns, partitioning tables, and using free previews instead of exploratory queries.
Supabase Beta October 2022
Supabase’s October 2022 update reports a broad set of product, documentation, and community releases, while announcing that Launch Week 6 will run from December 12–16. The new supabase-js v2 and supabase-flutter v1 SDKs reached full release, alongside a revamped Next.js quickstart with pre-built Auth UI and Auth Helpers and a Next.js 13 example for fetching and caching data from the Serverless API. Documentation covers Supabase Auth’s server-side rendering flow, and a database-testing framework uses pgTAP and pg_prove; Edge Functions also gained support for GET, PUT, PATCH, and DELETE requests. Additional changes include functions support in supabase-py, session refreshing, provider refresh tokens, request-location detection through X-Forwarded-For, expanded logging, and storage downloads via a query parameter, alongside three featured community apps and hiring announcements.
A love letter to React
React’s 2013 release had a lasting impact on frontend development and influenced Phoenix LiveView through its reactive component model, JSX colocation, and efficiency. Chris McCord explains how a component’s state and render function create a simple programming model, then shows LiveView adapting that model to a stateful server process with HTML-aware function components and slots. The discussion argues that tightly coupled markup and code should live together, while extensible components preserve HTML structure and support customization without bespoke templates. For rendering efficiency, React computes virtual DOM diffs, whereas LiveView tracks static and dynamic HEEx template parts, sends minimal diffs over the network, and applies client-side patches with morphdom. The conclusion credits React with shaping Phoenix’s direction and argues that backend frameworks should provide HTML-aware component systems.
Logbook: October 21 to 28, 2022
This weekly Logbook changes format from tracking every monthly change to an over-generalized weekly tl;dr of Fly.io updates intended to help with deploying or managing apps. It reports new Postgres documentation for performing failovers, regional failovers, setting up and attaching a database, and migrating one from Heroku, while warning that Fly Postgres is not fully managed and requires customer monitoring. The post also points to an example of integrating the Elastic Stack (ELK) with a Laravel app for user analytics and reporting, plus instructions for shutting down an idle Phoenix app on Fly Machines after a configurable period without requests. It explains that newly launched Fly apps do not yet deploy to Machines in the same way, while fly pg create does use Fly Machines, and directs readers to comparisons of the app versions. Finally, it notes colorblind-friendly dashboard improvements and previews more Postgres documentation work.
Mixing It Up: Remix Joins Shopify to Push the Web Forward
Shopify announces that the open-source web framework Remix and its team are joining the company to help developers deliver lightning-fast, resilient web experiences. Remix combines multi-page and single-page strengths through a full-stack React framework with data loading and code splitting handled without requiring front-end developers to manage those backend tasks. It also emphasizes progressive enhancement, with links, buttons, and forms working before JavaScript loads, and supports edge, service-worker, and Node runtimes. Shopify will use Remix across projects including Hydrogen; Hydrogen is moving from server components to Remix's data-loading pattern for faster performance and a simpler developer experience, with ~90% of developers' code expected to remain unchanged. Remix will remain independent and open-source while Shopify supports its roadmap and community.
The Management Poles of Developer Infrastructure Teams
Developer infrastructure managers must balance three poles: management support, system and domain expertise, and organization-wide road maps, with the tension intensified by teams that often lack dedicated product managers and support critical systems for hundreds or thousands of users with six to eight developers. Domain-focused teams build deep knowledge needed for maintenance, migrations, incident investigation, on-call coverage, and developer-experience improvements, but their local road maps can diverge from the department’s highest-impact opportunities. The proposed responses are temporary individual assignments, having a whole team contribute to another team’s goals, and forming a cross-functional tiger team; each shifts trade-offs among focus, maintenance, expertise, motivation, and management oversight. A month-long tiger team built the first Spin proof of concept, which led to a dedicated team, but no structure eliminates the tensions, so choices must be judged according to context.
What is PostgreSQL commitfest and how to contribute
PostgreSQL Commitfest is the community's month-long review process for patches intended for merging into core. Contributors submit patches to pgsql-hackers, create a Git branch from master, consolidate changes, generate a versioned patch, and associate its mailing-list thread with an open Commitfest entry. The process expects authors to explain rationale, respond to review, rebase or revise patches, and review a comparable number of other patches. Submitted changes are checked by CFBot across architectures and operating systems, while statuses such as "Needs review," "Ready for committer," and "Waiting on author" track progress. Commitfests occur five times yearly, with the March event tied to feature-freeze for the October major release.
A Software Engineer's Guide to Working Across Time Zones
Working across time zones can make live collaboration difficult when teammates are nine to 12 hours apart, as described by a developer based in Singapore. The proposed alternative is to treat much of the work as asynchronous and deliberately document progress, blockers, decisions, and context. Examples include taking turns across working hours, annotating non-obvious pull request changes, posting end-of-day summaries with links or investigation notes, and recording meetings with agendas, key decisions, and action items. During Shopify's Hack Days prototype work on Linkpop, a Singapore-based frontend developer handed off updates before sleeping while North American teammates extended backend functionality and responded to blockers. The post concludes that these practices become habits that support handoffs, reduce back-and-forth, preserve information for absent teammates and future reference, and can provide longer uninterrupted focus periods.
Using mobile sensor data to encourage safer driving
Dedicated vehicular inertial measurement units are prohibitively expensive to install across large fleets, while customer feedback lacks the coverage required to assess driving safety comprehensively. To monitor driver behavior at scale, Grab processes smartphone sensor data collected via a mobile telematics SDK at high sampling frequencies. A custom orientation algorithm calculates Euler roll and pitch angles from gravity and resolves yaw through principal component analysis to align phone measurements with vehicle axes. The pipeline applies low-pass or Kalman filtering to handle varying noise levels, followed by threshold peak-finding algorithms to detect harsh acceleration, braking, and cornering. Speeding events are identified by comparing GPS speeds against known map limits or aggregated speed profiles derived from free-flowing traffic.
supabase-flutter v1 Released
Supabase has released the stable v1 of its supabase-flutter SDK after several months of developer preview feedback and improvement. The release targets developer experience by reducing boilerplate and introducing more intuitive APIs, including removal of the required .execute() call from select, insert, update, delete, and stream operations; query errors are now thrown rather than returned. Authentication methods use more descriptive names such as signInWithPassword and signInWithOAuth, while onAuthStateChange returns a Dart stream. The SDK also adds first-class support for Realtime Multiplayer's broadcast and presence methods, including low-latency data sharing and client-status tracking, and the announcement provides updated documentation, a quick start guide, and a v0-to-v1 migration guide.
Hubble: Our Tool for Encapsulating and Extending Security Tools
Shopify’s Trust team faced rising complexity in securing a distributed device fleet, including many tools, provisioning and patching work, varied networks, and the loss of in-person remediation. Hubble was built as a unified layer over mobile device management and other security systems, ingesting and standardizing their data while sending commands back with granular access controls and centralized auditing. IT staff use it for inventory, device management, and security, while employees can view device health and compliance, receive remediation guidance, and manage test-device or beta participation. The platform also serves as a standardized source for automation, and the post presents encapsulation and cross-team investment as ways to reduce administrative overhead, support proactive security, and limit risk.
supabase-js v2 Released
Supabase-js v2 has been released after a release-candidate period during which the team incorporated user feedback and updated documentation, examples, quickstart guides, and a migration guide. The release’s major addition is built-in TypeScript support, including a CLI that generates types directly from a database. It also includes improvements aimed at addressing some of the largest pain points identified by users, with the announcement providing updates and differences from v1. Version 2.0 was produced through contributions from several Supatroopers, more than 100 contributors to the libraries, and more than 450 contributors to the documentation and websites; the listed package contributions include functions-js, gotrue-js, postgrest-js, realtime-js, storage-js, and supabase-js.
How to Structure Your Data Team for Maximum Influence
Embedded data science leaders may support multiple product and business areas, creating competing work streams, specializations, and stakeholder relationships. The post compares swim lanes and stochastic process assignments against principles of efficiency, influence, stakeholder clarity, stability, growth, and flexibility. It recommends a hybrid “diamond defense” in which scientists are loosely assigned to zones for specialization while a bullpen, including the manager and an additional scientist, supplies reserve capacity. Resources can be redirected to overloaded areas, and team members can rotate into vacancies or cover leave. The framework is presented as combining clear ownership and subject-matter depth with the flexibility to handle change, though the post warns that excessive rotation can create volatility and turnover.
Finding Relationships Between Ruby’s Top 100 Packages and Their Dependencies
RubyGems’ phased MFA rollout required owners of gems with at least 180 million downloads to use MFA, but raised a supply-chain question: could a popular gem depend on a less-downloaded gem that remained an account-takeover target? The investigation loaded rubygems.org data dumps to identify 112 gems above the threshold, queried the API for direct dependencies, and used Bundler with Gemfile.lock files to resolve transitive dependencies. It found 13 big gems with small direct dependencies and 24 with small dependencies overall, attributing the mismatch mainly to newer dependencies and gems shipped with Ruby, such as racc and rexml. Graph visualizations, breadth-first traversal, and a custom depth-first search then mapped dependency structure and paths; the post leaves open whether to do nothing or enforce MFA early for the 24 technically insecure gems.
Postgres Full Text Search vs the rest
The post compares PostgreSQL Full Text Search with dedicated search engines including MeiliSearch, OpenSearch, SQLite FTS, and Typesense, asking whether a separate system is justified for common application search. It presents a PostgreSQL setup using a generated stored column built with to_tsvector('english', ...) and a GIN index, queried with websearch_to_tsquery. The benchmark loads a movie dataset and focuses on query speed after data ingestion, using comparable scripts and default configurations with limited tuning. PostgreSQL was close in performance to several alternatives; in-memory SQLite was fastest, while Typesense and MeiliSearch handled misspellings correctly and OpenSearch ingested data quickly. The conclusion is that PostgreSQL can provide good-enough search quality and speed for basic cases without adding another system, although dedicated engines offer capabilities PostgreSQL does not.
Top topics
- Architecture830
- Performance812
- LLMs648
- Machine Learning617
- Open Source561
- AI Agents542
- AI454
- Developer Experience446
Weekly digest
The best engineering reads, once a week.