Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,535 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Perspectives on Onboarding: Joining Shopify as an Engineering Leader
Shopify’s onboarding program is presented as an alternative to the common first-day experience of receiving a laptop, documents, and a team without enough context. It gives new engineers, including engineering leaders, a structured period of roughly six weeks to learn the product, merchants, engineering practices, and company values, including by creating test shops and using the admin. Avanti Ketkar and Maria Robledo describe how this preparation helped them connect onboarding content to later work, ship a pull request after using a test shop, and build relationships and trust in a remote environment. Their accounts also cover the adjustment to learning before delivering impact, moving from a .NET background to Ruby and Rails, managing time zones, and keeping technical craft relevant while leading teams.
Logbook - 2022-06-23
Fly.io's June 23, 2022 logbook records a broad set of platform, dashboard, and flyctl updates. It announces YUL (Montreal, QC, Canada), adds Laravel and Laravel Octane detection to fly launch, and improves plain Docker build output when NO_COLOR=1 is set. Backend work optimizes API queries, reduces repeated app-image loads, moves app-discard cleanup into a retryable job, and adds SQL comments to help investigate database data-lock issues. The release also documents diagnostics, makes compliance documents public to prospective customers, and improves fly-proxy’s Rust coverage and PROXYPROTO v2 TLV support. Fixes address fly.toml environment variables, organization deletion, password-reset redirects, app secrets listing, metrics after fly apps move, dashboard errors, malformed SSL TLVs, and overzealous email risk checks.
Getting Started With Embeddings
Embeddings represent unstructured information such as text and images as numerical vectors in a shared semantic space. To demonstrate their utility, a simple semantic search engine is built over US Social Security Medicare frequently asked questions. The system generates 384-dimensional vector representations for thirteen FAQ entries by dispatching POST requests to the Hugging Face Inference API using the sentence-transformers/all-MiniLM-L6-v2 model. Incoming user queries are converted into matching vector representations and evaluated against stored dataset vectors using the util.semantic_search function from the Sentence Transformers library. By calculating cosine similarity scores, the system retrieves and ranks the five most semantically relevant questions without requiring custom keyword rules or massive labeled training sets.
How We Built Hydrogen: A React Framework for Building Custom Storefronts
Hydrogen is a React framework for building custom Shopify storefronts, created to help merchants use the Storefront API without giving up capabilities associated with Liquid storefronts. Shopify built it around React Server Components and React 18’s streaming server-side rendering, using component-level data fetching, a preload cache to parallelize nested queries, and experimental loggers for query waterfalls and unused GraphQL properties. The framework evolved through more than 1,200 pull requests and 3,000 commits, with rigid components removed after feedback and nearly all exported GraphQL fragments eliminated to make queries more discoverable. The post reports that some demo store routes cut load time in half and that Hydrogen is used in production at Allbirds, Shopify Supply, Shopify Hardware, and Shopify, while future plans include a new router and incremental adoption from Liquid storefronts.
Introducing ShopifyQL: Our New Commerce Data Querying Language
Shopify introduces ShopifyQL, a commerce-focused querying language intended to make data exploration accessible to Shopify Plus merchants without data science or coding backgrounds. It addresses the limits of dashboards and reports, which show what happened but may not explain why, and simplifies access to modeled commerce datasets that would otherwise require SQL and data-lifecycle knowledge. Its syntax uses clauses such as FROM, SHOW or VISUALIZE, BY, WHERE, SINCE, UNTIL, ORDER BY, COMPARE TO, and LIMIT, with direct charting, commerce-specific date ranges, and planned native period comparisons. ShopifyQL is available through the ShopifyQL Notebooks first-party app, while GraphQL APIs let partners submit queries and receive results from the underlying datasets. Shopify says it will use the language across Shopify Admin analytics and plans features including more visualizations, pivoting, and aggregate conditions.
Convert Transformers to ONNX with Hugging Face Optimum
Exporting Hugging Face Transformers models to the ONNX format can be accomplished through three different abstraction levels. The low-level approach utilizes torch.onnx.export, requiring manual specification of dummy inputs, input names, output names, opset versions, and dynamic axes configurations. At the intermediate level, the transformers.onnx package simplifies the conversion process by relying on FeaturesManager and prebuilt configuration objects to handle dynamic axis definitions automatically. The high-level method uses Hugging Face Optimum classes such as ORTModelForSequenceClassification by setting the from_transformers flag to True inside from_pretrained. This Optimum export leverages transformers.onnx internally and produces a model ready for immediate inference execution or integration into pipelines.
Python data loading with Supabase
Supabase offers a PostgreSQL-based backend-as-a-service platform alongside an open-source Python SDK for automating CRUD operations and data-intensive tasks. After spinning up a new project in the Supabase dashboard, developers can configure relational schemas in the Table Editor by establishing foreign key constraints between tables like Vendor and Product. Authentication for the SDK relies on passing project URL parameters and API keys, which can be securely stored and loaded through environment variables. Programmatic data population utilizes the supabase client library combined with the Faker package to generate realistic synthetic entries and push them directly to database tables using insert and execute calls. Populated records can subsequently be examined in real time through the dashboard Table Editor interface to verify schema integrity and data persistence.
How to make Rust leak memory (also: how to make it stop)
fly-proxy, a Rust-based proxy, leaked memory because its obsolete custom HoneycombLayer retained child spans under a long-lived parent span. The post follows a request from browser input through DNS, TCP, TLS, HTTP/2, and asynchronous Rust components including an aya-built eBPF program, tokio, hyper, and tracing, before explaining how the leak surfaced. In the custom layer, every closed child span was added to its parent’s collection, so each request consumed a small amount of memory while a high-level async task remained alive. Removing that legacy code fixed the primary problem: memory usage barely exceeded 50MB during a load test, while remaining growth came from queues and latency data retained for processing rather than the identified leak.
Making Open Source Safer for Everyone with Shopify’s Bug Bounty Program
Shopify describes how a bug bounty report led to application fixes and Rails contributions aimed at making open-source software safer. The 2021 Shopify Experiments report showed that a leaked development-environment secret could be used to generate trusted-looking data and reach remote code execution in a local development environment. The root cause was ActiveSupport’s MessageEncryptor using Ruby’s unsafe Marshal serializer by default in Rails versions up to 7.0.X, allowing untrusted data to trigger deserialization risks. Shopify first configured JSON in its application, then helped change the default serializers for MessageEncryptor and MessageVerifier to JSON in Rails 7.1.X, with an upgrade path for existing applications. It also developed a patch that monitors Marshal load and dump calls, raises on unexpected gem-originated calls, and runs in local development and continuous integration to prevent new usage.
Intel and Hugging Face Partner to Democratize Machine Learning Hardware Acceleration
Intel has joined Hugging Face's Hardware Partner Program to accelerate Transformer training, fine-tuning, and inference on Intel platforms. Large Transformer models introduce latency bottlenecks in production workloads like search and chatbots, where hardware-level optimization typically requires tedious trial and error. To streamline model optimization, the collaboration introduces Optimum Intel, an open-source library integrating the Intel Neural Compressor for automated quantization, pruning, and distillation. A demonstration applies post-training dynamic quantization to a fine-tuned DistilBERT classification model using a CPU-only PyTorch setup. The quantized model converted 38 Linear and 2 Embedding operators to 8-bit integers, reducing evaluation duration by 1.34x while keeping the accuracy drop within a 5% threshold.
How We Built Shopify Party
Shopify Party is a browser-based internal tool designed to make virtual hangouts more social and reduce video fatigue after Shopify moved to remote work. Its 3D spaces favor non-prescriptive play, letting participants explore and manipulate objects, join or leave at any point, and use a small control set rather than follow prescribed rules or win conditions. The initial version was built in three months with Unity 2020 and the Universal Render Pipeline, exported to WebGL, while Normcore synchronizes game state, provides VoIP and networked physics, and uses room time to keep autonomous objects aligned across clients. Performance work included third-party assets, GPU instancing for repeated models, simpler materials, and adjustable render resolution. The launch exceeded expectations, with hundreds of games played each week, and the tool is positioned as an option alongside video conferencing rather than a replacement.
Director of Machine Learning Insights [Part 3: Finance Edition]
Machine learning leaders in financial institutions face significant operational hurdles when integrating automated models into production environments. Practitioners must navigate complex legacy architectures, strict regulatory oversight, and privacy mandates tied to personally identifiable financial records. Integrating models often fails when organizations lack clear communication buffers, deploy models using improper prediction windows, or treat systems as opaque black boxes without verifying underlying mechanics. Despite these challenges, financial applications increasingly rely on machine learning for anti-money laundering compliance, automated fraud screening, and loan underwriting. Success in this regulated sector demands comprehensive evaluation across representative input spaces, robust ongoing model monitoring, and explainable decision outputs.
Logbook - 2022-06-10
Fly.io’s June 10, 2022 logbook lists platform, dashboard, deployment, and account-management changes for its globally distributed application hosting service. WireGuard peer synchronization now adds only peers whose state changed, while removing a global lock from proxy tracing instrumentation improves TLS handshaking and HTTP parsing under load; the post says these changes should speed userspace features, avoid API timeouts, and reduce response times. Deployments gain build-time secrets through fly deploy’s --build-secret flag, larger rolling-deploy batches that are expected to make multi-region releases 2–5x faster, and restored Turboku webhook redeployment from Heroku. Dashboard updates include intelligent longpolling fallback, visible certificate deletion controls, monitoring links, safer app deletion confirmation, and corrected metrics and coupon visibility, while the Jupyter launcher now requires a credit card after abuse.
8 Data Conferences Shopify Data Thinks You Should Attend
Shopify’s data scientists and engineers curated eight upcoming data conferences for 2022, covering virtual, hybrid, and in-person formats. The selection supports learning and development by helping practitioners hear from peers about applications, techniques, use cases, and opportunities in the wider data community. Hybrid options include Data + AI Summit, Transform, RecSys, and ODSC West, with topics spanning analytics, machine learning, open source technologies, recommender systems, MLOps, natural language processing, and big data. In-person choices—PyData London, KDD, Re-Work Deep Learning Summit, and Crunch—feature tutorials, research, business applications, AI challenges, data-team practices, and machine learning at scale, with locations including London, Washington D.C., Toronto, and Budapest.
How facial recognition technology keeps you safe
Grab utilizes facial recognition technology across its platform for driver authentication, passenger verification, and digital electronic Know Your Customer (e-KYC) processes. The core pipeline consists of image preprocessing through face detection and alignment, anti-spoofing checks, feature extraction into high-dimensional vector embeddings, and downstream verification or search. To counter spoof attacks like screen replays, synthetic moiré patterns are generated and cropped face patches are used during training and inference to focus on local structures rather than global semantic noise. Face verification challenges involving shallow ID datasets and masked faces are resolved using semi-Siamese training architectures and masked data augmentations.
The Journey to Cloud Development: How Shopify Went All-in on Spin
Shopify’s development environment, the Ruby-based dev tool and xhyve-backed railgun VM, no longer handled the growing number of repositories, services, integrations, and resource demands around its monoliths. Developers needed more services, faced sluggish code-build-test loops, and encountered costly state changes when tophatting collaborators’ work. The Environments team concluded that development environments needed cloud-like elasticity, then tested an automatically configured local Kubernetes cluster and easy creation of Google Compute Engine VMs; the source also describes Isospin and Spin as the resulting direction. Local Kubernetes did not gain broad organic traction, while the current Isospin implementation is described as the team’s best iteration for developers working on majestic monoliths. The team plans to evolve it into an application-development library built on dev, add dependency inference, and address the tension between persistent environments and Kubernetes pod relocation.
Enhancing Atlas Online Archive With Data Expiration And Scheduled Archiving
Atlas Online Archive moves infrequently accessed data from a cluster to a MongoDB-managed object store while enabling unified queries across both tiers. A new data expiration feature lets users set a deletion threshold for archived data, configurable from 7 to 9,125 days through the Atlas UI or Admin API, and rules can be edited after creation; previously, the only way to remove data was deleting the entire archive. Once data is deleted from the archive, it cannot be recovered, so rules must be defined carefully. Scheduled archiving addresses performance concerns by allowing users to define a recurring window—daily, weekly, or monthly—with a minimum duration of two hours, replacing the previous five-minute interval that could overload clusters during peak times. Users can edit the archive rule to specify both when to archive and when to delete, and both features are in preview to improve operational efficiency for Atlas customers.
The Annotated Diffusion Model
Denoising Diffusion Probabilistic Models generate data samples by learning to reverse a discrete-time forward diffusion process that incrementally adds Gaussian noise to inputs. The forward process adds noise across a predefined schedule of variance parameters up to a fixed number of steps, transforming data into an isotropic Gaussian distribution. Because the true reverse conditional probability distribution is intractable, a neural network optimizes the variational lower bound to approximate the distribution mean at each step while keeping the variance fixed. Sampling subsequently proceeds by iteratively denoising pure Gaussian noise through successive network evaluations to reconstruct a clean data sample. Although diffusion models demonstrate high generation quality, their primary drawback remains the requirement of multiple forward passes during inference compared to models like generative adversarial networks.
Deep Q-Learning with Space Invaders
Tabular Q-learning fails to scale to complex environments like Atari games due to massive observation spaces that make maintaining discrete state-action tables impractical. Deep Q-learning addresses this limitation by using a neural network to approximate Q-values for each possible action from input states. To process visual inputs effectively and capture temporal motion, raw game screens are grayscaled, resized to 84x84 pixels, and stacked in groups of four consecutive frames before passing through convolutional and fully connected layers. The training process stabilizes learning through experience replay buffers, fixed target networks that update periodically, and Double DQN architectures that decouple action selection from target evaluation. Agents can then be trained on environments like Space Invaders using frameworks such as RL-Zoo without tabular memory constraints.
Graph concepts and applications
Real-world systems generate dynamic, non-random connections that traditional statistical approaches fail to characterize or forecast. Graph models represent these structures through vertices and edges, abstracting complex networks into mathematically tractable relationships. Common data representation formats include the Resource Description Framework (RDF), which models subject-predicate-object triples with IRIs, literals, and blank nodes, and Labeled Property Graphs (LPGs), which store arbitrary key-value properties directly on nodes and edges. Graph databases, derived from the LPG model, treat relationships with equal weight to entities, delivering responsive traversals for highly interconnected systems. While they offer agility and explicit relationship modeling, graph databases lack a standardized query language and remain poorly suited for standard transaction-focused workloads compared to relational databases.
Top topics
- Architecture830
- Performance812
- LLMs648
- Machine Learning617
- Open Source561
- AI Agents542
- AI454
- Developer Experience446
Weekly digest
The best engineering reads, once a week.