Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,580 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Should I Open Source my Company?
Software founders often hesitate to make their companies open source due to concerns regarding code scrutiny, security exploits, and intellectual property theft. Supabase operated as an open-source business from its inception, finding that community scrutiny actually accelerates code refactoring and surfaces security vulnerabilities early through public reporting channels. In addition, rapid execution and developer experience provide stronger competitive defenses against larger competitors than proprietary code obfuscation or software patents. Open repos also transformed the company's hiring strategy by replacing traditional take-home tests and LeetCode interviews with evaluation of real-world pull requests and asynchronous discussions. Finally, external contributors expand product capability by independently developing niche integrations and platform adapters that core startup teams lack the bandwidth to build.
Supabase Launch Week 4
Supabase announced the schedule and lineup for Launch Week 4, detailing daily product releases and events starting on March 28, 2022. The week initiated with Community Day to highlight community contributions and open-source tooling utilized across the platform. Technical additions across the week included the introduction of GraphQL data fetching for PostgreSQL databases, the rollout of Supabase Enterprise, and the launch of Edge Functions. Final announcements on April 1 revealed Supabrew alongside Supabase Realtime with multiplayer features, concluding with the start of a ten-day virtual hackathon running through April 9. For the hackathon, titled Bring the Func, Supabase partnered with the Future Forest Company to plant a tree for every submitted project on madewithsupabase.com.
How We Built the Add to Favorite Animation in Shop
The Shop app’s Add to Favorite interaction animates a product thumbnail into the Favorites tab bar heart after a buyer favorites a product. The implementation is based on a motion spec that defines animated properties, target values, durations, easing curves, and ordering, with the thumbnail sequence followed by the icon’s color and bouncing movement. In React Native, Reanimated uses one shared progress value for the whole sequence, and interpolation maps progress ranges to opacity, scale, and vertical translation, including clamping and an instant disappearance at step 3. A 50 ms pause is included between thumbnail scaling and its downward movement. The shared value also creates a match cut between the disappearing thumbnail and the icon; a setTimeout-based handoff was rejected because a busy JS thread could delay the transition, while withDelay would keep timing on the UI thread.
How telematics helps Grab to improve safety
Grab developed an in-house telematics engine to monitor driver-partner habits, detect crashes, and improve ride safety beyond passenger feedback. The mobile telematics SDK collects accelerometer, gyroscope, and GPS readings, running on-device processing for immediate accident detection while batching ride data for post-trip safety reports. Because triaxial sensor sampling rates vary independently across diverse devices, the SDK synchronizes data through interpolation to a uniform time grid followed by decimation to an output data rate. Telematics signals are transformed to the frequency domain using Fourier Transform for local compression before backend upload. Grab also combines telemetry with spatial data to identify dangerous road zones and plans to expand on-device event detection across all platform verticals.
A Data Scientist’s Guide To Measuring Product Success
The guide addresses the problem of turning product success after launch into concrete metrics that reflect user impact rather than model or pipeline performance. It recommends starting with end-user goals, defining a main product goal and subgoals, translating them into primary questions, and selecting metrics before building a measurement plan. In Shopify’s example, the contextual analytics experience surfaces inventory data, with adoption measured through shops interacting with its navigation bar and product goals progressing from data-aware to data-savvy merchants. Metrics should be simple, precise, feasible, sensitive, and directional, while tripwires or anti-success metrics identify unwanted outcomes alongside desired gains. The plan pairs metrics with comparison measures and benchmarks, supports segmentation by merchant characteristics, and separates short-term, mid-term, and long-term measurement periods so results can be interpreted after release.
Machine Learning Experts - Margaret Mitchell
Machine learning models often internalize harmful dataset skews and fail to evaluate real-world contexts accurately. During work on vision-to-language generation, visual recognition systems learned to misinterpret destructive disasters as positive scenes because training images predominantly featured sunsets and fireworks. Addressing these structural flaws requires moving beyond benchmark optimization toward critical data analysis, dataset genealogies, and standardized ethical AI protocols. ML organizations must establish shared vocabularies for power differentials and dismantle competitive cultural norms that marginalize underrepresented contributors. Furthermore, lowering technical barriers—such as enabling non-engineers to inspect data without SQL—allows diverse stakeholders to evaluate models and prevent widening societal power divides.
Faster Pull Request Merges
Ramp’s growing Python pytest suite took over 12 minutes, while pull requests could wait up to one to two hours between merge queueing and production. Previously, Kodiak kept pull requests in a merge queue, requiring each PR to be up-to-date with master and rerun the full suite after every merge. Removing that requirement exposed a failure mode: parallel branches could introduce Alembic migrations with multiple heads, making the database migration history invalid. Ramp instead added migrations/migration-hash.txt, a hash of migration filenames that creates merge conflicts between migration-changing PRs while leaving other PRs free to merge; the deploy pipeline still runs tests on master before deployment. This reduced average merge time from more than an hour to 12 minutes.
Shopify’s Bug Bounty Program Raises Maximum Payout in 2022
Shopify’s Application Security team reports on its 2021 bug bounty results and announces program changes for 2022. Effective immediately, the maximum bounty for a CVSS 10.0 issue doubles from $50,000 to $100,000, with higher High and Critical brackets, while Shopify Plus, Shop, and Shop Pay services move into the Core asset category; mobile-client-specific issues remain Non-Core. The program received more than 3,000 reports, paid over $1 million in bounties and bonuses, and saw approximately 23% of reports judged valid, up from 19% in 2020. After its first full year using CVSS-based scoring, Shopify reports more consistent decisions and a reduction in average time to bounty from 249 hours to 137, while adding a CVSS guide, more resources, and further open-source support.
Announcing the 🤗 AI Research Residency Program
Hugging Face launched a nine-month Research Residency Program designed to train participants into impactful machine learning researchers. Residents collaborate directly with researchers from the Science Team to formulate research questions, develop novel machine learning techniques, and openly publish their findings. The initiative specifically encourages proposals from marginalized groups—including women, LGBTQ+ individuals, people of color, and working-class communities—to address disparities exacerbated by artificial intelligence progress. Applicants must demonstrate mathematical and programming abilities through coursework, open-source projects, and end-to-end proposals targeting positive societal impact. The full-time role is fully remote, provides benefits including medical coverage depending on location, and excludes concurrent student enrollment or employment.
Using Terraform to Manage Infrastructure
Shopify’s Conversations team used Twilio TaskRouter to route voice and chat tasks to agents according to routing rules and skills. As routing needs grew, configuration through Twilio’s GUI became difficult to understand and lacked clear change history, tests, and rollback support. Terraform provided an infrastructure-as-code workflow, backed by a reliable API, a Go client library, and a provider implementing CRUD operations; the team built a provider because Twilio did not have one at the project’s start. Resource definitions included schemas, dependencies, and an importer, while Terraform’s plan and apply commands previewed and executed changes. Integrated with Atlantis and GitHub pull requests, the workflow added plans, reviews, revision history, and easier rollbacks, while enabling business and support teams to submit infrastructure changes themselves.
Fine-Tune a Semantic Segmentation Model with a Custom Dataset
Fine-tuning a semantic segmentation model requires domain-appropriate training data and an efficient pipeline. Existing autonomous driving datasets feature roadway imagery captured by cars, creating a distribution mismatch for sidewalk-based delivery robots. To resolve this discrepancy, a dedicated dataset of sidewalk imagery is loaded from the Hugging Face Hub, split into training and test sets, and augmented on-the-fly using SegformerImageProcessor and torchvision. The smallest SegFormer architecture, B0, is fine-tuned using Hugging Face's Trainer API with mean Intersection over Union evaluation metrics. The final pipeline pushes the fine-tuned model to the Hub and executes inference by upsampling output logits to original image dimensions.
Accelerate BERT inference with Hugging Face Transformers and AWS Inferentia
Production deployments of BERT and Transformer architectures often face cost and latency challenges because these models are significantly larger and more computationally intensive than traditional algorithms. To optimize text classification workloads, developers can compile vanilla PyTorch models for AWS Inferentia using the AWS Neuron SDK and its tracing utilities. Because the Neuron SDK requires static tensor dimensions, the model is traced with fixed input lengths and packaged alongside a custom inference script configuring one Neuron Core per worker. Deploying the resulting artifacts to an Amazon SageMaker ml.inf1.xlarge endpoint yields an average latency of 5 to 6 milliseconds for a sequence length of 128 across 10,000 synchronous evaluation requests.
Image search with 🤗 datasets
Hugging Face datasets expanded its capabilities by introducing an Image feature type, enabling image processing and integration with vector indexing tools. The library was applied to a sample of historical book embellishments extracted via OCR from the British Library. Images were loaded using the ImageFolder loader, enriched with filename metadata, and pushed to the Hugging Face Hub. A FAISS index and CLIP embeddings were used to retrieve images matching natural language queries, such as categories, specific objects, and boolean operators. While the retrieval system demonstrated reasonable semantic search results across various prompts, full public deployment was avoided due to CLIP model card restrictions and potential bias in the historical dataset.
Creating a React Library for Consistent Data Visualization
Shopify found that teams across its more than 10,000 employees used different visualization tools, producing inconsistent chart styling and differing accessibility and print behavior. The Insights team built Polaris Viz, a React library intended to provide shared, accessible defaults while reducing repeated configuration, and focused this work on light and dark themes plus consumer-defined visual identities. The library replaces numerous visual-style props with a theme prop, stores themes through PolarisVizProvider, and uses createTheme to fill partial definitions with defaults and createThemes to preserve Default and Light themes. Themes can inherit from other themes, while a DataSeries colour can override theme series colors for dataset-specific gradients without creating additional themes. Polaris Viz was subsequently released as an open-source project.
Real-time data ingestion in Grab
Service teams at Grab historically had to dual-write transactional data into databases and Kafka, creating data integrity issues during transaction failures alongside substantial schema maintenance overhead. To overcome these limitations and eliminate burst reads from SQL-based queries, the Caspian team built a real-time ingestion platform synchronising MySQL, Aurora, and DynamoDB directly to Kafka. For MySQL and Aurora, the platform uses Debezium with Kafka Connect on ROW-format binlogs, while DynamoDB changes are captured via DynamoDB streams with auto-scaling AWS Lambda functions. Messages encoded in Protobuf are transported via Kafka and ingested into Amazon S3 using a Golang stream processor. This architecture supports search indexing in Elasticsearch, automated data lake pipelines, cross-region disaster recovery replication, and audit trails.
Guiding Text Generation with Constrained Beam Search in 🤗 Transformers
Constrained beam search introduces direct control over generated text in Hugging Face Transformers. Standard beam search operates token-by-token without knowing the optimal step to force specific words or phrases, making it difficult to enforce mandatory vocabulary or choose between alternative expressions. To resolve this, the generate interface accepts constraints through arguments such as force_words_ids and a list of Constraint subclasses like PhrasalConstraint. This mechanism enables disjunctive constraints, where generation must include at least one phrase from a provided set, alongside strictly required sequences. Consequently, practitioners can inject prior knowledge or formatting requirements directly at generation time rather than filtering candidate outputs afterward.
After the Refactor: A Path to Faster Rendering with Liquid-C
Liquid-C is an extension to the Liquid gem that parses, tokenizes, and renders templates, but some Liquid features, including If and For tags, still render in Liquid rather than Liquid-C. The work explains how Liquid-C compiles templates into bytecode for a customer VM and addresses the difficulty of moving instruction and constant pointers when control flow requires forward or backward jumps. It refactors instructions to store each constant’s index in the constants array, using a hash table to avoid duplicate constants and remove the constant pointer. The change reduced one example constants array from 157 objects to 104, but did not drastically improve theme rendering speed; it slightly increased memory usage and parsing time while preparing for GOTO or JMP operations and later If and For support.
Postgres Auditing in 150 lines of SQL
Traditional PostgreSQL auditing methods duplicate source table structures, creating maintenance overhead when schemas change. To solve this, a compact SQL implementation stores insert, update, and delete events across multiple tables in a single audit table using JSONB columns. Query performance is maintained by using a BRIN index on naturally ordered insertion timestamps, a B-tree index on internal table OIDs, and UUIDv5 identifiers hashed from primary keys. Row-level PL/pgSQL triggers automatically populate the audit log and can be enabled or disabled dynamically with dedicated tracking functions. While trigger-based auditing introduces minimal overhead at rates below 1,000 writes per second, write-heavy workloads may benefit from logging changes outside SQL with tools like pgAudit.
7 Tips For Optimizing Apache Flink Applications
Shopify presents lessons from operating large, stateful Apache Flink applications on Kubernetes with Google Kubernetes Engine, RocksDB, and checkpoints and savepoints stored in Google Cloud Storage. The guidance begins with profiling tools, including async-profiler, VisualVM, jemalloc with jeprof, and Eclipse Memory Analyzer, to investigate CPU, heap, native-memory, and leak-related problems. Disabling Kryo fallback exposed unsupported serializers such as Scala BigDecimal and Scala algebraic data types; replacing them with supported alternatives increased throughput by 20 percent. The post also recommends separate configuration profiles for high-throughput backfills and low-latency steady state, while describing a RocksDB native-memory failure in which disabling the block cache stopped repeated out-of-memory crashes without affecting application performance.
Test Budget: Time Constrained CI Feedback
Shopify’s core monolith contains more than 170,000 tests, and variable CI feedback times create delays and context switches as the test pool grows. The Test Infrastructure team evaluated a Test Budget approach that builds prioritized orders on top of deterministic test selection, then stops execution at a predetermined time to increase the chance of finding failures early while accepting some risk. Six criteria were compared—failure_rate, avg_duration, churn, coverage, complexity, and default/random—using Time to First Failure, APFD, and Convergence Index; historical results were stored and queried with Redis Sorted Sets. The analysis found failure_rate had the best detection rate, while churn and complexity were inappropriate; running 60% of the suite detected 80% of failures in the mean case, and 70% of the test-selection suite detected 50% at the fifth percentile.
Top topics
- Architecture830
- Performance813
- LLMs656
- Machine Learning619
- Open Source563
- AI Agents546
- AI461
- Developer Experience447
Weekly digest
The best engineering reads, once a week.