--- title: "Latest reads" description: "The engineering internet, summarised so you can actually read it." --- # Latest reads > The engineering internet, summarised so you can actually read it. ## Articles ### [Mitigating Deadlocks in high concurrency environments](https://yomu.fyi/post/mitigating-deadlocks-in-high-concurrency-environments.md) - Company: [Shopify](https://yomu.fyi/company/shopify.md) - Author: 2023-10-18 - Published: Aug 14, 2024 - Type: Problem & solution MySQL requires locks before updating existing records, and concurrent upserts that mix new and existing rows can deadlock when gap locks overlap, even when processes target different records. The post describes ETL jobs partitioned by account whose updates nevertheless contend because sequential primary-key ordering places records in shared index gaps. It proposes replacing the existing primary key with a composite key of account and id, aligning InnoDB's physical row ordering with the processes' update partitions; gap locks remain necessary, but are largely compartmentalized within each process. An additional unique index on id preserves standalone record identification and compatibility with existing joins, while application and ETL logic remain unchanged. The reported outcome was removal of artificial delays and significantly higher import throughput through reduced job wait time. ### [Improving Retrieval on Ramp with Transaction Embeddings](https://yomu.fyi/post/improving-retrieval-on-ramp-with-transaction-embeddings.md) - Company: [Ramp](https://yomu.fyi/company/ramp.md) - Author: Calix Huang, Anton Biryukov - Published: Aug 13, 2024 - Type: Problem & solution Ramp developed transaction embeddings to help automate accounting coding, where employees choose general ledger (GL) categories for transactions. The model represents enriched transaction features—such as merchant, category, department, location, amount, memo, spend program, and trip—as stringified documents labeled with chart-of-accounts codes. Starting from a pre-trained encoder, it is fine-tuned with sentence-transformers and BatchSemiHardTripletLoss, using large batches and contextual sampling to learn embeddings whose similarity is measured with cosine similarity. Triplet loss pulls same-label transactions together and pushes different-label examples apart, while informed mining addresses subtle distinctions such as Travel: Sales versus Travel: Engineering. The embeddings support GL coding suggestions, high-confidence defaults, business-spend similarity analysis, and transaction context for LLM-enabled features, with the stated aim of keeping predictions personalized and data private. ### [eBay Introduces Intuitive Search Redesign to Elevate Shopper Experience](https://yomu.fyi/post/ebay-introduces-intuitive-search-redesign-to-elevate-shopper-experienc.md) - Company: [Ebay](https://yomu.fyi/company/ebay.md) - Author: Prerna Dubey, Avital Derazon, Harshal Parikh and Kimi Robertson - Published: Aug 8, 2024 - Type: Announcement eBay redesigned its search experience after 18 months of user research and testing, targeting a more seamless, intuitive, and visually rich shopping journey. The update uses larger high-resolution listing images, a modernized information layout, redesigned navigation menus, and streamlined filters, with particular attention to visual categories such as Clothing, Shoes & Accessories, Jewelry & Watches, and Toys & Hobbies. A new Shopping View replaces the previous gallery view with a full-width display without sidebar ads, while delivery options are consolidated and the price filter adds an interactive graph showing price distribution. eBay says it will continue using artificial intelligence and machine learning to refine relevance, listing benefits, and seller details, with the stated aim of improving efficiency and trust for buyers and sellers. ### [Machine Learning in Content Moderation at Etsy](https://yomu.fyi/post/machine-learning-in-content-moderation-at-etsy.md) - Company: [Etsy](https://yomu.fyi/company/etsy.md) - Author: David Azcona - Published: Aug 6, 2024 - Type: Explainer Etsy describes a supervised-learning system for detecting potentially violating listings at marketplace scale, alongside community reporting and other automated controls. Training data combines logged and annotated positives, hard negatives, and randomly sampled easy negatives, with multimodal textual and imagery signals split by time for progressive evaluation. The architecture uses BERT-based text encoding with ALBERT for reduced compute, EfficientNet image encoding, late concatenation of modality representations, a softmax output, and focal loss for class imbalance. Etsy evaluates candidates offline and in production through API-serving, canary releases, and champion-challenger A/B tests, while using proxy recall because full manual review is infeasible. Frequent retraining and monitoring support ongoing iteration; the methodology has identified and removed more than 100,000 violations, alongside other tools and services. ### [How we improved translation experience with cost efficiency](https://yomu.fyi/post/how-we-improved-translation-experience-with-cost-efficiency.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Jie Zhang - Published: Aug 5, 2024 - Type: Problem & solution Grab faced translation quality and cost challenges in its booking chat system when tourism resumed after COVID restrictions eased. Device language settings proved unreliable for determining message language, prompting the team to use Lingua alongside booking context heuristics and third-party fallbacks for real-time detection. To replace expensive and inaccurate general translation APIs, Grab distilled a large open-source Hugging Face model into lightweight, language-specific models trained on ten million synthetic chat examples. A post-translation validation step verifies non-translatable entities such as numbers and emojis before falling back to external services if mutations occur. Caching layers were also introduced across translation paths to curb redundant on-the-fly execution. ### [Shipping support for module script integrity in Chrome & Safari](https://yomu.fyi/post/shipping-support-for-module-script-integrity-in-chrome-safari.md) - Company: [Shopify](https://yomu.fyi/company/shopify.md) - Author: 2023-10-18 - Published: Aug 2, 2024 - Type: Announcement Shopify’s checkout is a managed service requiring control over scripts to support high-performance, PCIv4 DSS-compliant checkout and prevent tampering. SRI could validate top-level resources but did not cover imported JavaScript modules, forcing a shim for dynamic imports that added maintenance and runtime overhead. The work revived Guy Bedford’s proposal to put integrity hashes in import maps, enabling unchanged dynamic imports while also covering static imports, top-level module scripts, and modulepreload. Support shipped in Chromium 127 for Chrome and Edge, and landed in Safari 18’s beta release, with just over two weeks of full-time work and about seven chronological weeks overall. ### [Making Machines Move](https://yomu.fyi/post/making-machines-move.md) - Company: [Fly.io](https://yomu.fyi/company/fly-io.md) - Published: Jul 30, 2024 - Type: Problem & solution Fly.io needed to restore the ability to drain physical workers after attached NVMe volumes made stateful Fly Machines tied to particular servers. Snapshot restoration could lose data, while killing a machine before copying its volume caused unacceptable downtime; copying first risked losing writes. The solution was an asynchronous clone operation built around Linux device mapper’s dm-clone: a new volume is created immediately, reads of unhydrated blocks are fetched from the original over iSCSI, writes go to the destination, and background copying rehydrates the rest. The work also required orchestration across flyd and a fix for Fly Postgres configurations containing literal 6PN IPv6 addresses. After cautious early migrations, the infrastructure team could drain hosts without much ceremony by summer 2024, though routine automated rebalancing remained a future goal. ### [Mastering API Gateway Integration: Salesforce, Heroku, and MuleSoft Anypoint Flex Gateway](https://yomu.fyi/post/mastering-api-gateway-integration-salesforce-heroku-and-mulesoft-anypo.md) - Company: [Heroku](https://yomu.fyi/company/heroku.md) - Author: Julián Duque - Published: Jul 29, 2024 - Type: Tutorial MuleSoft Anypoint Flex Gateway provides a pattern for exposing APIs and microservices running in Heroku Private Spaces while adding API management and security controls. The tutorial deploys an internal Heroku application, creates or imports an OpenAPI 3.0 specification in Anypoint Design Center, publishes it to Exchange, registers the gateway through Runtime Manager, and deploys it in the same Private Space. That placement lets internal-routing applications receive public traffic through the gateway while routing access through a centrally configured layer managed with MuleSoft and Heroku. It then applies a Client ID Enforcement policy in API Manager, showing that requests with incorrect credentials fail and requests using the issued Client ID and Client Secret can authenticate. The pattern is presented as a foundation for Heroku and Salesforce multi-cloud integrations. ### [Heroku CLI v9: Infrastructure Upgrades and oclif Transition](https://yomu.fyi/post/heroku-cli-v9-infrastructure-upgrades-and-oclif-transition.md) - Company: [Heroku](https://yomu.fyi/company/heroku.md) - Author: Anush DSouza - Published: Jul 25, 2024 - Type: Announcement Heroku CLI 9.0.0 is an architectural upgrade centered on moving all core commands from a partly legacy structure to oclif. Core commands are consolidated into one package, while improved unit and integration coverage, telemetry, and a simpler automated release process support contribution, maintenance, and pre-release testing. The oclif migration provides a modular command architecture with plugin support and is intended to make command behavior more consistent, maintenance easier, and provide better scalability. Users should expect breaking changes including altered whitespace, newlines, table formatting, colors, argument ordering, flag removals, and a required separator for additional flags; deprecated commands including outbound-rules, pg:repoint, orgs:default, certs:chain, certs:key were removed. Changes apply to core commands, not separately installed plugins, and the release can be installed with heroku update. ### [How to Use pnpm on Heroku](https://yomu.fyi/post/how-to-use-pnpm-on-heroku.md) - Company: [Heroku](https://yomu.fyi/company/heroku.md) - Author: Colin Casey - Published: Jul 19, 2024 - Type: Tutorial Heroku’s Node.js buildpack supports pnpm, a fast, disk-efficient alternative to npm, and early users reported 10–40% faster install times on Heroku deployments. pnpm uses a content-addressable store and symlinks to avoid duplicate packages, accelerates dependency installation by combining resolve, fetch, and linking, and enforces dependency boundaries. The tutorial uses Node.js v18 or higher and Corepack to initialize and pin pnpm 9, then creates an Express application whose pnpm-lock.yaml enables buildpack support and consistent package resolution. The demo server handles HTTP GET requests with “Hello pnpm!” and listens on process.env.PORT or defaults to 3000. It finishes by adding a Procfile, committing the project to Git, creating a Heroku app, and deploying with git push heroku main. ### [LLM-powered data classification for data entities at scale](https://yomu.fyi/post/llm-powered-data-classification-for-data-entities-at-scale.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Hualin Liu - Published: Jul 15, 2024 - Type: Problem & solution Grab needed to classify sensitive data at the table and column level across petabytes of database tables and streaming schemas. Manual schema-tiering campaigns had resulted in half of all schemas receiving overly strict Tier 1 access controls, while an initial automated service using regex patterns and third-party machine learning produced high false-positive rates and lacked customizability. To address this, the Caspian data engineering and governance teams enhanced their internal orchestration service, Gemini, by integrating GPT-3.5 via Azure OpenAI. Gemini aggregates classification requests into mini-batches, handles API rate limits, and uses prompt engineering—including few-shot examples, curated tag libraries, and explicit JSON DTO schemas—to reliably tag columns for data owner verification. ### [eBay Advertising Unveils New Dashboard, Personalized Insights, and Intuitive Campaign Tools](https://yomu.fyi/post/ebay-advertising-unveils-new-dashboard-personalized-insights-and-intui.md) - Company: [Ebay](https://yomu.fyi/company/ebay.md) - Author: eBay News Team - Published: Jul 8, 2024 - Type: Announcement eBay Advertising has introduced a redesigned experience centered on a new dashboard, a unified portfolio, and tools intended to simplify campaign management, increase listing visibility, and support seller growth. The dashboard, available through a new Seller Hub Advertising tab, combines campaign reporting with personalized recommendations, daily optimization tips, suggested campaigns based on historical data and best practices, and trend-based campaigns built using ebay.ai. Sellers can choose among Promoted Listings, Promoted Stores, and Promoted Offsite, with Promoted Listings offering priority or general strategies that differ in placement access, controls, and charging models. The announcement reports average gains of 50% in items sold for priority campaigns, 30% more sales for general-strategy listings, and 25% more sales from external channels with Promoted Offsite, while positioning real-time data and actionable insights as tools for adjusting advertising decisions. ### [Office Hours with Engineering Managing Director Mae Santos](https://yomu.fyi/post/office-hours-with-engineering-managing-director-mae-santos.md) - Company: [Two Sigma](https://yomu.fyi/company/two-sigma.md) - Author: Emily Majewski - Published: Jun 26, 2024 - Type: Explainer Mae Santos, Head of Critical Applications and Data Reliability Engineering at Two Sigma, leads a global team across New York, Houston, London, and Tokyo to support critical applications and data pipelines. While software teams frequently prioritize functional features over operational stability, designing systems with reliability in mind from the beginning allows for progressive enhancements without complete rewrites. System architecture should be guided by concrete goals across four key dimensions: availability, observability, scalability, and supportability. In addition, mapping out interconnections among systems, infrastructure, data, and human processes helps teams manage dependencies effectively. Applying automation across support and delivery workflows further reinforces reliability by minimizing human error and accelerating execution. ### [Optimizing Data Reliability: Heroku Connect & Drift Detection](https://yomu.fyi/post/optimizing-data-reliability-heroku-connect-drift-detection.md) - Company: [Heroku](https://yomu.fyi/company/heroku.md) - Author: Siraj Ghaffar - Published: Jun 26, 2024 - Type: Explainer Heroku Connect syncs Salesforce data with Heroku Postgres, but primary polling can miss changes when long-running Salesforce automations commit after the polling window advances, creating drift. The service uses secondary polls over the same bounds as primary polls to find and repair missed changes, with Bulk API or SOAP API selected according to dataset size and retries for failed polls. Recent improvements change Bulk API drift handling by creating one secondary bulk job for each primary bulk job, then adding corresponding batches instead of creating many jobs. Secondary tasks are now scheduled and executed with limited concurrency, reducing simultaneous database connections consumed while Postgres is updated and preserving capacity for applications. The post says these changes reduce load on Salesforce and improve operational reliability during heavy synchronization, while noting that Heroku Connect remains asynchronous rather than truly real-time. ### [AWS without Access Keys](https://yomu.fyi/post/aws-without-access-keys.md) - Company: [Fly.io](https://yomu.fyi/company/fly-io.md) - Published: Jun 19, 2024 - Type: Problem & solution The post describes how Fly Machines can access AWS resources such as an S3 bucket without distributing long-lived AWS access keys. It uses OpenID Connect (OIDC): AWS IAM trusts Fly.io’s OIDC identity provider, a role grants narrowly scoped permissions, and the app receives the role ARN as an environment variable. At boot, Fly’s Rust init requests a machine-specific OIDC token through the /.fly/api Unix socket, writes it to /.fly/oidc\_token, and sets AWS\_WEB\_IDENTITY\_TOKEN\_FILE and AWS\_ROLE\_SESSION\_NAME so the AWS SDK can call STS AssumeRoleWithWebIdentity. AWS verifies the token, issues short-lived credentials, and evaluates the role’s IAM policy before serving the requested object. The trust policy can constrain tokens by audience and subject, including an organization, Fly App, or machine; the same pattern can authenticate against other OIDC-compliant providers, although Azure and GCP lack equivalent init integration. ### [Enhancing Cloud Usage Forecasting, Monitoring & Optimizing](https://yomu.fyi/post/enhancing-cloud-usage-forecasting-monitoring-optimizing.md) - Company: [Etsy](https://yomu.fyi/company/etsy.md) - Author: Anthony Tambasco - Published: Jun 17, 2024 - Type: Explainer After Etsy completed its 2020 migration from an on-premise data center to Google Cloud Platform (GCP), the resulting FinOps team developed a global approach to forecasting, monitoring, and optimizing cloud spend. Costs are grouped into core infrastructure, machine learning and product enablement, data enablement, and Dev, with trailing Cost Per Visit (CPV) serving as the primary forecasting metric after regression tests found visits most correlated with costs. BigQuery and Looker dashboards, alerts, weekly Finance comparisons, and recurring cost-owner and leadership readouts support variance detection, while costs are tracked centrally without chargebacks to preserve flexibility. Optimization work combines FinOps, engineering, and Architecture collaboration, producing GCS storage reductions through monitoring, automation, TTLs, and Intelligent tiering, plus compute, model-deployment, and network compression improvements. The next priority is better visibility into shared machine learning platform spend to improve experiment ROI analysis and identify optimization opportunities. ### [eBay’s Christine Su Named to Fast Company’s 2024 Queer 50 List](https://yomu.fyi/post/ebay-s-christine-su-named-to-fast-company-s-2024-queer-50-list.md) - Company: [Ebay](https://yomu.fyi/company/ebay.md) - Author: eBay News Team - Published: Jun 14, 2024 - Type: Announcement eBay Head of Product Christine Su has been named to Fast Company’s 2024 Queer 50 list for the second consecutive year. The list recognizes LGBTQ leaders across industries, while Su’s role covers product work for AI Recommendations, Homepage, and My eBay. She describes using AI to personalize the eBay experience for buyers and sellers, aiming to connect people with products and communities aligned with their passions. Since her first inclusion in 2023, eBay has added AI-powered features including Shop the look, an immersive carousel that curates complementary fashion looks from a buyer’s past shopping history for fashion discovery within the marketplace. Su says the intended experience should feel personal, support discovery, and connect users with people who share their passions. ### [Make Better Decisions by Embracing Uncertainty](https://yomu.fyi/post/make-better-decisions-by-embracing-uncertainty.md) - Company: [Ramp](https://yomu.fyi/company/ramp.md) - Author: Ryne Carbone - Published: Jun 11, 2024 - Type: Problem & solution Ramp uses multi-armed bandits to choose among third-party bank-linking providers when provider performance varies by financial institution and available data is noisy, incomplete, or stale. Instead of maintaining separate, static A/B tests, it applies Thompson Sampling, modeling each provider’s success probability with a Beta distribution and repeatedly sampling, routing traffic to the highest sampled value, then updating the model from the result. Simulations examined changing success-rate trends, noise, volume, rolling-window size, trend type, and initial conditions; the algorithm generally exploited clearly better providers, split traffic more evenly when rates were close, and recovered eventually from incorrect starting conditions. In a sequential quasi-experiment, Thompson Sampling increased financial-institution linking success by 10% and reduced customers with manually uploaded bank statements by 25%. ### [Efficient Visual Representation Learning And Evaluation](https://yomu.fyi/post/efficient-visual-representation-learning-and-evaluation.md) - Company: [Etsy](https://yomu.fyi/company/etsy.md) - Author: Eden Dolev - Published: Jun 7, 2024 - Type: Explainer Etsy uses image vector representations for visual search and similar-item recommendations, making low-latency, memory-aware models important for online query inference and downstream retrieval. The work compares EfficientNetB0, a compact CNN, with EfficientFormer-l3, which downsamples multiple blocks and applies attention only in its final stage; its representation uses the last block’s eight-head attention and average-pools the sequence. Fine-tuning adds an embedding and multiple classification heads, while multitask training supervises taxonomy, color, and review-photo labels; both models train on two 16GB P100 GPUs. Evaluation rebuilds a brute-force candidate index after each epoch and measures Recall@5 and Recall@10 on interaction and intra-item retrieval datasets, with early stopping available. EfficientFormer-l3 improves reported offline recall by about 5%, 17%, and 1.8% across three tasks, while experiments report lifts in CTR, purchase rate, click-recall@100, and clicks. ### [Profile-guided optimisation (PGO) on Grab services](https://yomu.fyi/post/profile-guided-optimisation-pgo-on-grab-services.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Yonghao Hu - Published: Jun 5, 2024 - Type: Benchmark Profile-guided optimisation (PGO) improves Go application performance by feeding production CPU profile data into subsequent compiler builds. Teams evaluated PGO across multiple Grab services, including TalariaDB and an orchestrator within the Catwalk service, using Go 1.20 and Go 1.21 builds. TalariaDB saw substantial resource savings after compiling Docker images with a 360-second pprof profile, resulting in at least a 10% drop in CPU usage, a 30% drop in memory consumption, and a 38% reduction in persistent volume usage for event ingestion queues. Conversely, testing on Catwalk showed only a 5% performance gain after adjusting the profiling duration from 59 seconds to 6 minutes, an outcome deemed insufficient to justify pipeline modifications. Broader adoption across monorepo services remains constrained until the central build process supports PGO arguments and automated profile retrieval. [Newer posts](https://yomu.fyi/posts/page/186.md) · [Older posts](https://yomu.fyi/posts/page/188.md)