Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,668 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
What is Data Residency? Data Residency Concerns for Global Applications
Global applications can move customer data across borders, making data residency a compliance concern for organizations serving multinational users. The post defines data residency as legal requirements governing where data may be stored and processed, then distinguishes it from data sovereignty’s jurisdictional reach and data localization’s requirement that storage and processing remain within a jurisdiction. It notes that GDPR and CCPA focus on compliant handling and individual rights rather than necessarily mandating storage locations, while China’s Personal Information Protection Law requires certain Chinese citizens’ PII to be stored within China. The proposed approach uses cloud regions to align storage with local requirements, citing Heroku Enterprise Private Dynos across ten regions and Heroku Shield Private Spaces’ PCI, HIPAA, ISO, and SOC certifications; the post concludes that these features can make compliance more achievable while improving control and potentially application performance.
Accident Forgiveness
Public cloud billing can turn accidental deployments, configuration errors, runaway VM allocation, or leaked CI/CD environments into unexpectedly enormous charges, making teams hesitant to experiment. Fly.io announces an initial “Accident Forgiveness” policy: customers with any support contract can request refunds for unexpected charges, almost without questions, while the company reserves the right to cut off abuse. The longer-term goal is an automated billing system with line-item vetoes, but fraud—especially cryptocurrency mining—requires an incremental rollout; the first version uses manual follow-up reminders to investigate incidents. A rebuilt billing system is intended to support future accident protection, including detecting month-over-month spikes, flagging unusual deployments before billing, and identifying opportunities for reserved pricing and backdated reservations.
Approaches to keep sending OTP over SMS... for now
SMS and voice OTPs remain broadly applicable for identity verification, but the post argues they should eventually be replaced because SIM swapping, phishing, uncontrolled secret-delivery channels, device-linking gaps, and slower logins weaken them. Until then, Okta will require organizations to bring their own telephony provider, using a telephony inline hook to send codes and report outcomes. The implementation can be built in-house, outsourced, or delegated to a managed service, with regional regulations, traffic peaks, reliability, branding, security, and available resources shaping the choice. A hook has three seconds to return success or failure before Okta’s heavily rate-limited failover sends the message; teams must also secure endpoints and API keys, monitor logs, and handle vendor or delivery failures.
Building an Event-Driven Architecture with Managed Data Services
Event-driven architecture (EDA) addresses the need to process and respond to continuous streams of real-time activity from applications, e-commerce systems, wearables, and other IoT sensors. Using Apache Kafka, independent producers publish events to topics while consumers subscribe and trigger downstream processes, keeping services decoupled and independently scalable. Kafka also distributes large data streams in real time and replicates data across synchronized broker nodes, so a node failure does not lose data or interrupt operation. Apache Kafka on Heroku is presented as a fully managed service, with multi-tenant basic and single-tenant private plans, automatic broker scaling, and Standard-tier-and-above configurations starting with three brokers and extending to eight. Recommended integration practices include mapping producer-consumer flows, using transactions, topic and schema management, and delivery guarantees, plus monitoring performance and logging Kafka operations.
We're Cutting L40S Prices In Half
Fly.io lowered the hourly price of NVIDIA L40S GPUs to $1.25, making them available at the same price as A10s after customer data showed that A10s—not the more powerful cards—were its most popular GPU. The post argues that most users adding GPUs to fast networks are running inference rather than training, making networking, memory, object storage, and request latency more important than access to an H100. The L40S is described as an AI-optimized L40 with AI compute performance comparable to an A100, while the L40 retains graphics hardware and is designed for rack use with more memory and lower energy consumption than gaming GPUs. The intended combination places model parameters, datasets, and compute near one another on Fly.io’s Anycast network and Tigris object storage. Examples include Llama 3.1 70B, Flux, Whisper, SegAlign, and DOOM Eternal.
Mitigating Deadlocks in high concurrency environments
MySQL requires locks before updating existing records, and concurrent upserts that mix new and existing rows can deadlock when gap locks overlap, even when processes target different records. The post describes ETL jobs partitioned by account whose updates nevertheless contend because sequential primary-key ordering places records in shared index gaps. It proposes replacing the existing primary key with a composite key of account and id, aligning InnoDB's physical row ordering with the processes' update partitions; gap locks remain necessary, but are largely compartmentalized within each process. An additional unique index on id preserves standalone record identification and compatibility with existing joins, while application and ETL logic remain unchanged. The reported outcome was removal of artificial delays and significantly higher import throughput through reduced job wait time.
Improving Retrieval on Ramp with Transaction Embeddings
Ramp developed transaction embeddings to help automate accounting coding, where employees choose general ledger (GL) categories for transactions. The model represents enriched transaction features—such as merchant, category, department, location, amount, memo, spend program, and trip—as stringified documents labeled with chart-of-accounts codes. Starting from a pre-trained encoder, it is fine-tuned with sentence-transformers and BatchSemiHardTripletLoss, using large batches and contextual sampling to learn embeddings whose similarity is measured with cosine similarity. Triplet loss pulls same-label transactions together and pushes different-label examples apart, while informed mining addresses subtle distinctions such as Travel: Sales versus Travel: Engineering. The embeddings support GL coding suggestions, high-confidence defaults, business-spend similarity analysis, and transaction context for LLM-enabled features, with the stated aim of keeping predictions personalized and data private.
eBay Introduces Intuitive Search Redesign to Elevate Shopper Experience
eBay redesigned its search experience after 18 months of user research and testing, targeting a more seamless, intuitive, and visually rich shopping journey. The update uses larger high-resolution listing images, a modernized information layout, redesigned navigation menus, and streamlined filters, with particular attention to visual categories such as Clothing, Shoes & Accessories, Jewelry & Watches, and Toys & Hobbies. A new Shopping View replaces the previous gallery view with a full-width display without sidebar ads, while delivery options are consolidated and the price filter adds an interactive graph showing price distribution. eBay says it will continue using artificial intelligence and machine learning to refine relevance, listing benefits, and seller details, with the stated aim of improving efficiency and trust for buyers and sellers.
Machine Learning in Content Moderation at Etsy
Etsy describes a supervised-learning system for detecting potentially violating listings at marketplace scale, alongside community reporting and other automated controls. Training data combines logged and annotated positives, hard negatives, and randomly sampled easy negatives, with multimodal textual and imagery signals split by time for progressive evaluation. The architecture uses BERT-based text encoding with ALBERT for reduced compute, EfficientNet image encoding, late concatenation of modality representations, a softmax output, and focal loss for class imbalance. Etsy evaluates candidates offline and in production through API-serving, canary releases, and champion-challenger A/B tests, while using proxy recall because full manual review is infeasible. Frequent retraining and monitoring support ongoing iteration; the methodology has identified and removed more than 100,000 violations, alongside other tools and services.
How we improved translation experience with cost efficiency
Grab faced translation quality and cost challenges in its booking chat system when tourism resumed after COVID restrictions eased. Device language settings proved unreliable for determining message language, prompting the team to use Lingua alongside booking context heuristics and third-party fallbacks for real-time detection. To replace expensive and inaccurate general translation APIs, Grab distilled a large open-source Hugging Face model into lightweight, language-specific models trained on ten million synthetic chat examples. A post-translation validation step verifies non-translatable entities such as numbers and emojis before falling back to external services if mutations occur. Caching layers were also introduced across translation paths to curb redundant on-the-fly execution.
Shipping support for module script integrity in Chrome & Safari
Shopify’s checkout is a managed service requiring control over scripts to support high-performance, PCIv4 DSS-compliant checkout and prevent tampering. SRI could validate top-level resources but did not cover imported JavaScript modules, forcing a shim for dynamic imports that added maintenance and runtime overhead. The work revived Guy Bedford’s proposal to put integrity hashes in import maps, enabling unchanged dynamic imports while also covering static imports, top-level module scripts, and modulepreload. Support shipped in Chromium 127 for Chrome and Edge, and landed in Safari 18’s beta release, with just over two weeks of full-time work and about seven chronological weeks overall.
Making Machines Move
Fly.io needed to restore the ability to drain physical workers after attached NVMe volumes made stateful Fly Machines tied to particular servers. Snapshot restoration could lose data, while killing a machine before copying its volume caused unacceptable downtime; copying first risked losing writes. The solution was an asynchronous clone operation built around Linux device mapper’s dm-clone: a new volume is created immediately, reads of unhydrated blocks are fetched from the original over iSCSI, writes go to the destination, and background copying rehydrates the rest. The work also required orchestration across flyd and a fix for Fly Postgres configurations containing literal 6PN IPv6 addresses. After cautious early migrations, the infrastructure team could drain hosts without much ceremony by summer 2024, though routine automated rebalancing remained a future goal.
Mastering API Gateway Integration: Salesforce, Heroku, and MuleSoft Anypoint Flex Gateway
MuleSoft Anypoint Flex Gateway provides a pattern for exposing APIs and microservices running in Heroku Private Spaces while adding API management and security controls. The tutorial deploys an internal Heroku application, creates or imports an OpenAPI 3.0 specification in Anypoint Design Center, publishes it to Exchange, registers the gateway through Runtime Manager, and deploys it in the same Private Space. That placement lets internal-routing applications receive public traffic through the gateway while routing access through a centrally configured layer managed with MuleSoft and Heroku. It then applies a Client ID Enforcement policy in API Manager, showing that requests with incorrect credentials fail and requests using the issued Client ID and Client Secret can authenticate. The pattern is presented as a foundation for Heroku and Salesforce multi-cloud integrations.
Heroku CLI v9: Infrastructure Upgrades and oclif Transition
Heroku CLI 9.0.0 is an architectural upgrade centered on moving all core commands from a partly legacy structure to oclif. Core commands are consolidated into one package, while improved unit and integration coverage, telemetry, and a simpler automated release process support contribution, maintenance, and pre-release testing. The oclif migration provides a modular command architecture with plugin support and is intended to make command behavior more consistent, maintenance easier, and provide better scalability. Users should expect breaking changes including altered whitespace, newlines, table formatting, colors, argument ordering, flag removals, and a required separator for additional flags; deprecated commands including outbound-rules, pg:repoint, orgs:default, certs:chain, certs:key were removed. Changes apply to core commands, not separately installed plugins, and the release can be installed with heroku update.
How to Use pnpm on Heroku
Heroku’s Node.js buildpack supports pnpm, a fast, disk-efficient alternative to npm, and early users reported 10–40% faster install times on Heroku deployments. pnpm uses a content-addressable store and symlinks to avoid duplicate packages, accelerates dependency installation by combining resolve, fetch, and linking, and enforces dependency boundaries. The tutorial uses Node.js v18 or higher and Corepack to initialize and pin pnpm 9, then creates an Express application whose pnpm-lock.yaml enables buildpack support and consistent package resolution. The demo server handles HTTP GET requests with “Hello pnpm!” and listens on process.env.PORT or defaults to 3000. It finishes by adding a Procfile, committing the project to Git, creating a Heroku app, and deploying with git push heroku main.
LLM-powered data classification for data entities at scale
Grab needed to classify sensitive data at the table and column level across petabytes of database tables and streaming schemas. Manual schema-tiering campaigns had resulted in half of all schemas receiving overly strict Tier 1 access controls, while an initial automated service using regex patterns and third-party machine learning produced high false-positive rates and lacked customizability. To address this, the Caspian data engineering and governance teams enhanced their internal orchestration service, Gemini, by integrating GPT-3.5 via Azure OpenAI. Gemini aggregates classification requests into mini-batches, handles API rate limits, and uses prompt engineering—including few-shot examples, curated tag libraries, and explicit JSON DTO schemas—to reliably tag columns for data owner verification.
eBay Advertising Unveils New Dashboard, Personalized Insights, and Intuitive Campaign Tools
eBay Advertising has introduced a redesigned experience centered on a new dashboard, a unified portfolio, and tools intended to simplify campaign management, increase listing visibility, and support seller growth. The dashboard, available through a new Seller Hub Advertising tab, combines campaign reporting with personalized recommendations, daily optimization tips, suggested campaigns based on historical data and best practices, and trend-based campaigns built using ebay.ai. Sellers can choose among Promoted Listings, Promoted Stores, and Promoted Offsite, with Promoted Listings offering priority or general strategies that differ in placement access, controls, and charging models. The announcement reports average gains of 50% in items sold for priority campaigns, 30% more sales for general-strategy listings, and 25% more sales from external channels with Promoted Offsite, while positioning real-time data and actionable insights as tools for adjusting advertising decisions.
Office Hours with Engineering Managing Director Mae Santos
Mae Santos, Head of Critical Applications and Data Reliability Engineering at Two Sigma, leads a global team across New York, Houston, London, and Tokyo to support critical applications and data pipelines. While software teams frequently prioritize functional features over operational stability, designing systems with reliability in mind from the beginning allows for progressive enhancements without complete rewrites. System architecture should be guided by concrete goals across four key dimensions: availability, observability, scalability, and supportability. In addition, mapping out interconnections among systems, infrastructure, data, and human processes helps teams manage dependencies effectively. Applying automation across support and delivery workflows further reinforces reliability by minimizing human error and accelerating execution.
Optimizing Data Reliability: Heroku Connect & Drift Detection
Heroku Connect syncs Salesforce data with Heroku Postgres, but primary polling can miss changes when long-running Salesforce automations commit after the polling window advances, creating drift. The service uses secondary polls over the same bounds as primary polls to find and repair missed changes, with Bulk API or SOAP API selected according to dataset size and retries for failed polls. Recent improvements change Bulk API drift handling by creating one secondary bulk job for each primary bulk job, then adding corresponding batches instead of creating many jobs. Secondary tasks are now scheduled and executed with limited concurrency, reducing simultaneous database connections consumed while Postgres is updated and preserving capacity for applications. The post says these changes reduce load on Salesforce and improve operational reliability during heavy synchronization, while noting that Heroku Connect remains asynchronous rather than truly real-time.
AWS without Access Keys
The post describes how Fly Machines can access AWS resources such as an S3 bucket without distributing long-lived AWS access keys. It uses OpenID Connect (OIDC): AWS IAM trusts Fly.io’s OIDC identity provider, a role grants narrowly scoped permissions, and the app receives the role ARN as an environment variable. At boot, Fly’s Rust init requests a machine-specific OIDC token through the /.fly/api Unix socket, writes it to /.fly/oidc_token, and sets AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_SESSION_NAME so the AWS SDK can call STS AssumeRoleWithWebIdentity. AWS verifies the token, issues short-lived credentials, and evaluates the role’s IAM policy before serving the requested object. The trust policy can constrain tokens by audience and subject, including an organization, Fly App, or machine; the same pattern can authenticate against other OIDC-compliant providers, although Azure and GCP lack equivalent init integration.
Top topics
- Architecture834
- Performance815
- LLMs659
- Machine Learning621
- Open Source563
- AI Agents553
- AI469
- Developer Experience448
Weekly digest
The best engineering reads, once a week.