Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,535 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Fly Answers Questions
Fly Answers Questions compiles responses about custom domains, configuration recovery, Turboku deployment, database latency, TLS termination, global databases, continuous deployment, and idle-app scaling. Fly allows multiple custom hostnames per application and automatically provisions Let’s Encrypt certificates after validation; a deleted fly.toml can be restored with flyctl config save -a , with -c supporting another filename. For Turboku, Heroku slugs run as containers in a nearby datacenter rather than at the edge, while region selection can reduce database latency and retain edge SSL termination. The service supports TLS-only termination up to HTTP/1.1, GitHub Actions deployment, Redis for local caching, and a replication system for distributing changes, while keeping one running instance for idle apps instead of scaling them to zero.
Scaling Fly for All
Fly makes its scaling controls available to all users, allowing them to manage application instance counts across regions and choose virtual-machine sizes. The default plan places an application in one region, while Fly’s autopilot can add an instance when a connection limit of roughly 20–25 is reached and select its location. The new flyctl scale regions command reports or sets regions and per-region minimum counts; for example, ams=1 hkg=1 sjc=1 keeps at least one instance in each location, with weights influencing where new instances are preferred. Users can also run flyctl scale vm to apply a VM size to all current and future instances, triggering a new app version and redeployment. Available sizes range from micro-1x with 128 MB to cpu8mem8 with 8 GB, while the article notes that the system was later superseded.
Tackling UI Test Execution Time Imbalance for Xcode Parallel Testing
Parallel test execution in Xcode can suffer from test time imbalance when tasks finish at significantly different times across parallel simulator workers. Analysis of Xcode scheduling logs shows that the runner groups tests by test class and dispatches all tests from the same class to a single simulator. Attempts to customize the suite by swizzling XCTestSuite fail because made-up suites initialize only after tests are dispatched. To overcome this grouping constraint, unique tokens or test names are appended to the class name component in `-only-testing` command-line arguments. This trick forces Xcode to treat each test as an independent class, successfully distributing individual tests across separate workers.
Refactoring Legacy Code with the Strangler Fig Pattern
Shopify’s Ruby on Rails Shop model had grown beyond 3,000 lines, accumulated numerous responsibilities, and developed weak semantic boundaries, making store settings a target for extraction. The team used Flog’s ABC-based score—assignments, branches, and calls—to identify a particularly disordered area, then applied Martin Fowler’s Strangler Fig Pattern to move the locked_settings behavior into Capital::SettingsToLock and a dedicated record/data source. The seven-step process defined and tested a new interface, redirected callers, introduced and backfilled the new data source, switched reads, and finally stopped legacy writes and removed the old column and code. Keeping the old system available during the transition supported incremental changes, monitoring, reversibility, and the stated no-downtime requirement; the completed example left the new system in place with legacy remnants removed.
How to do Custom Domains with Fly
Custom Domains with Fly is an application guide for using Fly to proxy traffic or serve applications across multiple custom domains. It explains how to configure an NGINX server for external sites and how to attach host and domain names to Fly applications with the flyctl command line. For automated workflows, it describes the Fly GraphQL API operations for adding, removing, checking, and deleting hosts and associated certificates, and points to a repository of ready-rolled Node applications demonstrating the API. The guide's speed run retrieves an app's IPv4 and IPv6 addresses, adds corresponding A and AAAA DNS records, creates a certificate with flyctl, and uses another command to monitor issuance. The resulting setup provides auto-renewing Let's Encrypt TLS certificates, edge TLS, and HTTP/2 support; the guide also states that Fly applications have no limit on attached certificates and can scale on demand.
Building with Web Components and Their Advantages Over Third-party Libraries
Web development moved from HTML, CSS, and JavaScript toward proprietary libraries such as React, Angular, and Vue, creating interoperability concerns that Web Components aim to address through browser standards. The post traces this shift from early W3C work and Google’s Polymer Project to a toolkit of custom elements, HTML templates, and Shadow DOM, with examples showing how custom tags and reusable markup are defined. Custom elements encapsulate JavaScript behavior, templates defer markup rendering until requested, and Shadow DOM isolates internal styles and elements from surrounding page content. The resulting components can be packaged as dependencies and consumed across most browsers without loading a front-end framework, while libraries including Polymer, Stencil, and Lightning Web Components can abstract the underlying APIs.
The Evolution of Kit: Automating Marketing Using Machine Learning
Kit began as a heuristic, rules-based virtual employee that simplified Facebook advertising for Shopify merchants by selecting products and budgets while defaulting other settings to best practices. The post describes its evolution into an artificially intelligent assistant that uses merchant spending behavior and store signals such as traffic and sales to recommend marketing actions. Kit's machine-learning design combines regression to predict likely spending with classification to estimate whether a budget and store state are likely to generate sales; training uses feature engineering, Apache Spark, Google Cloud ML Engine, and monitored model metrics. For real-time predictions, Bigtable stores engineered features, while a Tornado web container delegates model computation to TensorFlow Serving. The resulting system powered one third of Kit's marketing campaigns.
How to generate text: using different decoding methods for language generation with Transformers
Auto-regressive language generation relies heavily on decoding methods to convert predicted next-token probability distributions into coherent text. Greedy search selects the single most probable token at each timestep, but it frequently leads to repetitive phrasing and misses high-probability sequences hidden behind low-probability intermediate words. Beam search alleviates this issue by maintaining multiple parallel hypotheses and can be paired with n-gram penalties to reduce immediate repetitions. Sampling approaches, including top-K and top-p sampling, offer alternative paths that can yield more fluent, varied open-ended generation. Using the Hugging Face transformers library, practitioners can configure these strategies via parameters on the model generate function.
Fly Friday - News and Tips
This Fly weekly update collects product news, community announcements, and a flyctl usability improvement. It points to guidance for building GraphQL APIs with Fly and for running a private DNS over HTTPS (DoH) service, while noting Firefox’s default DoH rollout in the USA. It also announces a new Fly user community portal with GitHub or social-media sign-in, alongside Turboku’s Product Hunt launch for faster Heroku apps. The technical note covers flyctl 0.0.96’s -c/--completion option: supplying bash or zsh generates a completion file, with zsh users able to redirect it to $fpath[1]/_flyctl and Bash users required to choose a location and source it from .bashrc.
Returning 575 Terabytes of Storage Space to Our Users
Android Vitals data revealed that 15.7% of Grab users had less than 1GB of free device storage and uninstalled the app at 1.2 times the normal rate. To understand on-device storage consumption, the team instrumented session launches using the Android StorageManager API to collect binary size, cache folder size, and total footprint metrics. Analysis showed unusually large cache sizes driven by orphaned cache folders from discontinued third-party libraries, including an image library replacement of Picasso by Glide. An automated cleanup routine deployed in app updates purged legacy cache directories upon launch. This mechanism reclaimed 575 terabytes of junk data across more than 13 million devices, averaging 40MB per user.
GraphQL, Hasura and Fly
Hasura provides an open source GraphQL backend for accessing PostgreSQL, mapping database schemas and roles to simplify queries. It can also query other GraphQL services, allowing multiple services to be combined, and includes an interactive console for analysis and exploration. The post presents deploying Hasura on Fly using its Docker image to create a global GraphQL backend, with configuration required for the database location, secrets, and interactive-console availability. It points to a published deployment guide that walks through those steps and frames Fly as a simpler way to run Hasura, while directing readers to Fly Docs and a hands-on path for deploying a first app.
Creating Native Components That Accept React Native Subviews
React Native’s standard native-module documentation does not explain how to reuse an existing native container while rendering React Native views inside it. This Android-focused Kotlin guide builds a native bottom-sheet component with a ViewGroupManager, inflating a CoordinatorLayout XML layout whose second child becomes the draggable sheet content. It overrides addView to place the first React Native child in the main container and subsequent content in the bottom sheet, then exposes the component through TypeScript with style and children props. A sheetState prop accepts collapsed or expanded, while BottomSheetBehaviour emits a BottomSheetStateChange event so native gestures update React state. The guide concludes that ViewGroupManager enables flexible React Native layouts around native functionality, while noting that platform parity remains important.
Fly Friday - Customizing with Dockerfiles
Fly Friday describes how to customize a Docker image before deploying it to Fly, building on an earlier example that deployed Docker’s official httpd image. The default image serves files from /usr/local/apache2/html and displays “It works”; replacing that content requires adding local HTML files to the image. A two-line Dockerfile uses FROM httpd:2.4 as the base and COPY ./public_html/ /usr/local/apache2/htdocs/ to place the contents of a public_html directory into the new image, which Fly then deploys. The post also notes that the directory must be created and populated with HTML, and points to a video and Fly Docs, including a hands-on path to a free account and first app deployment.
Grab-Posisi - Southeast Asia’s First Comprehensive GPS Trajectory Dataset
Grab released Grab-Posisi, a GPS trajectory dataset covering Singapore and Jakarta designed to overcome the poor coverage, low sampling rates, and missing contextual metadata common in existing public datasets. Captured in April 2019 from drivers in transit, the dataset comprises 84,000 trajectories, over 80 million GPS pings, and more than 1 million kilometres recorded at a one-second sampling frequency. Each entry in the 2 GB Apache Parquet dataset records geographic coordinates, timestamp, accuracy radius, bearing, speed, mobile operating system, and vehicle mode. To protect privacy, driver personal information is encrypted and trip start and end locations are removed. The data enables applications including automated road network reconstruction, map matching, real-time traffic forecasting, and municipal infrastructure planning.
Your Circuit Breaker is Misconfigured
Misconfigured circuit breakers can leave an application effectively unavailable during a dependency outage by allowing timed-out requests to consume worker capacity. Using Shopify’s Semian implementation, the post explains how name, error_threshold, error_timeout, half_open_resource_timeout, and success_threshold govern circuit opening, recovery, and wasted utilization across failing service instances. It models timeout spikes for a single worker, notes that separate names isolate services and instances, and presents an equation for steady-state additional utilization, while noting that context-switch costs are not included. A live test closely matched the equation’s prediction. In a Rails worker with two threads and 42 Redis instances, changing half_open_resource_timeout from 0.25 seconds to 50ms and error_timeout from 2 to 30 seconds reduced modeled extra utilization from 263% to 4%, with slower recovery; an author’s edit says success_threshold does not affect steady-state utilization.
New API Rate Limiter
Zoom announced an updated V2 REST API Rate Limiter launching in March 2020 to replace its previous one-size-fits-all throttling system. Rather than applying uniform constraints across all subscription tiers, the new framework cross-references customer Account Types against API Request Groups categorized by infrastructure importance and load intensity. While most API rate limits are increasing, resource-intensive Dashboard APIs are reduced from 1 request per second to 6 or 12 requests per minute depending on account level. Throttled HTTP 429 responses will return X-RateLimit-Reset timestamps for per-minute limits and ISO8601 Retry-After timestamps for per-day limits. Additionally, the system introduces lock-key restrictions on specific user identifier path variables to restrict concurrency and enforce daily meeting creation caps.
Developing Zoom Marketplace Apps w/ ngrok
Developers frequently use ngrok to establish fast, introspectable tunnels to localhost when building Zoom Marketplace applications instead of configuring complex reverse proxies like NGINX or Apache. However, using basic ngrok tunnels in production exposes systems to short URL expiration windows, single points of failure, and scalability bottlenecks. To maintain application availability, developers can purchase an ngrok license to secure a service-level agreement and support. Additionally, teams should reserve dedicated subdomains rather than using auto-generated endpoints, run ngrok as a monitored background service on a cloud provider or data center, and place a load balancer in front of the tunnel to handle heavy traffic.
What’s New and What’s Coming for Zoom Marketplace in 2020?
Zoom announced several updates to its App Marketplace alongside roadmap deadlines scheduled for early 2020. Marketplace enhancements include functional app categories, additional URL fields for Terms of Use and documentation, and support for defining multiple webhook event subscriptions within a single application. Administrators and developers gained webhook logs, enhanced pre-approval permission settings, and the ability to test draft applications directly in the installed apps interface for Data Compliance API compliance. To accommodate surging traffic, Zoom introduced revamped V2 REST API rate limits and breaking behavioral changes to the List Users API. Crucially, the platform set an absolute end-of-life date of April 26, 2020, for all legacy V1 APIs, requiring immediate migration to V2 endpoints.
Fly Friday - Flyctl and Ports
Flyctl adds a new -p option to flyctl apps create, allowing users to set an app’s internal port during initial creation. The numeric value is written to the generated fly.toml configuration, replacing the default port of 8080. This addresses workflows using prebuilt Docker images, where users may not control the port and previously had to create the app, edit fly.toml, and change the internal port manually. Available from Flyctl release 0.94, the flag removes that edit and is shown in the first Fly Friday video. The post defines the internal port as the port an application uses to communicate with the world, while the command-line setting supplies that value before deployment configuration is generated.
How to train a new language model from scratch using Transformers and Tokenizers
Recent library enhancements simplify pretraining custom language models from raw text corpora. A small 84-million-parameter RoBERTa architecture comprising six layers, twelve attention heads, and a hidden dimension of 768 was pretrained on a 3 GB Esperanto text dataset. Training began by fitting a 52,000-vocabulary byte-level Byte-pair encoding tokenizer, which natively handled Esperanto diacritics and reduced average sequence lengths by approximately 30 percent compared to GPT-2 tokenization. The model was trained using masked language modeling, evaluated using fill-mask pipelines, and subsequently fine-tuned for part-of-speech tagging via standard token classification scripts. Final model artifacts and configurations were uploaded for community reuse via pre-trained model interfaces.
Top topics
- Architecture830
- Performance812
- LLMs648
- Machine Learning617
- Open Source561
- AI Agents542
- AI454
- Developer Experience446
Weekly digest
The best engineering reads, once a week.