Loading…
Topic
160 posts about Deployment. Every summary links to the original.
See the idea
A/B testing trades traffic, time, and exposure risk for evidence. This explainer maps the benefit of a controlled comparison, the real cost of running one, and the condition that decides when measuring beats shipping on judgment.
Bun packs a runtime, package manager, bundler, and test runner into one binary. Here is how those four parts fit together and where the seams still show.
CloudFormation is a declarative reconciler: you describe the target state in a template, and it provisions, updates, and reports on the stack that results. Here is the loop.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Hugging Face has integrated spaCy pipelines into the Hugging Face Hub to simplify the distribution, testing, and deployment of natural language processing models across the community. More than 60 canonical models from the spaCy 3.1 release are hosted in the spaCy organization, alongside discoverable community pipelines. Users can package and upload models directly through the command line using the spacy-huggingface-hub extension, which auto-generates required metadata and model cards. Hosted pipelines offer direct pip installation from live wheel URLs, interactive browser exploration, and production-ready HTTP endpoints through the Inference API. The platform provides immediate interactive widget support for named entity recognition components, with upcoming support planned for text classification and part-of-speech tagging pipelines.
Hugging Face and Amazon introduced an inference solution integrating Hugging Face Transformers with Amazon SageMaker. The SageMaker Hugging Face Inference Toolkit and specialized Deep Learning Containers enable deployment of trained models or publicly available Hub models to managed production endpoints. Users configure endpoints with minimal code using the SageMaker Python SDK, standard pipelines, or custom inference scripts that override default behaviors. The architecture supports models stored in Amazon S3 as well as direct references via HF_MODEL_ID and HF_TASK environment variables. Deployments benefit from native AWS infrastructure features, including built-in monitoring, Identity and Access Management permission controls, and Virtual Private Cloud connectivity.
Shopify describes how its SSL team moved merchants’ stores to HTTPS and ultimately secured more than 4.5 million domains with Let’s Encrypt. The effort began against certificate providers whose APIs were not designed for fully automated provisioning: errors often arrived as human-readable messages, and imposed throttling could have stretched initial issuance to about 100 days. Shopify tested the ACME protocol, valued its standardized specification and open-source implementation, and first added Let’s Encrypt as a backup certificate authority to improve emergency key or certificate-chain rotation. The team rolled out that integration in a few hours, compared with months for earlier providers, then made Let’s Encrypt its primary authority after finding it reliable. The account presents this change as evidence that ACME can make certificate automation more responsive and allow certificate authorities to be added without redesigning Shopify’s infrastructure.
Supabase announced a series of platform updates, documentation additions, and tooling enhancements for June 2021 ahead of its scheduled Launch Week. Developers can now provision a Postgres database directly through a new Vercel integration and enable authentication using Discord logins. Supabase Storage added support for upsert operations alongside public buckets that allow unauthenticated file access. The web dashboard introduced a restart button for debugging and a table policy editor with templates to configure Row Level Security. Additional backend improvements include timestamps for PostgREST error logs, an OpenAPI mode patch ignoring anonymous privileges for Prisma users, and an XSS fix in the Auth0 Next.js library.
Conveyor, an in-house continuous delivery system at Grab, introduces hermetic deployments by tracking application code alongside static and dynamic configuration parameters. This hermeticity guarantees that production releases use combinations of versioned artifacts previously verified in staging, preventing rollback incompatibilities. Conveyor replaces single multi-environment pipelines with decoupled pipelines, while automating cluster locking, deployment slot scheduling, release note generation, and canary monitoring with automated rollbacks. These automations reduced production deployment failure rates from 1.5% to an average of 0.3% over a three-month period. Grab also doubled the volume of production changes between 2018 and 2020 while saving more than 5,000 man-days of engineering effort in 2020.
Hugging Face and Amazon formed a strategic partnership establishing Amazon Web Services as Hugging Face's preferred cloud provider to simplify training and deploying Transformer models. To facilitate this collaboration, the teams introduced Hugging Face Deep Learning Containers optimized for PyTorch and TensorFlow in Amazon SageMaker. The integration provides an extension to the SageMaker Python SDK, enabling data science teams to configure distributed training jobs across single-node and multi-node GPU clusters. Workflows integrate directly with SageMaker Automatic Model Tuning, SageMaker Studio, and Amazon S3 for managed experiment tracking and artifact storage. The open-source containers allow trained models to be exported for inference via SageMaker or published directly to the Hugging Face Model Hub while utilizing EC2 Spot instances to reduce compute expenses.
Engineering teams at Grab initiated a transition to containerized microservices to standardize environments, enhance security, and decouple services from internal runtime tooling. During this migration, developers noticed that statically-linked Go service binaries were reaching bloated sizes over 100 MB. By analyzing the binaries using the open-source tool go-binsize-viz alongside the Go nm toolchain, the team visualized compiled symbols as interactive treemaps. This analysis revealed that 11 MB of unused message format symbols were being pulled in because a generic interface shared a directory with auto-generated streaming code. Restructuring the packages to isolate interfaces from generated code successfully decreased the binary size down to 78 MB.
Fly introduces persistent volumes for apps and a new remote-build path designed to make Docker image builds faster. Volumes are created through flyctl, backed by encrypted ext4 filesystems on Linux LVM2 thin pools, and attached one-to-one with app instances on compatible hosts. Volumes survive reboots and deployments and are described as performance-competitive with, usually slightly faster than, EBS, but raw-volume resilience remains the customer’s responsibility. Because volume count limits instance count and prevents an extra canary volume, volume-backed apps use rolling deployments; the post identifies replicated database clusters, caches, metrics, and restorable data as suitable cases. Remote builds now match jobs to automatically created builder apps with attached storage for cached layers and manifests; builders expose authenticated Docker servers, terminate after 10 minutes of inactivity, and are not restarted after successful exit.
Continuous deployment makes changing a running system risky, especially for a platform serving more than a million merchants and changing code multiple times daily. The post presents beta flags as a control mechanism: per-Subject BetaFlags support explicit activation, while BetaRollouts use a percentage and a digest-based modulo calculation to assign a stable, growing subset as rollout increases. It then places a higher-level Feature abstraction above those primitives, allowing direct targeting, percentage rollout, per-Subject opt-outs, and a 100% kill switch that can halt a feature without deleting thousands of database records. The guidance also covers testing permutations, notifying owners through BetaIdentifier metadata, and removing stale flags after rollout. These patterns provide operational control and safer deployment, while requiring substantial implementation and ongoing cleanup.
Fly’s command-line tool, available as both flyctl and fly on most platforms, includes shortcuts and options that reduce directory changes and browser navigation. App selection can use -a appname instead of relying on the current directory’s fly.toml; fly deploy can also build from a specified child directory, such as ./mygreatchildapp. When a fly.toml is present, the command asks for confirmation that the supplied app name should be used. Multiple deployment variants can be kept in separately named configuration files and selected with -c or --config, including when deploying from a monorepo tree. Browser-oriented commands include fly open for an app’s .fly.dev URL, fly dash metrics for dashboard metrics, and fly docs for the documentation’s top page.
Fly.io introduces configurable VM shutdown behavior for applications whose clients, connections, or transactions need more than the default five-second termination window. After sending SIGINT, Fly normally waits five seconds before forcefully terminating a VM; the new kill_timeout setting extends that wait to as much as five minutes on Shared CPU VMs or 24 hours on dedicated CPU VMs, while kill_signal selects among several signals. The post demonstrates HAProxy’s soft-stop mode with SIGUSR1 to drain existing TCP connections and Postgres’s smart shutdown with SIGTERM to let open transactions commit before termination. It also describes infrastructure considerations: deploys and other releases can disrupt in-flight connections, so Fly drains connections where possible and offers ways to reduce reliance on shared HTTP and TLS handlers, plus stateless UDP and experimental stateless TCP load balancing.
Shopify organized a cross-company program to prepare its platform for Black Friday and Cyber Monday (BFCM), coordinating more than 2,500 people responsible for over 1,000 deploys across services. Engineering Program Managers absorbed administrative, communication, and reporting work so BFCM leads could focus on technical challenges, while departmental champions assigned preparation tasks. Preparation covered maintenance, game days, load tests, capacity increases, weekly tracking, and a Caution Tape bot requiring risk-to-value assessments for pull requests in critical areas. Deploys were slowed through smaller batches, longer canaries, and a planned weeklong lock while communication used Slack, GitHub, documentation, presentations, emails, and office hours. The effort coincided with $5.1+ billion in BFCM sales and load testing plus resource scaling paid off, but developer productivity declined amid inconsistent messaging; future plans called for a more balanced approach, automation, and reusable communication templates.
The November 2020 Fly changelog records new example applications, Fly CLI enhancements, and platform updates. Guides now cover Gogs, MQTT with Mosquitto, MinIO, Node-RED, and Redis with and without TLS, and demonstrate preview persistent Volumes for database persistence, data storage, preserved programming state, and queued messages. The CLI added bulk secret imports from key=value files, including multiline values enclosed in triple double quotes, while list commands gained filtering, sorting, and JSON output; the Deno builtin added a Version setting defaulting to 1.5.4. Platform changes included an upgrade to Firecracker v0.23, improved restart and connection handling, timeout fixes, and stronger application-context isolation. The changelog also notes new crash-log context, humanized app times, improved image resolution, and corrected internal port and metrics handling.
The October Fly changelog records platform and CLI updates, centered on preview persistent disks, configurable builtins, environment-variable configuration, and status monitoring. The new volumes commands let Fly applications create persistent disks whose data survives restarts, deployments, and suspension; the feature remains in preview. Applications can now set environment variables in fly.toml instead of using the secrets feature, while builtin settings expose controls such as Deno permissions and HTTPS auto-upgrading and logging for the static web server. flyctl 0.0.145 also adds Python/Procfile builtin support, a --watch status mode that polls every five seconds, a --rate option, default-org selection in init, and fixes for token checks and wildcard certificate domains.
Incident #2105 began during a production deployment of a Go-based Git server after a configuration variable had been set but not transmitted to production instances. Staging deployment completed successfully before the same change was attempted in production. The deployment sent a new code version to one batch, whose unauthenticated calls to a private API produced HTTP 401 responses that were returned publicly; the author paused before deploying the remaining batches and resolved the issue by synchronizing configuration, killing affected processes, and allowing them to restart. The post argues that the failure was enabled by a process requiring separate, infrequent manual actions rather than being solely an individual omission. Its proposed remedy is to combine configuration synchronization with code deployment, or alternatively warn when deployment begins with unsynchronized variables, supported by a blameless retrospective.
September’s Fly CLI changes focus on easier installation, updates, and Windows 10 use while also adding static-site deployment options and app-initialization controls. The installer still installs flyctl but now creates a symbolic link for fly, beginning a planned migration to fly as the CLI’s actual name; fly version update now downloads and installs the latest release. New hugo-static and staticplus builders support static websites, with hugo-static running a Hugo build before deployment and staticplus adding HTTPS redirection for custom domains, though staticplus is slated for retirement. Windows received PowerShell-based installation and updating, while fly init gained --overwrite and --nowrite for controlling fly.toml changes and supporting deployments under alternate app names recorded in fly.alias. The release also included DNS and domains command updates, Alpine avoidance for builtins, a Deno builder change removing the deps.ts requirement, and improved allocation displays.
Around the end of 2018, Grab's backend architecture consisted of roughly 270 services managed through fragmented, manual deployment workflows. Engineers copied release parameters between build logs, wiki pages, Slack bots, and multiple Jenkins jobs, leading to high operational friction and an average of 10 business days between production updates for a service. To streamline delivery, Grab built Conveyor, an internal automation platform built on top of open-source Spinnaker. Conveyor introduced a custom user interface focused on pipeline visibility and a pipeline-as-code DSL called Artificer using Jsonnet files in the monorepository. The platform automatically registers build artifacts with commit metadata to eliminate manual parameter entry and automatically provisions integration, staging, and production pipelines.
This Q&A addresses three operational questions about Fly: running non-HTTP applications, identifying connection IP ranges, and renaming applications. Non-HTTP apps need a network service for Fly’s default TCP or optional HTTP health checks; otherwise, Fly may kill them, although Worker process support is planned. For connectivity controls, Fly says AS40509 ranges may not match source addresses seen by servers and advises treating Fly IPs as changeable, while client certificates over TLS offer improved authentication. Future IPv6 per-application addresses could support firewall allowlists. Renaming currently requires creating a new app with flyctl init, claiming the desired name, changing fly.toml, redeploying, and manually restoring secrets, certificates, hostnames, scaling, regions, and other configuration, while sacrificing the original app’s history.
Flyctl v0.0.139 adds built-in builders for Node, Ruby, Deno, Go, and static web servers, allowing an app to be initialized and deployed without a Dockerfile. The release also lets flyctl init select a public Docker image, stores that choice in fly.toml, and supports repeatable image builds with configurable ports and health checks. Region visibility improves through backup-region listings and a (b) marker in flyctl status, while guided certificate creation supplies DNS instructions for traffic direction and ownership verification. Additional changes include --import, --local-only, timestamped JSON status output, orgs and dns commands, and clearer deployment and authentication messaging. The changelog notes that orgs and dns remain works in progress, and version 0.0.139 fixes an issue with builtin init.
DNS traffic management applies data-driven DNS responses to direct clients to different service endpoints, extending DNS’s basic role of mapping domain names to IP addresses. The post introduces DNS through its history from HOSTS.TXT, explains zones, records, nameservers, root servers, TTL, and negative TTL, then connects caching behavior to routing decisions. Low TTLs can make changes visible sooner, but repeated DNS resolution adds latency: with a one-second resolution time, a 15-second TTL could add 5,760 seconds per day, compared with 1,440 seconds at 60 seconds; resolver behavior can also make delays slightly longer. Use cases include active-passive failover, active-active traffic sharing, progressive green-blue deployments, and location-based routing, while Shopify’s system evolved into an automated, self-served platform serving more than 40 domains across 12 teams and over 100M requests per 24 hours.
The best engineering reads, once a week.
