Loading…
Topic
Release Engineering
7 posts about Release Engineering. Every summary links to the original.
- 7 posts
- 6 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
How Etsy gets its mobile apps ready for peak traffic
Mobile apps cannot be reverted after release, so Etsy’s peak traffic preparation focuses on observability and mitigation decisions made months in advance rather than capacity scaling during the event. The company runs a code slush with three phases: mindful merging in mid-October, infrastructure fortification, and peak requiring senior sign-off for any change. To avoid tooling disruption, Etsy ran both legacy and new crash reporters simultaneously through a dual-reporting layer, letting engineers use the tool they knew and providing comparison data to eventually sunset the old platform. Before peak, each crash-prone feature is ensured to have a remote flag that disables the failing code path without an app store release. Crash rate is translated into an estimate of lost orders via business analysts, making technical health discussable with teams and the CTO.
Why model versioning is not enough for production AI
Model versioning alone is insufficient for production AI because changes in inputs, preprocessing, prompts, retrieval, tool contracts, and serving settings can alter behavior independently. The post proposes a versioned release manifest that captures all interdependent components, including model revision, prompt revision, retrieval index revision, embedding revision, pipeline revision, and runtime revision covering token limits, batching, and timeouts. This manifest enables atomic release testing and rollback, with an evaluation gate that tests the full retrieval-generation path using deterministic checks and semantic rubrics on a versioned dataset. Practical steps include canary deployments that compare candidate and control signals, with explicit stop conditions and rollback procedures that restore compatible dependencies rather than just older model weights. The post concludes that the smallest useful implementation is a manifest, evaluation job, load test, release-aware traces, and a rehearsed rollback, and asks whether the on-call engineer can identify and restore a compatible known-good version.
Announcing Evals and Releases: Evaluate Fin before, during, and after you go live
Fin’s new Evals and Releases provide an evaluation and rollout system for testing changes before launch, controlling deployment, and assessing live conversations, alongside Monitors. Evals group multi-turn Simulations built from real conversations around a theme, then score them pass/fail using deterministic checks and an LLM judge against criteria such as required responses, Procedures, or data connector calls. Releases let teams edit content, Procedures, and Guidance away from live Fin, run Evals on proposed changes, collaborate, and publish to everyone, ramp traffic, A/B test, or roll back. Monitors evaluates every live conversation and can turn failures into new Simulations for subsequent testing. The combined workflow addresses probabilistic behavior, regression risk, and drift as support content and policies change, while Operator can draft changes, tests, and Releases for review and approval.
How We Made All Swiggy Apps Operations AI-Native
Swiggy’s mobile engineering teams faced operational overload as releases, production signals, and reviews required switching among Jira, Confluence, Bitrise, Play Console, Crashlytics, New Relic, Google Sheets, ImageKit, and Slack. The response was an operating layer built around Slack-native agent workflows, reusable AI skills, MCPs, and scheduled jobs connected to release, observability, QA, cost, and internal mobile systems. It can generate readiness updates, group Jira work, identify missing owners and blockers, summarize vitals, analyze alerts, follow up TRM actions, and recommend RAMP, HOLD, or ROLLBACK for staged rollouts by correlating Crashlytics, Play Vitals, and adoption. Humans still approve rollout actions, while recurring channel reports make observability and accountability more consistent and reduce clerical integration work. The stated model lets agents handle mechanical load while engineers retain judgement, ownership, product quality, architecture, and execution.
Building Zone Failure Resilience in Apache Pinot™ at Uber
Uber describes how it made Apache Pinot clusters resilient to zone failures while preserving query and ingestion availability for Tier-0 use cases. It maps Uber’s virtual resource isolation group IDs to Pinot server pools, uses pool-based instance assignment with replica-group segment assignment, and requires at least as many isolation groups as the table’s replication factor. During migration, automated pool registration, configuration backfills, granular APIs, selective rebalancing, and rollback support were used across more than 400 Tier 0-1 clusters and thousands of tables. An internal drill found that one remaining replica continued serving traffic without query degradation after a zone went down, while isolation-group-based claim and release policies enabled parallel restarts and sequential rollouts with baking time.
Shipping is your company's heartbeat
Software becomes valuable only when it reaches customers, because deployment starts the feedback loop that tests assumptions and enables the next improvement. The post argues that although software deployment can now take seconds or minutes, teams often retain habits from an era of infrequent, costly releases, making shipping itself a source of risk and delay. It uses time to production (TTP) as a technical and organizational lens: a one-minute fix can be tested, reviewed, merged, deployed, and communicated immediately, while hours, days, or weeks encourage waiting, batching, deploy windows, larger releases, and growing bug backlogs. Continuous shipping is presented as a company-wide practice that improves feedback, decision-making, customer responsiveness, momentum, motivation, and hiring, while requiring ongoing effort to reduce TTP and treat shipability as a design concern.
Mobile Release Engineering at Scale with Shipit Mobile
Shopify built Shipit Mobile to replace error-prone, manual mobile app releases with a shared platform for creating, viewing, and managing releases. The process creates a release branch from master, builds a candidate for each commit, increments build numbers, and starts separate CI pipelines for testing and release builds. Shipit Mobile uploads the same binary and metadata to distribution channels including Google Play, the App Store, and internal support access, using store APIs and Fastlane, while Slack notifications make release status visible beyond the release captain. Developers can bypass CI when necessary for emergencies, and configuration favors convention over configuration for project metadata. After more than six months in production, the platform reduced release complexity, shortened the release cadence from three weeks to one week, and helped more people become release captains.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.