Loading…
Topic
134 posts about Migrations. Every summary links to the original.
See the idea
Amazon EVS runs VMware Cloud Foundation on AWS bare metal, so the design question becomes where each workload's data lives. This layered diagram shows how vSAN, FSx for ONTAP, and cross-Region DR divide the work.
A side-by-side comparison of legacy Tomcat defaults and the strict defaults in Tomcat 11, showing which behaviors change and what to verify before you upgrade a Java web tier.
Learn the step-by-step process for deploying .NET 10 applications to Heroku, from buildpack detection to SDK pinning, and understand the lifecycle of .NET LTS releases.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Musinsa’s Audience Engine powers a customer data platform that targets audiences across 16.4 million members. It stores each user-to-audience relationship as a separate row, producing roughly 1.1 billion cohort-to-user mappings across 2,500 audiences and potentially 16 million new rows when one audience is added. Its self-hosted ClickHouse deployment coupled storage and compute, forced workloads to share a cluster, and created operational burdens as data and query demands grew. Musinsa migrated to ClickHouse Cloud on AWS, using separated S3-backed storage, workload-specific compute, automatic idling, vertical autoscaling, and ClickPipes for direct real-time ingestion. The company reports an 86.5% reduction in storage costs and a total cost of ownership reduction of up to 71.4%, while shifting engineering effort toward CDP business logic.
Thumbtack migrated 1,500 Android screenshot tests from Firebase Test Lab to Roborazzi to eliminate slow and error-prone testing on remote emulators. The engineering team ported test suites from androidTest to test in batches of 20, adjusting test rules and annotations for Robolectric. Because Roborazzi runs in a single JVM process with parallel execution, the team had to debug shared font cache pollution and cross-platform shadow rendering inconsistencies between macOS and Debian. They resolved configuration repetition by mapping Robolectric annotations into module-level robolectric.properties files and avoided stale cached images by clearing Roborazzi's cache via a custom Gradle task. The migration significantly decreased CI runtimes while making local test recording and verification easier.
ClickHouse reports a set of updates to its Managed Postgres beta, aimed at simplifying onboarding and advancing its unified Postgres-and-ClickHouse data stack. The redesigned flow supports sample-data loading or PostgreSQL migration through ClickPipes, built-in SQL exploration, replication to ClickHouse, and cross-system queries through pg_clickhouse; services also notify users when disk usage remains above 85%, with console, email, and optional Slack channels. Observability now includes searchable PostgreSQL logs, CPU, memory, disk, IOPS, connection and per-table metrics, query insights, a Prometheus-compatible /metrics endpoint, and an OpenAPI slow-query API. Backups use more CPU and disk throughput on larger servers, making backup and restore times predictable even on 10+ TB servers, while extensions add pg_re2 and improvements to pg_clickhouse and pg_stat_ch, including regex pushdown, Arrow batching, sampling, and safer worker exits; Google Cloud Platform availability and further maintenance, configuration, and integration work are planned.
Yelp migrated a 1.4-million-line JavaScript monorepo from Flow to TypeScript after community support and package ecosystem tooling shifted toward TypeScript. To maintain continuous type safety without code freezes, engineers converted the codebase package by package starting at the deepest internal dependencies. The toolchain used flowts for code conversion and flowgen to turn generated TypeScript declaration files back into Flow-compatible headers for downstream consumers. Webcore dogfooded the tooling across 90 packages, automated tracking through internal metrics, and assisted feature teams with bottleneck conversions. Completed in February 2026, the migration increased overall type coverage from 83.15% to 96.44% and unlocked TypeScript-only tooling.
Checkout Blocks upgraded all five UI extensions from React and Shopify’s legacy Remote UI bridge to remote-dom with Preact and Polaris web components, while moving from JavaScript to TypeScript and API version 2026-01. The migration addressed checkout’s performance sensitivity and Shopify’s requirement that deployments move off versions earlier than 2026-01. A parallel core-next library allowed extensions to transition incrementally, with shared providers, telemetry, hooks, helpers, and rules adapted before legacy core was removed. To meet the remote-dom CLI’s 64KB gzip limit, the team dropped react-reconciler, replaced liquidjs with a smaller parser validated against more than 42,000 lines of merchant configurations, replaced dayjs, and retained markdown-to-jsx through Preact compatibility. Transferred bundles fell 40% to 85%, while weighted ELT improved 8% at P50 and 7% at P90.
Yelp migrated its frontend React monorepo from the deprecated apollo-tooling to GraphQL Codegen to ensure ongoing maintenance and build reliability. Because the monorepo spans over 500 packages with thousands of generated type files and custom nested type imports, the Client Data team prioritized a transparent migration that avoided codebase-wide changes. The team rejected approaches requiring syntax updates or altered import paths, choosing instead to configure GraphQL Codegen and contribute upstream plugin changes so output structures and naming conventions matched the legacy output. Migrating packages incrementally through small pull requests enabled continuous feedback while allowing developers across dozens of teams to maintain their workflows without editing existing code. Completing the migration to GraphQL Codegen v6 simplified Yelp's build tooling, eliminated deprecated dependencies, and left upstream compatibility enhancements available for the wider open-source community.
Trainy’s Pluto is an open-source experiment tracker that separates analytics metrics in ClickHouse Cloud from relational run metadata in Postgres. Its original metadata store, Amazon RDS Postgres, was too expensive for the volume observed and contributed to slower initial page loads, so Trainy migrated roughly 40–50 GB to ClickHouse Managed Postgres, which uses local NVMe storage and keeps both databases in one account. After migration, initial page loads became 2–3 times faster, while a complex query reported as taking 10–15 minutes previously returned in five seconds; visualization queries over runs with hundreds of thousands of steps typically return in 50–100 milliseconds at the P50. Trainy reports 100% reliable ingestion with zero dropped data points and says the unified stack could support future joins between GPU metrics and run metadata, as well as governed agent access.
Databricks presents Lakebase as a fully managed, serverless, standard Postgres database for combining operational and analytical workloads on its Data + AI Platform. The platform separates compute from storage, integrates with the lakehouse through Synced Tables and Lakebase CDF, and uses Unity Catalog for governance; copy-on-write branching and autoscaling to zero are described as core infrastructure primitives. The post showcases partner-built, ready-to-deploy accelerators spanning technology, finance, marketing, sales, supply chain, human resources, customer service, and operations. Examples include PostgreSQL migration assessment, multi-agent Genie orchestration, stateful enterprise agents, autonomous data reliability, governed contact-center intelligence, and project operations management. These offerings package Lakebase patterns into migration controls, domain-specific solutions, and agent frameworks intended to accelerate modernization and reduce transformation complexity.
The post presents migration from Azure Synapse to Databricks as a way to reduce platform complexity while supporting data engineering, analytics, machine learning, real-time pipelines, and AI applications. It distinguishes migration paths for Dedicated SQL Pools, serverless SQL, Spark Pools, orchestration, governance, and downstream BI connectivity, noting that accumulated T-SQL logic and dependencies require the most redesign, testing, and validation. The proposed programme uses discovery and assessment with Lakebridge Profiler and Analyzer, then selects an architecture, sequences workloads, pilots a real use case, and scales migration in waves. It recommends automating code conversion, reconciling row counts, aggregations, hashes, and tolerances, avoiding direct Synapse distribution-to-Delta partition mappings, and planning compaction, enablement, and rollback. Reported examples include Casey’s halving operational data delivery times from eight hours to four and Italgas cutting workload costs by 73%.
Okta has introduced Journeys, a task-oriented documentation format designed to streamline how developers learn and implement identity management. Rather than requiring engineers to manually assemble information across disparate guides, articles, and API references, each Journey provides an end-to-end curriculum for a specific project. Content is organized into three distinct phases: foundational knowledge in Learn, architectural decision-making in Plan, and curated implementation steps in Build. The initial release includes six Journeys aimed at Okta Customer Identity builders who are implementing authentication, registration, branding, and user management. Future additions will address AI agent governance, integrations for the Okta Integration Network, and migrations from the Classic Engine to the Okta Identity Engine.
ClickHouse Agents is now available for all Managed Postgres services in ClickHouse Cloud, extending its plain-English agent interface to operational data. The beta service, powered by Claude and built on LibreChat, combines an agent builder and chat interface with a sandboxed code interpreter; it is read-only by default and supports MCP-compatible systems. Agents can inspect Postgres schemas, run read-only queries, return the SQL used, and reason across Postgres and ClickHouse by querying each engine independently before merging results. Additional skills guide migration from Postgres to ClickHouse through discovery, access-pattern analysis, schema design, type mapping, CDC ingestion, and validation, while other workflows use metrics, Query Insights, and EXPLAIN to rank performance fixes. Planned extensions include scheduled recommendations and safe automated remediation, but the current release focuses on exploration, analysis, monitoring, and guided database work.
Clever, an identity platform for K-12 education, faced a fragmented logging setup as its 400 services and 200 AWS Lambdas generated about 150 TB of uncompressed logs monthly. Datadog indexed only about 10% and retained it for three days, while the remainder was in Athena and required SQL. Clever migrated to ClickHouse Cloud on AWS, indexing all logs with 60-day retention and reporting 200× more searchable log volume at no additional cost, with 10× compression. Its design extracts frequently queried JSON fields into columns, stores remaining attributes in a map alongside raw JSON, uses skipping indexes, date partitions, a 60-day TTL, and reverse Unix timestamp ordering; this ordering produced an 8–10× query speedup. A custom query layer translates LogQL to SQL, while separate development and production services and multi-region deployment address isolation and availability.
An ETL migration with hundreds of stored procedures, schedulers, scattered permissions, and a warehouse renewal deadline needs workload-by-workload decisions rather than a single rewrite strategy. The framework assigns work among Databricks SQL, Spark Declarative Pipelines (SDP), and PySpark or Spark SQL notebooks. SQL tasks suit single statements, Unity Catalog-governed stored procedures handle procedural logic, SDP manages dependencies, retries, quality constraints, and batch-plus-streaming, while notebooks cover complex logic, ML feature engineering, integrations, and large or tightly controlled Spark workloads. It recommends four phases—assessment, quick wins, modernization, and optimization—using profiling, side-by-side validation, and parallel runs before retiring legacy systems. Migration tools can automate 60–80% of initial conversion, but architecture choices remain essential: the goal is consolidating orchestration, metadata, lineage, permissions, and validation rather than reproducing technical debt.
Vibe.co, recently acquired by Walmart, uses ClickHouse Cloud to provide real-time campaign reporting for thousands of Connected TV advertisers across billions of ad impressions. It migrated from Postgres after aggregation-heavy OLAP queries made its pre-aggregation layer a bottleneck, limiting interactive slicing and requiring new jobs for reporting use cases. The team evaluated self-hosted ClickHouse, another managed option, and ClickHouse Cloud, choosing the latter for operational simplicity, support from ClickHouse engineers, and separation of storage and compute with more predictable costs. Vibe uses refreshable materialized views for hot/cold tiering, keeping the last 30 days in a hot table; stored data grew from about 100 GB to over 2 TB without rearchitecting, while more than 90% of client campaign reports respond in under 100ms.
Netflix transitioned its managed batch compute infrastructure from a homegrown solution called Compute Managed Batch to Kueue on its Titus container platform. CMB previously relied on custom scheduling and admission-only fair sharing without preemption, making feature development cumbersome as the Kubernetes ecosystem evolved. To modernize the platform transparently, Netflix mapped internal tenants to Cohorts and leaf tenants to ClusterQueues and LocalQueues while routing jobs through a custom Kueue router. Kueue operates alongside existing Titus scheduling profiles rather than replacing the kube-scheduler, preserving cluster placement efficiency. The migration was completed in four weeks across millions of batch workloads, significantly increasing average resource utilization through preemption-based fair sharing.
Spyne’s CDC pipeline streamed changes from MySQL and MongoDB through self-managed Debezium, Kafka, and ClickHouse, but table onboarding required 1 to 1.5 hours of manual work and schema drift could silently exclude new columns. The team migrated to ClickPipes and ClickHouse Cloud, where pipelines can be configured in minutes and automated schema evolution applies upstream changes to corresponding ClickHouse tables without manual intervention or restarts. To manage snapshot pauses, Spyne isolated about 10 critical historical tables in dedicated ClickPipes and grouped lower-priority tables separately. The migration also required using FINAL in downstream queries to handle ReplacingMergeTree eventual consistency, while binlog gaps may still require full table resynchronization and higher costs. Spyne reports less operational toil, faster feature and analysis workflows, and improved data consistency, accepting reduced recovery control and query-time CPU trade-offs.
Data warehouse modernization addresses legacy systems that cannot scale efficiently with growing data volumes, real-time analytics, machine learning, and self-service access. The proposed roadmap spans two to four years for large estates, moving from assessment and architecture design through high-impact workload migration, governance embedding, and optimization rather than relying on a risky big-bang cutover. Its target architecture favors a lakehouse or enhanced cloud data warehouse, with open formats such as Apache Iceberg or Delta Lake, separate compute and storage, and Bronze, Silver, and Gold layers supporting incremental ELT and lineage. The source says modernization can reduce infrastructure maintenance costs by 30–50%, compress query latency from hours to seconds, and halve redundant ETL pipelines, while also improving governance for sensitive data and enabling BI, machine learning, and generative AI workloads on a shared foundation.
Open Electricity uses ClickHouse to serve live Australian energy data through a public platform and API spanning roughly one billion records. Migrating from Postgres and TimescaleDB addressed queries that could take more than five minutes, reducing a benchmark to just over three seconds and to 60 milliseconds with a materialized view. The new setup keeps heavy relational workloads in Postgres while ClickHouse handles analytics, using ReplacingMergeTree base tables and more than 20 materialized views for different aggregation patterns. It also uses substantially less hardware than the previous configuration: four cores, four GB RAM, and 200 GB disk versus 16 cores, 32 GB RAM, and 1.6 TB, while serving data added daily and API users.
Cisco Talos operates a threat-intelligence reputation service that classifies file hashes from Cisco product telemetry, enabling researchers to make security decisions about benign and potentially malicious artifacts. Previously, a custom pipeline moved data from Databricks Delta and S3 through EKS, where staging and aggregated tables supported an API; the self-managed design retained about a petabyte of compressed, replicated data on 32 high-compute nodes. The team rebuilt the system in four phases, using Terraform, SSO/SAML, sharded ClickPipes for a 12-month backfill, shadow traffic, and a validated cutover to ClickHouse Cloud on AWS. The resulting architecture uses EventBridge, SQS, ClickPipes, SharedMergeTree, object storage, and AWS PrivateLink, delivering less than a minute of queue lag, zero data loss, a 100% response match across more than 1,000 hashes, zero downtime, roughly 90% lower storage costs, and about 75% lower TCO while reclaiming over 300 engineer hours annually.
Postgres-to-Postgres migrations between providers can require manual schema handling, data movement, CDC, and cutover coordination, with risks around database objects and long-running loads. Postgres to Postgres ClickPipes, now in public beta, provides a fully managed migration service in ClickHouse Cloud designed to reduce operational overhead and support minimal downtime. It automates schema dump and restore, including secure connections, role conflicts, version differences, transactional application, retries, and UI or notification-based error reporting, while PeerDB supplies parallel initial loads and CDC capabilities. Independent pull and push processes use retries and alerting for transient errors, and the roadmap includes features to assist with cutover into Postgres by ClickHouse.
The best engineering reads, once a week.
