Loading…
Topic
MySQL
36 posts about MySQL. Every summary links to the original.
- 36 posts
- 9 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Inside Hyperdrive: How we solved a rare Hyperdrive MySQL caching bug
Hyperdrive is Cloudflare’s distributed database connection pooler for Workers; its MySQL support became GA after a beta that uncovered over 20 bugs, including a rare caching issue. The most challenging bug caused consistently empty query responses 0.001% of the time on cache-enabled instances, requiring deep protocol analysis. The root cause was that the Client and Endpoint nodes share a data center cache but use different MySQL capability flags, so a cache entry written by one node could be encoded with capabilities that the other node cannot correctly translate. When the reading node encountered such a mismatched entry, it either sent an early EOF or hung, resulting in empty rows for the Worker. The fix was to always write a canonical version of the query result bytes encoded with Hyperdrive’s own capabilities, regardless of which node stored it, which resolved the issue and raised reliability to the 99.99% target.
Loading Parquet data into MySQL with ClickHouse
The post demonstrates how to move Parquet data into MySQL and query the resulting table using ClickHouse table functions. It sets up MySQL 8.4 in Docker, runs clickhouse-local with a configuration file and named collection, and inspects StackOverflow votes stored in a 2024 Parquet file on S3. An INSERT INTO FUNCTION mysql statement loads the file into a MySQL InnoDB table with 12,296,977 rows, while subsequent examples query MySQL from ClickHouse using either translated ClickHouse syntax or an opaque MySQL SQL string. The post also shows named-collection credentials and a query that identifies posts with both upvotes and downvotes, returning ten ranked results. It concludes that this workflow stays within ClickHouse and notes that table functions also support Postgres, SQLite, and MongoDB.
MySQL CDC connector for ClickPipes is now Generally Available
MySQL CDC connector for ClickPipes is now Generally Available, enabling replication from MySQL and MariaDB into ClickHouse Cloud for analytical workloads. The release adds automatic parallel snapshotting: ClickPipes selects a logical partition key from a table’s primary key, including string and UUID types, and distributed processing reduced a benchmarked 1TB snapshot from over a day on one thread to about an hour across 32 workers. Production safeguards now require GTID-based replication and at least 72 hours of binlog retention, while new CPU, memory, network, and source, destination, and end-to-end lag metrics improve troubleshooting. MySQL CDC ClickPipes also support OpenAPI and Terraform 3.14.0+, are available across all ClickHouse Cloud service tiers, and began billing on September 1, 2026.
Using Triggers On Schemaless, Uber Engineering's Datastore Using MySQL
Schemaless triggers provides a scalable, fault-tolerant, and lossless way to listen for changes in Schemaless, Uber Engineering’s datastore using MySQL, and drives workflows including trip billing, analytics-warehouse ingestion, and cross-datacenter replication. Schemaless stores immutable cells and all their versions, allowing each shard to function as a partitioned change log with monotonically increasing added IDs while retaining random-access APIs. The framework tails shards through get_cells_for_shard, assigns them to worker processes under a leader, persists offsets in shared storage, and invokes column-specific callbacks such as bill_rider; handlers must be idempotent because retries can occur. Failed workers’ shards are reassigned, repeatedly failing cells can move to a separate queue, processing continues during leader or shared-storage outages although redistribution pauses, and replicas insulate triggers from database-node failures.
The Architecture of Schemaless, Uber Engineering's Trip Datastore Using MySQL
Schemaless is Uber's scalable, fault-tolerant datastore whose immutable cells are identified by row key, column name, and ref key, while JSON content remains schemaless and supports eventually consistent secondary indexes. Its worker nodes accept HTTP requests, route them to storage nodes, and aggregate results, while storage data is partitioned into typically 4096 shards, each replicated across one master and two minions in multiple data centers. Writes go to the master and replicate asynchronously, with idempotent retries, circuit breakers, failover, and configurable reads from masters or minions. To reduce loss during master failure, buffered writes persist each request to a primary and secondary cluster, though buffered data may not be immediately readable. Each shard maps to a MySQL database, where an auto-increment added_id primary key supports linear insertion and trigger processing, and MessagePack with ZLib compresses cell JSON blobs.
Building a serverless AI assistant at Pelago: concept to care in two weeks
Pelago engineered a serverless, event-driven AI assistant on AWS within two weeks to provide substance use recovery coaches with contextual response suggestions. Operating under strict Protected Health Information constraints, the system maintains clinical safety through human-in-the-loop review and keeps data confined to an Amazon VPC. Incoming messages flow via AWS AppSync and Amazon DynamoDB to an Amazon SNS topic, which fans out to independent AWS Lambda functions. The asynchronous Chat Assistant Lambda queries history, invokes Amazon Bedrock, and persists suggestions into Amazon RDS MySQL in under 10 seconds. When coaches access a conversation via Amazon API Gateway, pre-generated suggestions load in under 100 milliseconds.
Session revocations at scale
Canva gateways check in-memory session revocations on every request, but pulling over a million revocations from MySQL during deployments created severe database stampedes. To eliminate this bottleneck, the team transitioned to storing a twelve-hour sliding window of revocations in Amazon S3 partitioned into 30-minute chunks. Each revocation is packed into a 16-byte binary structure within a sorted flat array, allowing gateways to query records via direct binary search without object deserialization. Asynchronous workers batch updates into S3 using ZooKeeper leader election and conditional PUT requests for optimistic concurrency control. This architecture reduced the in-memory cache footprint by 87.5% and allowed the team to reduce their MySQL database to just two read replicas.
How Spyne simplified their CDC pipeline with ClickPipes and ClickHouse Cloud
Spyne’s CDC pipeline streamed changes from MySQL and MongoDB through self-managed Debezium, Kafka, and ClickHouse, but table onboarding required 1 to 1.5 hours of manual work and schema drift could silently exclude new columns. The team migrated to ClickPipes and ClickHouse Cloud, where pipelines can be configured in minutes and automated schema evolution applies upstream changes to corresponding ClickHouse tables without manual intervention or restarts. To manage snapshot pauses, Spyne isolated about 10 critical historical tables in dedicated ClickPipes and grouped lower-priority tables separately. The migration also required using FINAL in downstream queries to handle ReplacingMergeTree eventual consistency, while binlog gaps may still require full table resynchronization and higher costs. Spyne reports less operational toil, faster feature and analysis workflows, and improved data consistency, accepting reduced recovery control and query-time CPU trade-offs.
Ten years of ClickHouse in open source
ClickHouse’s tenth-anniversary history describes its path from an internal web-analytics experiment to an open-source analytical database with more than 2,000 contributors. The original problem was processing rapidly growing, real-time logs while answering flexible reports quickly; tested MySQL extensions and standalone analytical databases did not meet the stated scale of 100 billion records per day and 500 columns. The project built column-oriented storage, lightweight compression, custom aggregation structures, background merges, and query support for grouping, aggregation, filtering, and sorting. Released publicly on June 15, 2016, ClickHouse is presented as a level-three open-source project with public contribution guidelines, code review, testing, CI, documentation, and a roadmap, while contributors receive credit even when implementations are substantially rewritten.
Modernizing Artifact Storage at Uber
Uber’s legacy artifact platform used two five-node clusters with local disks, asynchronous replication, and GeoDNS failover to support builds across monorepos and microrepos. As artifact volume grew, limited node capacity, inconsistent replication, manual hardware evacuation, and risky multi-terabyte upgrades created operational and data-loss risks. Uber moved to a managed SaaS platform with multi-region availability, cross-region replication, and cloud-native blob storage, then added an internal validation proxy in front of it. The proxy uses MySQL metadata and conditional requests, including If-None-Match and If-Modified-Since, to validate artifacts against the SaaS origin while reusing unchanged bytes; failures can fall back to passthrough, preserving correctness at higher latency and egress cost. The design reduced operational risk, improved observability, and made artifact delivery more cost-efficient at scale.
Ready for your busiest day: How we scale
Fin describes how its customer service platform is designed to absorb high-volume events while preserving customer isolation, availability, and operational control. At daily peaks, it handles over 150,000 customer requests per second and more than 70,000 asynchronous requests per second; its source-of-truth data spans 128 Vitess shards, while Elasticsearch supports over 650TB, 1.7 trillion documents, and peaks above 40,000 requests per second. The architecture favors AWS primitives, Vitess managed by PlanetScale, online and incremental migrations, fair queues, feature flags, heartbeat metrics, and automatic rollback, with roughly 250 production ships per workday and merge-to-production time under 10 minutes. The stated conclusion is that scale depends less on headline capacity than on clear scaling levers, tenant fairness, customer-outcome monitoring, reversible change, and routine infrastructure improvement without planned customer-impacting downtime.
We replaced Redis with MySQL for inventory reservations—and it scaled
Shopify replaced its Redis-based inventory reservation system with MySQL to unify reservations with the inventory ledger and provide ACID coordination between payment and inventory updates. The new design stores one row per sellable unit in a bounded pool of up to 1,000 rows per item/location combination, uses MySQL 8's SKIP LOCKED to avoid waiting on locked units, and replenishes depleted pools. Composite primary keys reduced row locks, READ COMMITTED avoided gap locks that interfered with replenishment, and consistent lock ordering prevented deadlocks. The investigation found connection-pool usage elsewhere in checkout, rather than reservation query speed alone, was the immediate ceiling; cleanup removed 50% of primary-database reads and 33% of transactions, while configuration changes added headroom. Shopify validated the design in shadow mode, then rolled it out gradually with a Redis rollback path, meeting high-throughput targets during peak 2025 traffic.
uAct - Unified Action Platform
Uber built uAct to consolidate action items and approval requests generated by its many internal applications, addressing notifications that could be lost among email and follow-ups. It combines real-time fetching with scheduled offline ingestion through Apache Camel, stores requests, and merges them in received and sent views. Users can act through deep links or within uAct, where the Action Dispatcher authenticates and authorizes actions and the Audit Trail Provider records them for compliance. Its onion-architecture backend uses orchestrators, application-specific mappers, MySQL for requests and CacheFront for preferences, while parallelized paginated calls and strict timeouts support performance as integrations grow. The platform has been implemented across Uber and also provides configurable views, hiding, manager filters, observability, privacy controls, and planned Slack notifications, reporting, nudging, and additional integrations.
Terraform & OpenAPI for ClickPipes is now Generally Available
ClickHouse Cloud announced the general availability of ClickPipes resources in Terraform and OpenAPI, expanding infrastructure-as-code support across its managed data ingestion platform. The release resolves previous beta gaps by adding full connector coverage, including Change Data Capture for Postgres, MySQL, and MongoDB, alongside support for BigQuery, Azure Blob Storage, and reverse private endpoints. Operators can provision new data ingestion pipelines, pause or resume sync operations, and scale resources programmatically through OpenAPI endpoints or the ClickHouse Terraform provider starting at version 3.14.0. Existing ClickPipes can also be imported into Terraform state using native import blocks and automated configuration generation. Although SSH tunneling remains under active development, nearly all UI capabilities now map cleanly to programmatic interfaces.
Migrating Etsy’s database sharding to Vitess
Etsy migrated its long-running MySQL shard management to Vitess vindexes after an architecture spanning about 1,000 tables and shards, 425 TB of data, and roughly 1.7 million requests per second. The legacy ORM used a single index database to map records to shards, creating a scaling bottleneck and single point of failure. To avoid moving existing data, engineers built custom SQLite lookup and hybrid vindexes, changed new shard assignments to match Vitess’s hash algorithm, and introduced routing incrementally table by table with traffic ramps, rollback, and query compatibility testing. After five years, about 2,500 pull requests, and 6,000 queries, the migration removed the index database dependency, made scaling operations take days rather than months, simplified developer access, and kept database performance largely unchanged without a mass data migration.
Mitigating Deadlocks in high concurrency environments
MySQL requires locks before updating existing records, and concurrent upserts that mix new and existing rows can deadlock when gap locks overlap, even when processes target different records. The post describes ETL jobs partitioned by account whose updates nevertheless contend because sequential primary-key ordering places records in shared index gaps. It proposes replacing the existing primary key with a composite key of account and id, aligning InnoDB's physical row ordering with the processes' update partitions; gap locks remain necessary, but are largely compartmentalized within each process. An additional unique index on id preserves standalone record identification and compatibility with existing joins, while application and ETL logic remain unchanged. The reported outcome was removal of artificial delays and significantly higher import throughput through reduced job wait time.
Horizontally scaling the Rails backend of Shop app with Vitess
Shop app’s hockey-stick growth pushed its Ruby on Rails backend and MySQL primary datastore toward capacity, with multi-terabyte storage, week-long schema migrations, and throttled background jobs. After splitting databases and scaling surrounding systems, the team adopted Vitess to shard and re-shard MySQL while retaining SQL compatibility and self-managed primary datastores. They organized user-owned data into a sharded “users” keyspace using user_id, kept other data in an unsharded “global” keyspace, and built application-layer query verifiers to validate routing, distribution, and transaction correctness before migration. The move made schema migrations take hours instead of weeks, removed the reported capacity-driven throttling, and enabled further scaling by adding shards, while introducing concepts such as Sequences, Vindexes, and VSchemas that developers must learn.
Safely Adding NOT NULL Columns to Your Database Tables
Shopify’s Database Migrations team investigated whether Large Hadron Migrator (LHM) can safely add NOT NULL columns to existing MySQL tables with minimal downtime. LHM creates a shadow table, installs triggers for INSERT, UPDATE, and DELETE operations, and copies records in batches before renaming the tables. The experiments varied whether the new column had a DEFAULT value or UNIQUE index and whether MySQL used strict or non-strict mode, evaluating trigger compatibility and record counts. A NOT NULL column with a DEFAULT was backward compatible, but adding a UNIQUE index could cause data loss; without a DEFAULT, strict mode broke compatibility while non-strict mode used implicit defaults, which may be undesirable.
How we store and process millions of orders daily
The Grab Order Platform processes millions of food and mart transactions daily, requiring high throughput, fault tolerance, and reduced cloud costs across transactional and analytical workloads. To meet these demands, the engineering team decoupled their database architecture by using Amazon DynamoDB for critical OLTP queries and MySQL RDS for historical OLAP queries. DynamoDB handles online order lifecycles with strong consistency, utilizing sparse Global Secondary Indexes for ongoing orders and TTL configurations to limit storage growth. Updates propagate asynchronously to MySQL RDS through a Kafka ingestion pipeline backed by Amazon SQS retries and timestamp-based version checks. This dual-database approach isolated core transaction availability from analytical queries and delivered significant cloud cost savings.
How Kafka Connect helps move data seamlessly
Grab's real-time data platform team, Coban, implemented a managed Kafka Connect ecosystem on Kubernetes to streamline moving data in and out of Apache Kafka. To resolve dual-write consistency issues and capture pre- and post-change data, the team integrated Debezium connectors to capture MySQL binlog events and accommodate database DDL migrations. For disaster recovery and stream migrations, Coban deployed MirrorMaker2 connectors managed via Terraform to handle message mirroring and consumer offset translation across AWS regions. Additionally, they developed a custom converter utilizing Confluent Schema Registry to transform Protobuf-serialized Kafka records into JSON for ingestion into Azure Event Hubs. This architecture enabled zero-downtime cluster migrations and robust cross-region disaster recovery.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.