---
title: "Google Cloud"
description: "61 posts about Google Cloud, summarised, each linking to the original."
---

# Google Cloud
> 61 posts about Google Cloud, summarised, each linking to the original.

## Articles

### [Celebrando la Excelencia: Premios Globales de Emparejar de MongoDB 2025](https://yomu.fyi/post/celebrando-la-excelencia-premios-globales-de-emparejar-de-mongodb-2025.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Olivier Zieleniecki
- Published: Sep 18, 2025
- Type: Announcement

MongoDB presented its 2025 Global Partner Awards to honor partner organizations across cloud infrastructure, systems integration, generative AI, and enterprise data management. Microsoft received the Global Cloud Partner award for joint go-to-market solutions integrating MongoDB Atlas with native Azure services across healthcare, telecommunications, and financial services. Amazon Web Services earned the Global AI Cloud Partner award after delivering an Amazon Bedrock and MongoDB Atlas workflow implementation for Novo Nordisk that reduced processing time from twelve weeks to ten minutes. Confluent was recognized as Global Technology Partner for exceeding 550 joint deployments leveraging Apache Kafka for real-time data streaming and event-driven AI architectures. Other honored partners include Google Cloud for joint sales development programs, Accenture for software engineering centers of excellence, LangChain for retrieval-augmented generation tooling, and IBM for Watsonx.ai integrations.


### [Hommage à l’excellence : MongoDB Global Partner Awards 2025](https://yomu.fyi/post/hommage-a-l-excellence-mongodb-global-partner-awards-2025.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Olivier Zieleniecki
- Published: Sep 18, 2025
- Type: Announcement

MongoDB announced the recipients of its 2025 Global Partner Awards, recognizing enterprise collaboration across cloud infrastructure, artificial intelligence, and systems modernization. Microsoft received the Global Cloud Partner award for joint integrations with Azure, while Amazon Web Services earned the Global AI Cloud Partner award following deployments combining Amazon Bedrock and MongoDB Atlas. Google Cloud was recognized for joint go-to-market initiatives, and Accenture received honours for establishing a dedicated MongoDB Center of Excellence within its software engineering division. Additional recognitions included Confluent for event-driven streaming deployments exceeding 550 customer implementations, BigID for data governance, gravity9 for modernization services, and Alibaba Cloud for managed database services. IBM and Accenture Federal Services were also acknowledged for strategic enterprise impact across mainframe architectures and public sector programs.


### [SAIR: Accelerating Pharma R&D with AI-Powered Structural Intelligence](https://yomu.fyi/post/sair-accelerating-pharma-r-d-with-ai-powered-structural-intelligence.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Arman Zaribafiyan, Georgia Channing, Rudi Plesch, Zane Beckwith
- Published: Sep 2, 2025
- Type: Announcement

SandboxAQ released the Structurally Augmented IC50 Repository (SAIR), an open-source dataset containing 5.24 million computationally co-folded 3D protein-ligand structures paired with empirical IC50 binding potency data. The repository addresses training data scarcity in structure-based drug discovery, where over 40 percent of the included target proteins lack experimental structures in the Protein Data Bank. To construct the dataset, engineers executed over 130,000 GPU hours of the Boltz1 co-folding model on 760 NVIDIA H100 processors hosted via NVIDIA DGX Cloud on Google Cloud Platform. Infrastructure optimizations maintained over 95 percent GPU compute utilization, compressing the generation timeline from three months to three weeks. Quality validation using PoseBusters confirmed that 97 percent of the predicted complexes met physical plausibility and chemical sanity standards.


### [Augmented commerce: Machine learning at Shopify](https://yomu.fyi/post/augmented-commerce-machine-learning-at-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 4, 2025
- Type: Explainer

Shopify frames commerce as a complex, continuous optimization problem in which merchants have different definitions of success and need support across the business lifecycle. Its machine-learning work uses a historical dataset covering millions of businesses to optimize product metadata, transaction fraud assessment, merchant assistance, product discovery, capital decisions, behavioral modeling, and search query rewriting. The systems include fine-tuned Qwen multimodal models, LLaMa models and MCPs for Sidekick, Nomic embeddings for billions of products, a tabular transformer for forecasting merchant GMV, and experiments with HSTU sequence models. To support this work, Shopify partners with GCP, Nebius, CentML, and Toloka, while providing access to GPUs, data services, frontier models, and internal tooling; the post also presents these efforts as part of its hiring pitch.


### [Implementing scalable metered billing with Stripe: How Edgee handles billions of events](https://yomu.fyi/post/implementing-scalable-metered-billing-with-stripe-how-edgee-handles-bi.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Alex Casalboni
- Published: Jun 24, 2025
- Type: Problem & solution

Edgee needed usage-based billing for a business combining flat-rate subscriptions with charges for proxy requests and delivered data events, while processing billions of monthly requests across more than 100 edge locations. Its architecture captures web data at the edge through a proxy and records raw metering data centrally in Google BigQuery. To control several terabytes of data, Edgee aggregates usage by tenant and sends hourly Stripe meter events through AWS Lambda, with graduated pricing and idempotency handled by Stripe. A Go configuration defines meters, prices, flat-rate plans, and pricing tiers, while retry logic and idempotent updates support resilience during disruptions. The resulting billing system remains stateless and low-latency while providing accurate monthly invoicing and a simple hosted customer interface.


### [Unlocking powerful analytics with Stripe Data Pipeline and Google BigQuery](https://yomu.fyi/post/unlocking-powerful-analytics-with-stripe-data-pipeline-and-google-bigq.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Sushant Jain
- Published: Mar 21, 2025
- Type: Tutorial

As businesses scale, the post addresses the need to move Stripe revenue data into BigQuery for recurring reporting and analysis. It describes configuring Stripe Data Pipeline to export transactions, invoices, subscriptions, and payments to Google Cloud Storage, then using a Python-based Google Cloud Run Job to discover the latest snapshot through data\_load\_times.json and enumerate each table’s Parquet files. The loader sends those files to BigQuery with the Google Cloud clients, creates missing tables, uses WRITE\_TRUNCATE to retain the latest snapshot, and parallelizes table loads with ThreadPoolExecutor while collecting errors. After deployment and a test execution, the workflow is verified in BigQuery and scheduled to run every six hours, providing an automated path to refreshed Stripe data for analytics.


### [How Shopify improved consumer search intent with real-time ML](https://yomu.fyi/post/how-shopify-improved-consumer-search-intent-with-real-time-ml.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 18, 2024
- Type: Explainer

Shopify describes how it built near-real-time text and image embedding pipelines to support semantic search, replacing keyword matching with representations that enable similarity-based retrieval. Using Google Cloud’s Dataflow and Apache Beam, the pipelines process roughly 2,500 embeddings per second, preprocess images, run GPU-backed inference, and write outputs to both a data warehouse and an event topic for downstream ingestion. The engineering discussion focuses on trade-offs among memory, throughput, latency, and cost: reducing harness threads cut memory use about 2.6x and allowed a return to cheaper workers, while model sharing reduced throughput. For batching, bursty input produced bundles of one, and stateful batching was rejected because its shuffle increased latency; Dataflow’s per-process model instances instead provided enough parallelism to saturate GPUs.


### [Enhancing Cloud Usage Forecasting, Monitoring & Optimizing](https://yomu.fyi/post/enhancing-cloud-usage-forecasting-monitoring-optimizing.md)
- Company: [Etsy](https://yomu.fyi/company/etsy.md)
- Author: Anthony Tambasco
- Published: Jun 17, 2024
- Type: Explainer

After Etsy completed its 2020 migration from an on-premise data center to Google Cloud Platform (GCP), the resulting FinOps team developed a global approach to forecasting, monitoring, and optimizing cloud spend. Costs are grouped into core infrastructure, machine learning and product enablement, data enablement, and Dev, with trailing Cost Per Visit (CPV) serving as the primary forecasting metric after regression tests found visits most correlated with costs. BigQuery and Looker dashboards, alerts, weekly Finance comparisons, and recurring cost-owner and leadership readouts support variance detection, while costs are tracked centrally without chargebacks to preserve flexibility. Optimization work combines FinOps, engineering, and Architecture collaboration, producing GCS storage reductions through monitoring, automation, TTLs, and Intelligent tiering, plus compute, model-deployment, and network compression improvements. The next priority is better visibility into shared machine learning platform spend to improve experiment ROI analysis and identify optimization opportunities.


### [The So-fine Real-time ML Paradigm](https://yomu.fyi/post/the-so-fine-real-time-ml-paradigm.md)
- Company: [Etsy](https://yomu.fyi/company/etsy.md)
- Author: Kyle Gallatin
- Published: Sep 5, 2023
- Type: Problem & solution

During Etsy’s CodeMosaic hackathon, a team investigated stateful training and online learning to reduce the roughly 40-hour delay between user actions and model-weight updates. They split the work across real-time training data, an incremental-learning service, and evaluation, using Etsy’s Beacon Main Kafka stream, Kafka SQL, Rivulet, and TensorFlow-based Ads models. The data team joined multiple sources despite Avro schema difficulties, while the service team loaded an older model and nearly implemented incremental weight updates; the planned head-to-head evaluation against daily batch training was not completed. The project estimated that replacing daily cold-start training for four ads models could save about $212,000 annually in Google Cloud costs, with possible metric gains from earlier events. Production would still require infrastructure and safeguards against sudden performance degradation.


### [The Complex Data Models Behind Shopify's Tax Insights Feature](https://yomu.fyi/post/the-complex-data-models-behind-shopify-s-tax-insights-feature.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Feb 8, 2023
- Type: Explainer

Shopify Tax’s Tax Insights feature provides merchants with state-level information about sales-tax liability and approaching liability, amid U.S. rules that vary by jurisdiction, product, threshold, and evaluation period. The implementation modified existing data models, added four new ones, and represented changing tax rules with lookup data containing valid\_from and valid\_to dates, including thresholds, date ranges, inclusions, exclusions, and inclusive or exclusive comparisons. Teams prototyped models in BigQuery SQL, implemented them in PySpark and Spark jobs, and combined merchant monthly taxable-sales aggregates with the rule model to evaluate economic nexus; the architecture moved source data through GCS and a warehouse into a final model. Each run publishes merchant-keyed insights to Google Cloud Bigtable, where the core application retrieves them for the frontend, while warehouse data supports analytics and other processes.


### [Lessons Learned From Running Apache Airflow at Scale](https://yomu.fyi/post/lessons-learned-from-running-apache-airflow-at-scale.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 23, 2022
- Type: Explainer

Shopify describes operating Apache Airflow 2.2 on Kubernetes with the Celery executor and MySQL 8 for data extraction, machine-learning training, Iceberg maintenance, and DBT modeling. In its largest environment, Shopify runs over 10,000 DAGs, averages over 400 tasks running at a given moment, and executes over 150,000 runs per day, while facing slow file access, metadata growth, ownership tracking, DAG-author control, traffic bursts, and resource contention. The authors describe replacing per-pod GCSFuse access with an in-cluster NFS volume synchronized to GCS, retaining GCS as the user-facing source of truth; they also use 28-day metadata retention, a YAML manifest, DAG policies, and scheduling controls including pools, priority weights, and Celery queues. These measures are presented as ways to improve file performance, limit database growth, identify owners, enforce constraints, and reduce workload interference, though the retention policy does not support durable history-dependent features such as long-running backfills.


### [7 Tips For Optimizing Apache Flink Applications](https://yomu.fyi/post/7-tips-for-optimizing-apache-flink-applications.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Mar 8, 2022
- Type: Tutorial

Shopify presents lessons from operating large, stateful Apache Flink applications on Kubernetes with Google Kubernetes Engine, RocksDB, and checkpoints and savepoints stored in Google Cloud Storage. The guidance begins with profiling tools, including async-profiler, VisualVM, jemalloc with jeprof, and Eclipse Memory Analyzer, to investigate CPU, heap, native-memory, and leak-related problems. Disabling Kryo fallback exposed unsupported serializers such as Scala BigDecimal and Scala algebraic data types; replacing them with supported alternatives increased throughput by 20 percent. The post also recommends separate configuration profiles for high-throughput backfills and low-latency steady state, while describing a RocksDB native-memory failure in which disabling the block cache stopped repeated out-of-memory crashes without affecting application performance.


### [Shopify's Path to a Faster Trino Query Execution: Infrastructure](https://yomu.fyi/post/shopify-s-path-to-a-faster-trino-query-execution-infrastructure.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 9, 2021
- Type: Problem & solution

Shopify’s Trino-based interactive query infrastructure faced inconsistent latency and operational failures as workloads grew to 15 Gbps and more than 300 million rows per second. The team analyzed query volumes, resource-heavy datasets, CPU wall time, and failure scenarios, then used a Trino Query Replicator to reproduce historical degradation and isolate issues involving compressed Kafka JSON, shared cluster classes, resource allocation, JVM settings, and dataset statistics. They added workload-specific and ephemeral clusters, reduced concurrent queries, applied recommended JVM recompilation settings, and limited task drivers to 16; they also scaled worker pods to 61 cores and 220 GB memory, although scaling alone was unstable. The infrastructure changes produced a stable environment and reduced P95 execution time to below 30 seconds, moving more queries into the zero-to-five-second range, while the five-second target remained a future goal.


### [My Journey to a serverless transformers pipeline on Google Cloud](https://yomu.fyi/post/my-journey-to-a-serverless-transformers-pipeline-on-google-cloud.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Dominici
- Published: Mar 18, 2021
- Type: Tutorial

The post documents Maxence Dominici’s effort to deploy a Transformers sentiment-analysis pipeline on Google Cloud for classifying customer reviews from Discord. After considering AI Platform Prediction and App Engine, he chose Cloud Run with a Docker image because custom checkpoint handling, dependencies, and resource requirements made the other options unsuitable. The implementation uses a PyTorch DistilBERT model fine-tuned on SST-2, Flask, Gunicorn, and a GET request protected by an optional API key; the shown handler retrieves a review but passes the hard-coded text “that was great” to the classifier. Cloud Run is configured with 4 GB of memory, one worker, and one thread to limit memory pressure and billing. Requests take under five seconds including model loading, while cold starts may add about ten seconds, and the author reports that the pretrained model’s accuracy was already impressive.


### [Hugging Face on PyTorch / XLA TPUs](https://yomu.fyi/post/hugging-face-on-pytorch-xla-tpus.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: ds, Lysandre
- Published: Feb 9, 2021
- Type: Announcement

PyTorch / XLA integrates Cloud TPU training into Hugging Face transformers while preserving the standard Trainer interface. The implementation adds an xla device type and uses lazy execution to trace intermediate representation graphs before lowering them to XLA Higher Level Operations for compilation. Parallel data loading through MpDeviceLoader overlaps CPU tracing with TPU execution, while xm.optimizer\_step coordinates gradient consolidation across data-parallel replicas. Checkpoints use xm.save to transfer tensors via CPU storage and restrict disk writes to the master process. Benchmarks on a Cloud TPU v3-8 system training bert-large-uncased on WikiText-103 demonstrate completion times of 178.4 minutes in FP32 precision and 106.4 minutes in BF16 precision.


### [Managing Google Cloud Platform Project-Wide SSH Keys](https://yomu.fyi/post/managing-google-cloud-platform-project-wide-ssh-keys.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jan 27, 2021
- Type: Problem & solution

Google Cloud project-wide SSH keys can remain in project metadata indefinitely after a user connects to a VM, potentially allowing matching private-key holders to access Linux VMs with root access unless project-wide keys are blocked. Shopify developed SSH-Pruner because OS Login did not work with Google Kubernetes Engine, while deleting all keys also removed Google’s keys, disrupted scripts, and created slow metadata updates. The new Go application uses Google Cloud APIs to find organization projects with Compute Engine enabled, reads the “ssh-keys” metadata, parses entries, retains unexpired keys, writes pruned common instance metadata, and records readable logs; Google-managed “sshKeys” entries are skipped. SSH-Pruner was already running as a cron job on some projects, with a slow rollout planned to limit workflow disruption.


### [Capacity Planning at Scale](https://yomu.fyi/post/capacity-planning-at-scale.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 3, 2020
- Type: Explainer

Shopify’s BFCM preparation combines traffic forecasting, resource planning, and platform-wide scalability testing to reduce the risk of outages during its busiest sales period. The engineering team worked with data scientists to model expected traffic, reviewed historical numbers, added safety margins, and created a master Google Cloud resource plan covering CPUs, storage, machine types, locations, and project-level quotas. Because 2020 brought unusual uncertainty from COVID-19-driven changes in online commerce, Shopify modeled multiple demand scenarios and chose the more aggressive growth cases, while retaining buffers for changes and regional failover. It also ran the Oktoberfest scale-up on Google Kubernetes Engine, using internal load-testing tooling powered by go-lua and embedded Lua to simulate storefront browsing, cart additions, checkout, and mock payment processing. The exercise exposed bottlenecks, which teams fixed before repeating the test; the successful rerun increased confidence for BFCM and reinforced a longer-term aim of more automated, dynamic, and resilient systems.


### [How to Build a Production Grade Workflow with SQL Modelling](https://yomu.fyi/post/how-to-build-a-production-grade-workflow-with-sql-modelling.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 19, 2020
- Type: Problem & solution

Shopify’s Starscream data platform handled 76,000 jobs and wrote 300 terabytes daily, but its PySpark workflow was slow and cumbersome for commonplace reporting. An audit found that about 70 percent of PySpark jobs were full batch queries that did not require generalized computing, while users often wrote SQL before translating it into Python. The team built Seamster with dbt and Google BigQuery, adding source and model structures, ownership metadata, Python-based unit tests with fixed mock inputs, CI validation, and warehouse rules. A base layer isolates raw-source changes, while models are organized into base, application-ready, and presentation layers and defined through SQL, schema, README, and test files. After a multiweek beta, most users reported shipping models in days rather than weeks, although dbt’s incremental support could not safely handle some late-arriving-data, key-resolution, rebuild, and very large-model cases.


### [The Evolution of Kit: Automating Marketing Using Machine Learning](https://yomu.fyi/post/the-evolution-of-kit-automating-marketing-using-machine-learning.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Mar 4, 2020
- Type: Explainer

Kit began as a heuristic, rules-based virtual employee that simplified Facebook advertising for Shopify merchants by selecting products and budgets while defaulting other settings to best practices. The post describes its evolution into an artificially intelligent assistant that uses merchant spending behavior and store signals such as traffic and sales to recommend marketing actions. Kit's machine-learning design combines regression to predict likely spending with classification to estimate whether a budget and store state are likely to generate sales; training uses feature engineering, Apache Spark, Google Cloud ML Engine, and monitored model metrics. For real-time predictions, Bigtable stores engineered features, while a Tornado web container delegates model computation to TensorFlow Serving. The resulting system powered one third of Kit's marketing campaigns.


### [How Shopify Manages Petabyte Scale MySQL Backup and Restore](https://yomu.fyi/post/how-shopify-manages-petabyte-scale-mysql-backup-and-restore.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 1, 2019
- Type: Problem & solution

Shopify manages a petabyte-scale fleet of MySQL servers across replica-sets, or shards, in three Google Cloud Platform regions, where file-based Xtrabackup archives made shard restores take more than six hours. To reduce that Recovery Time Objective, the team built Kubernetes CronJobs around the Compute Engine Persistent Disk snapshot API, scheduling snapshots every 15 minutes while accounting for regions, zones, instance roles, and MySQL consistency variables. Persistent Disk snapshots take about 20 minutes initially and typically under 10 minutes incrementally; restoring the latest snapshot to a new volume and starting MySQL, including InnoDB instance recovery and replication-lag recovery, typically brings RTO below 30 minutes. Retention tooling keeps the latest two snapshots plus policy-defined dailies and weeklies, while verification checks startup, GTID auto-positioning, and InnoDB page corruption; offsite tooling compresses, encrypts, and transfers exported data to another provider.


[Newer posts](https://yomu.fyi/topic/gcp/page/2.md) · [Older posts](https://yomu.fyi/topic/gcp/page/4.md)