---
title: "Shopify's Path to a Faster Trino Query Execution: Infrastructure"
description: "Shopify’s Trino-based interactive query infrastructure faced inconsistent latency and operational failures as workloads grew to 15 Gbps and more than 300 million rows per second. The team analyzed query volumes, resource-heavy datasets, CPU wall time, and failure scenarios, then used a Trino Query Replicator to reproduce historical degradation and isolate issues involving compressed Kafka JSON, shared cluster classes, resource allocation, JVM settings, and dataset statistics. They added workload-specific and ephemeral clusters, reduced concurrent queries, applied recommended JVM recompilation settings, and limited task drivers to 16; they also scaled worker pods to 61 cores and 220 GB memory, although scaling alone was unstable. The infrastructure changes produced a stable environment and reduced P95 execution time to below 30 seconds, moving more queries into the zero-to-five-second range, while the five-second target remained a future goal."
---

# Shopify's Path to a Faster Trino Query Execution: Infrastructure

[Shopify](https://yomu.fyi/company/shopify) · 2023-10-18 · Jul 9, 2021

**Type:** Problem & solution

## Summary

Shopify’s Trino-based interactive query infrastructure faced inconsistent latency and operational failures as workloads grew to 15 Gbps and more than 300 million rows per second. The team analyzed query volumes, resource-heavy datasets, CPU wall time, and failure scenarios, then used a Trino Query Replicator to reproduce historical degradation and isolate issues involving compressed Kafka JSON, shared cluster classes, resource allocation, JVM settings, and dataset statistics. They added workload-specific and ephemeral clusters, reduced concurrent queries, applied recommended JVM recompilation settings, and limited task drivers to 16; they also scaled worker pods to 61 cores and 220 GB memory, although scaling alone was unstable. The infrastructure changes produced a stable environment and reduced P95 execution time to below 30 seconds, moving more queries into the zero-to-five-second range, while the five-second target remained a future goal.

## Context

As Shopify’s data volume and interactive workloads grew, Trino queries became inconsistent and sometimes stalled because clusters were overloaded. The target was a P95 query latency below five seconds, approximately a 30-fold reduction, while some queries were taking one to five minutes.

## Approach / What changed

Shopify investigated resource usage and failure scenarios, using a Trino Query Replicator to reproduce historical load. The team introduced workload-specific and ephemeral clusters, routed experiment queries separately, reduced concurrent queries, applied JVM recompilation settings, limited task drivers per query to 16, and increased worker capacity. Horizontal and vertical scaling alone did not produce stable results.

## Takeaways

- A Trino Query Replicator let Shopify recreate past performance-degradation states and investigate the error classes and workloads responsible for slow queries.
- Routing experiment queries to a dedicated cluster reduced interference with ad hoc workloads, while tooling also enabled users to create temporary clusters that were automatically removed after a defined TTL.
- Setting task.max-drivers-per-task to 16 limited a query with about 29,000 running splits to around 7,200 active splits, preventing it from monopolizing the cluster.

**Tags:** [Google Cloud](https://yomu.fyi/topic/gcp), [Kubernetes](https://yomu.fyi/topic/kubernetes), [Monitoring](https://yomu.fyi/topic/monitoring), [Performance](https://yomu.fyi/topic/performance), [Scalability](https://yomu.fyi/topic/scalability)

- Source: [Shopify](https://shopify.engineering/faster-trino-query-execution-infrastructure)
- Source URL: https://shopify.engineering/faster-trino-query-execution-infrastructure
- Ingested by Yomu: 2026-08-30T15:29:50.897Z

[Read original post](https://shopify.engineering/faster-trino-query-execution-infrastructure)