Adobe Firefly: Simplified observability with Amazon Managed Prometheus

Summary

Adobe Firefly powers generative creative features using a large-scale GPU training infrastructure hosted on Amazon Elastic Kubernetes Service. As workloads scaled to thousands of nodes and GPUs, Adobe's self-managed Prometheus setup struggled with query latency across high-cardinality telemetry data. To resolve these performance bottlenecks, the team moved two million critical time series metrics to Amazon Managed Service for Prometheus while using managed scrapers alongside their existing infrastructure. This setup accelerated query response times by more than 28x over 24-hour windows, reducing query execution times to approximately ten seconds. Additionally, the migration expanded practical monitoring windows from six hours to 24 hours, giving teams reliable visibility across long-running distributed training jobs.

Context

Adobe operates GPU-based model training infrastructure on Amazon Elastic Kubernetes Service (Amazon EKS) to power Adobe Firefly. As training scaled across thousands of nodes and GPUs, generating high-cardinality telemetry across compute, memory, and network layers, their self-hosted Prometheus infrastructure could not meet query performance and availability requirements for large data volumes.

Approach / What changed

Adobe iteratively migrated critical telemetry, encompassing 2 million time series metrics for job monitoring, pod/node health, and GPU health, to Amazon Managed Service for Prometheus. They deployed Amazon Managed Service for Prometheus collectors (managed scrapers) on Amazon EKS to forward metrics directly into managed workspaces alongside their existing self-managed Prometheus setup.

Takeaways

• Query performance improved by up to 28.8x for 24-hour windows, reducing queries that previously timed out at 60 seconds to approximately 10 seconds.
• Observability query windows expanded from a practical limit of 6 hours to 24 hours, aiding lifecycle analysis for distributed jobs spanning 256 or more nodes.
• Adobe used managed scrapers to ingest metrics alongside their self-managed deployment, enabling an incremental migration of 2 million critical time series metrics without workflow disruption.

Related reading

• Meta · Aug 3, 2026

  GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model

  Meta's Generative Ads Recommendation Model (GEM) trains across thousands of GPUs using a hybrid architecture featuring trillions of sparse embedding parameters and billions of dense parameters. Standard training infrastructure built for large language models struggles with recommendation-specific workloads characterized by jagged user activity histories, asymmetric attention patterns, and numerical sensitivity. To overcome these bottlenecks, engineering teams introduced custom kernels like Jagged Flash Attention and Generalized Dot-Product Attention alongside MXFP8 mixed-precision execution. Pairing these kernel optimizations with topology-aware 5D parallelism doubled end-to-end training efficiency to 20–25% Model FLOPs Utilization while scaling total training FLOPs fourfold over twelve months.

• Grab · Sep 23, 2025

  Powering Partner Gateway metrics with Apache Pinot

  Grab needed to power real-time analytics dashboards for its Partner Gateway, tracking API status codes and latency across datasets reaching 6.8 billion rows over 30-day windows. Initial aggregation queries on datasets exceeding 150GB frequently timed out past 10 seconds, failing to meet the platform's 300-millisecond service level agreement. To support low-latency Online Analytical Processing queries, Grab routed metric streams through Apache Kafka and Apache Flink into Apache Pinot. Query execution was then accelerated by partitioning Kafka topics by metric name, adding rounded time interval columns, and implementing Star-tree indexes for multidimensional pre-aggregation.