Loading…
How Standardizing Product Telemetry Reduced Time to Insight by 97%
SalesforceScott Nyberg
Summary
Salesforce faced scaling bottlenecks when individual engineering teams maintained fragmented, custom telemetry pipelines requiring manual data transformations and dashboard creation every release. To resolve these silos, the engineering organization established the Product Data Platform (PDP) using a standardized custom schema built atop Monitoring Cloud infrastructure. This framework mandates core telemetry fields for consistent metric generation while offering governed optional attributes for product-specific flexibility. An AI-driven Model Context Protocol tool assists developers by recommending compliant instrumentation patterns directly within their workflows. Today, the unified platform processes 45 billion rows of data daily across 19,000 distinct events and 2,000 product features, reducing dashboard refresh latency by 97 percent from one month to daily updates.
Context
Individual Salesforce product teams historically built customized, fragmented telemetry pipelines that required engineers to manually transform events into adoption dashboards every release. This duplicate effort created data silos, lacked a centralized adoption view across products like Agentforce and Data 360, and forced teams to wait roughly a month for metrics.
Approach / What changed
Salesforce created the Product Data Platform (PDP), introducing a standardized telemetry framework and schema on Monitoring Cloud infrastructure featuring both mandatory fields and governed optional attributes. Automated pipelines process raw event streams into uniform adoption metrics. Additionally, the team introduced an AI-driven Model Context Protocol (MCP) tool to automatically recommend compliant instrumentation configurations to engineers.
Takeaways
- The Product Data Platform replaced custom, per-team pipelines by standardizing core telemetry schemas on Monitoring Cloud while supporting governed optional attributes for flexibility.
- Automating metric pipelines reduced time to insight by 97%, moving dashboard refresh cycles from approximately one month down to a single day.
- Standardized telemetry now processes 45 billion rows across 19,000 events and 2,000 product features daily, powering downstream Model Context Protocol plugins for AI queries.
Related reading
How Agentforce-Powered AI Security Workflows Accelerate Incident Response
Salesforce engineers expanded Security Center from a basic conversational interface into an Agentforce-powered stateful investigations platform for incident response. Managing enterprise security incidents required handling long-running lifecycles, disparate telemetry formats, and finite LLM context windows that risk reasoning pocket collapse. To prevent overflowing context limits, the architecture partitions telemetry into contextual segments and uses AI-driven summarization pipelines before feeding data to the agent. Because non-deterministic LLM outputs break traditional deterministic testing, the team implemented AI-driven evaluation pipelines using simulated customer interactions to evaluate response behavior rather than exact wording. Ongoing work focuses on grounding public foundation models with curated, Salesforce-owned security knowledge to improve platform-specific remediation guidance.
Scott NybergHow to Evaluate Production AI Agents: Measure System Outcomes, Not Conversations
Evaluating AI agents solely on conversational transcripts fails when agents must invoke tools to modify external production systems. Language models can convincingly narrate that an action occurred while completely omitting the underlying tool call, leaving database or billing records unchanged. Outcome-based evaluation addresses this divergence by treating external system mutations rather than generated dialogue as the actual source of truth. Frameworks such as CRMAgentBench implement persistent, stateful environments with strict all-or-nothing validation, verifying tool arguments, execution sequence, final record modifications, and the total absence of unauthorized actions. Furthermore, measuring repeated success across independent executions using pass^k metrics exposes operational variance and inconsistency that standard single-shot scores and pass@k metrics conceal.
Scott NybergRemoving the Security Barrier to Agentforce Adoption
Enterprise AI adoption across regulated industries stalls when internal governance mandates private, dedicated network paths for sensitive data. To eliminate the need for manual VPN tunnels and complex proxy layers, Salesforce re-engineered Private Connect into a multi-cloud networking platform connecting customer environments to Hyperforce services. The Private Connect v2.0 architecture utilizes private links, transit gateways, and direct endpoint routing alongside an automated control plane for rapid Hyperforce regional rollouts. In addition to supporting cross-substrate interconnects for Azure and decoupling endpoint management, the platform incorporates a generic connector framework supporting tools like Snowflake, Databricks, and Kafka. Today, the system provisions secure connectivity in under 30 minutes while processing roughly 120 TB of data and 683 million requests monthly across 15 AWS regions.
Scott NybergHow AI Rebuilt Salesforce’s Decades-Old Localization Pipeline
Salesforce needed to deliver major releases across 34 languages as product localization volume surged by more than 35% under fixed budgets and unchanged delivery windows. To address these scaling constraints, the engineering team replaced their legacy translation workflows with an LLM-powered orchestration pipeline integrating prompt engineering, context engineering, and multi-stage validation. The architecture transforms existing style guides, glossaries, and terminology databases into structured context, routing interface strings through specialized prompt stages—up to 85 stages for some languages—to enforce product context, grammar, and branding rules. Combined with an automated validation loop spanning AI editing, AI validation, and human review, this system reduced localization costs by 50 to 90% while accelerating turnaround times across enterprise releases.
Scott Nyberg