---
title: "CX Score: How we built a metric support leaders can defend"
description: "CX Score was built to give support leaders a defensible view of customer experience across conversations, addressing whether an automated metric can be trusted. The team grounded its criteria in how experienced support professionals assess strong, mixed, and poor interactions, then created a dataset of thousands of conversations reviewed by experts, with two reviewers where possible and disagreement resolution. It operates as a production system using specialized prompt engineering, support-focused model configuration, neutral defaults for ambiguous evidence, and multi-stage evaluation across LLM models. Precision, recall, F1, regression, and bias checks set a required F1 above 0.8, while field testing with real customers examined varied tickets, languages, agent types, and conversation lengths. The stated conclusion is that CX Score is designed to be inspected, explained, and defended rather than treated as a black-box output."
---

# CX Score: How we built a metric support leaders can defend

[Intercom](https://yomu.fyi/company/intercom) · Mark O'Donovan · Feb 13, 2026

**Type:** Explainer

## Summary

CX Score was built to give support leaders a defensible view of customer experience across conversations, addressing whether an automated metric can be trusted. The team grounded its criteria in how experienced support professionals assess strong, mixed, and poor interactions, then created a dataset of thousands of conversations reviewed by experts, with two reviewers where possible and disagreement resolution. It operates as a production system using specialized prompt engineering, support-focused model configuration, neutral defaults for ambiguous evidence, and multi-stage evaluation across LLM models. Precision, recall, F1, regression, and bias checks set a required F1 above 0.8, while field testing with real customers examined varied tickets, languages, agent types, and conversation lengths. The stated conclusion is that CX Score is designed to be inspected, explained, and defended rather than treated as a black-box output.

## Context

The metric is intended to give support teams a complete view of customer experience across conversations, showing what is working, what is not, and why. Its stated audience includes support leaders who need to communicate impact and justify customer-service metrics to executives.

## Approach / What changed

The team defined quality criteria from experienced support professionals, collected thousands of real conversations across industries, languages, channels, and agent types, and used expert review with disagreement resolution to create consensus labels. CX Score combines specialized prompt engineering, support-focused model configuration, neutral defaults for ambiguous evidence, and a multi-stage evaluation process. The team required an F1 score above 0.8, checked for regressions and bias, and conducted field testing with real customers before release.

## Takeaways

- CX Score’s criteria were based on how experienced support professionals evaluate strong, mixed, and poor conversations, rather than on generic LLM assumptions.
- Ambiguous quality signals default to neutral, and each model revision is evaluated for precision, recall, F1, regression, bias, and agreement with human judgment.
- Before release, the system was shadow-tested against an older model and then assessed by a controlled customer group across varied tickets, languages, agent types, and conversation lengths.

## FAQ

### How was CX Score aligned with expert human judgment?

Thousands of real conversations were manually reviewed by experienced support professionals, with two reviewers per conversation where possible and disagreement resolution to create stable consensus labels. CX Score was trained and tested to behave like an expert reviewer.

### What evaluation standard did CX Score have to meet?

The team evaluated precision, recall, and F1, which balances precision and recall, and set an explicit requirement of an F1 score above 0.8. Evaluations were repeated after revisions to check for regressions and biases, with particular attention to negative experiences.

### How was CX Score tested before release?

The team shadow-scored conversations with both old and new models, checked behavior across agent types and conversation lengths, and then rolled out the system to a controlled customer group. Customers assessed whether scores made sense, reasons were clear, and insights were actionable.

**Tags:** [AI](https://yomu.fyi/topic/ai), [Customer Experience](https://yomu.fyi/topic/customer-experience), [LLMs](https://yomu.fyi/topic/llm), [Model Evaluation](https://yomu.fyi/topic/model-evaluation)

- Source: [Intercom](https://www.intercom.com/blog/cx-scores-how-we-built-a-metric-support-leaders-can-defend)
- Source URL: https://www.intercom.com/blog/cx-scores-how-we-built-a-metric-support-leaders-can-defend
- Ingested by Yomu: 2026-09-17T14:06:28.921Z

[Read original post](https://www.intercom.com/blog/cx-scores-how-we-built-a-metric-support-leaders-can-defend)