Announcing Monitors: Opening the AI black box

Summary

Intercom announced Monitors, a new Fin product area intended to complete its observability suite alongside Insights and Recommendations. It addresses the limits of random sampling, CSAT, and spreadsheet-driven quality assurance by letting teams select conversations with filters or natural-language criteria, including risk signals, and maintain repeatable samples. Custom Scorecards then define business-specific evaluation criteria, weighting, and critical failures, with scoring performed by AI, humans, or both to produce an overall quality score. Flagged conversations enter a Review Queue for assignment, notes, status tracking, and follow-up fixes, while reporting compares scores with CX Score, resolution rate, Monitors, and Scorecards over time. Monitors for Fin conversations is live, with human-agent QA, real-time alerts, and knowledge-base evaluation identified as planned additions.

Takeaways

1. 1

   Monitors can target high-risk or complex conversations, such as financial vulnerability or unresolved repeated answers, while also maintaining consistent samples for benchmarking.

2. 2

   Custom Scorecards can combine AI and human review, weight criteria, and designate failures that invalidate an entire evaluation.

3. 3

   The Review Queue tracks conversations through review and remediation statuses, keeping notes, assignments, and suggested fixes within Intercom.