---
title: "Announcing Evals and Releases: Evaluate Fin before, during, and after you go live"
description: "Fin’s new Evals and Releases provide an evaluation and rollout system for testing changes before launch, controlling deployment, and assessing live conversations, alongside Monitors. Evals group multi-turn Simulations built from real conversations around a theme, then score them pass/fail using deterministic checks and an LLM judge against criteria such as required responses, Procedures, or data connector calls. Releases let teams edit content, Procedures, and Guidance away from live Fin, run Evals on proposed changes, collaborate, and publish to everyone, ramp traffic, A/B test, or roll back. Monitors evaluates every live conversation and can turn failures into new Simulations for subsequent testing. The combined workflow addresses probabilistic behavior, regression risk, and drift as support content and policies change, while Operator can draft changes, tests, and Releases for review and approval."
---

# Announcing Evals and Releases: Evaluate Fin before, during, and after you go live

[Intercom](https://yomu.fyi/company/intercom) · Brian Donohue · Aug 13, 2026

**Type:** Announcement

## Summary

Fin’s new Evals and Releases provide an evaluation and rollout system for testing changes before launch, controlling deployment, and assessing live conversations, alongside Monitors. Evals group multi-turn Simulations built from real conversations around a theme, then score them pass/fail using deterministic checks and an LLM judge against criteria such as required responses, Procedures, or data connector calls. Releases let teams edit content, Procedures, and Guidance away from live Fin, run Evals on proposed changes, collaborate, and publish to everyone, ramp traffic, A/B test, or roll back. Monitors evaluates every live conversation and can turn failures into new Simulations for subsequent testing. The combined workflow addresses probabilistic behavior, regression risk, and drift as support content and policies change, while Operator can draft changes, tests, and Releases for review and approval.

## Context

As AI Agents handle more volume and complex work, teams need to validate many possible conversation scenarios and assess more than final-answer correctness. Fin’s probabilistic behavior, evolving content and policies, and the risk that one change creates unintended drift make manual oversight difficult at scale.

## Approach / What changed

Evals use multi-turn Simulations, deterministic checks, and an LLM judge to test Fin against defined criteria. Releases isolate changes to content, Procedures, and Guidance for collaboration and testing before controlled publication, while Monitors assess live conversations and feed failures back into future Evals and Releases. Operator can coordinate these steps as proposals for approval.

## Takeaways

- An Eval can be rerun after changes to detect regressions, with each Simulation showing its transcript, event log, outcome, and pass/fail result.
- A Release supports gradual rollouts, A/B tests using metrics such as resolution rate, escalation rate, and CSAT, plus pausing or one-step rollback.
- Monitors can turn a conversation that falls short of quality standards into a Simulation added to a relevant Eval.

**Tags:** [AI Agents](https://yomu.fyi/topic/ai-agents), [Model Evaluation](https://yomu.fyi/topic/model-evaluation), [Monitoring](https://yomu.fyi/topic/monitoring), [Release Engineering](https://yomu.fyi/topic/release-engineering)

- Source: [Intercom](https://www.intercom.com/blog/announcing-evals-and-releases)
- Source URL: https://www.intercom.com/blog/announcing-evals-and-releases
- Ingested by Yomu: 2026-09-09T08:19:45.277Z

[Read original post](https://www.intercom.com/blog/announcing-evals-and-releases)