Announcing Evals and Releases: Evaluate Fin before, during, and after you go live

Summary

Fin’s new Evals and Releases provide an evaluation and rollout system for testing changes before launch, controlling deployment, and assessing live conversations, alongside Monitors. Evals group multi-turn Simulations built from real conversations around a theme, then score them pass/fail using deterministic checks and an LLM judge against criteria such as required responses, Procedures, or data connector calls. Releases let teams edit content, Procedures, and Guidance away from live Fin, run Evals on proposed changes, collaborate, and publish to everyone, ramp traffic, A/B test, or roll back. Monitors evaluates every live conversation and can turn failures into new Simulations for subsequent testing. The combined workflow addresses probabilistic behavior, regression risk, and drift as support content and policies change, while Operator can draft changes, tests, and Releases for review and approval.

Takeaways

1. 1

   An Eval can be rerun after changes to detect regressions, with each Simulation showing its transcript, event log, outcome, and pass/fail result.

2. 2

   A Release supports gradual rollouts, A/B tests using metrics such as resolution rate, escalation rate, and CSAT, plus pausing or one-step rollback.

3. 3

   Monitors can turn a conversation that falls short of quality standards into a Simulation added to a relevant Eval.