OpenEnv in Practice: Evaluating Tool-Using Agents in Real-World Environments

Summary

OpenEnv is an open-source framework from Meta and Hugging Face for evaluating AI agents against real systems rather than simulations, with stateful environments that connect to tools and APIs. Turing's production-grade Calendar Gym applies this model to calendar operations, combining access control lists, partial visibility, temporal reasoning, and multi-step workflows in isolated sessions. Agents often handled individual actions but became less reliable on longer or ambiguous tasks: success was close to 90% with explicit calendar identifiers and roughly 40% when requests used natural-language descriptions. More than half of failures involved malformed arguments or incorrect action ordering despite selecting the correct tool, while structured validation, permission, and datetime errors provided concrete recovery signals.

Takeaways

1. 1

   Success was close to 90% for tasks with explicit calendar identifiers but dropped to roughly 40% when equivalent requests used natural-language descriptions.

2. 2

   More than half of failed interactions involved malformed tool arguments or incorrect action ordering, even when the agent selected the correct tool.

3. 3

   Common integration failures included schema validation errors, 401/403 permission errors, and RFC3339 or timezone formatting errors; structured remediation can support recovery.