tau-bench: A Benchmark for Tool-Agent-User…

tau-bench evaluates tool-using agents inside multi-turn retail and airline conversations with domain policies and mutable databases. Read it to see how final-state grading separates fluent talk from completed work, why pass^k measures repeated reliability, and why the original scores are now historical rather than a current leaderboard.

Reading focus: How a simulated user, policy document, API tools, and hidden target state form one stateful agent task. Why the reward checks both the final database and required information in the agent's reply. How pass^k exposes inconsistent agents and why corrected task releases matter for present-day comparisons.

ICLR 2025. Shunyu Yao et al.. 34 min read, medium difficulty.