ICML 2026ORAL · SEOUL
Sierra University of Toronto Vector Institute Princeton University

τ2-Bench

Evaluating Conversational Agents in a Dual-Control Environment
τ (“Tau”) = Tool · Agent · User interaction
Victor Barres1* · Honghua Dong1,2* · Soham Ray1 · Xujie Si2 · Karthik Narasimhan1,3
1Sierra   2U. Toronto & Vector Institute   3Princeton Language & Intelligence
leaderboard
Live leaderboard
code
Code & data
paper + code  github.com/sierra-research/tau2-bench
leaderboard  taubench.com
→ / space advance · ← back · F fullscreen
Evaluating conversational agents
Org →sets policy Agent ⇄one conversation Customer

A user talking to an agent — and the agent answers for the organization, not the user.

"Hi — I just landed in Seoul and my phone says No Service. Fix it, please. And I'm not paying extra charges."
One call · four demands
Reason
Find the root cause — nobody hands it over.
Act
Real writes in the org's systems.
Be trusted
The org's policy wins — every time.
Communicate
Guide a person, step by step.

This is the setting the original τ-bench formalized (τ = Tool · Agent · User — Yao et al., 2024) · this paper: τ²-bench, its second generation

Watch one call: "No Service", abroad
Agent · follows policy, sees the CRMCustomer · holds the phone
"Hi, this is Sara Kim on 555 0164 — I just landed in Seoul and my phone says No Service."
"Sorry about that, Sara — you're verified.get_customer_by_phone()
What does the status bar on the phone show?"
"Umm…get_network_status()
there's a little airplane icon up top?"
"That's Airplane Mode, from your flight. Toggle it off, please."
"Done.toggle_airplane_mode()
…still No Signal though."
"Let me check your line.get_line_details()
You're abroad and roaming isn't active on your plan. I've enabled it — standard charges apply, per policy.enable_roaming()
To accept and connect: Settings → turn Data Roaming on."
"OK, found it…toggle_data_roaming()
it's back! Full bars, 5G — 감사합니다!"
User's world · the phone
✈
No ServiceNo Signal5G ✓
📍 Seoul, Korea
Airplane Mode
Data Roaming
coupled
Agent's world · the CRM
line L1001
Intl. roaming offon ✓
Scored on the final world state
user.phone.signal == True
crm.line.roaming == True
pass^1 = 1deterministic · no judge

The fix took both hands.

τ²-bench: give the user a world
τ-bench (2024) · the user only talks τ²-bench (this paper) · the user acts too
Agent policyverify, then act
→
AGENTguides the user
→
Agent toolsenable_roaming13 tools · act on the CRM
→
⇅messages
User scenario"just landed · No Service"
→
USERtalks — that's allacts on the phone
→
✗ no toolswords only
User toolstoggle_data_roaming30 tools · act on the phone
→
Shared World
one state · two views
Agent DB · line L1001
Intl. roaming offon ✓
( ) coupled
✗ no device
nothing to act on
✈
No ServiceNo Signal5G ✓
📍 Seoul, Korea
Airplane Mode
Data Roaming
τ-bench couldn't simulate the call you just watched. Its user can only talk: no device, no state, nothing to toggle. toggle_airplane_mode() ✗

So τ² gives the user a world: its own tools and device, coupled to the agent's — dual control over shared state (formally, a Dec-POMDP).

More realistic — and, as we'll see, more reliable at once. Retail & airline stay as single-control baselines, backward-compatible with τ-bench.

Tasks provable by construction: break a working world

Stateful worlds, not tables. Each domain is a live environment with real side effects — so a task is authored by initializing a broken world: pick independent root causes and compose them, on either side of the world.

The customer sees one problem: "No Service." Underneath, any mix of causes can produce it.
RC₁ · user world · airplane ON
initturn_airplane_mode_on()
RC₂ · both worlds · no roaming abroad
initset_location("Seoul, KR")
initset_roaming(line, off)
User's world · the phone
✈
No ServiceNo Signal5G ✓
📍 Home · US📍 Seoul, Korea
Airplane Mode
Data Roaming
coupled
Agent's world · the CRM
line L1001
Intl. roaming offon ✓

The solution the agent + user must find

undo RC₁
1toggle_airplane_mode()
undo RC₂ · needs both hands
2enable_roaming(line)
3toggle_data_roaming()

RC₂ is a cross-world cause: "abroad" is set in the user's world, "roaming off" in the agent's — only together do they break the phone. Its fix takes two hands, and every cause adds steps: difficulty is a dial.

✓ Success is checked in the world: does the status bar show signal? We injected the causes, so the fix provably exists.

15
atomic subtask groups
→
2,285
provably-correct tasks
→
114
balanced & released
The bottleneck is coordination, not reasoning
The results · pass^1 by domain τ-retail τ-airline τ²-telecom
0 20 40 60 80 100 79 50 49 Claude 3.7 −30 66 51 44 GPT-4.1-mini −22 71 59 42 o4-mini −29 74 56 34 GPT-4.1 −40 ▲ Δ = pass^1 drop vs. retail

Telecom — the dual-control domain — is the hardest for every model. Why?

The ablation · telecom · same model, same tasks, same tools
0 20 40 60 80 100 52 34 GPT-4.1 −18 67 42 o4-mini −25 ▲ Δ = drop when the agent must guide instead of act
Solo: the user hands the phone over; the agent drives every tool itself.
Dual-control (default): the agent must guide the user, who acts.

The only change is who executes. Guiding a person costs ~20 points.

Turn the difficulty dial and success collapses
pass^1 · telecom · by number of user actions the task needs
0 20 40 60 80 77 ≈2 actions e.g. service 50 ≈4 actions e.g. mobile data 16 ≈6 actions e.g. MMS
Issues form a difficulty hierarchy
MMS~6 actions · multi-stage
Mobile data~4 actions · multi-stage
requires
Service~2 actions · base fix
The harder issues are multi-stage and first require a working Service: difficulty compounds.

Success falls toward zero past ~7 user actions. Long-horizon coordination was the wall — keep that in mind for what happens next.

The environment shapes behavior: give the user simulator a world
User-sim error per conversation · critical = irrecoverable
prompt-only user (τ-bench) env-grounded user (τ²-bench) 0 5 10 15 critical user-sim error % · lower is better Airline · τ-bench · prompt-only (47% total error) 13% critical Retail · τ-bench · prompt-only (40% total error) 12% critical Telecom · τ²-bench · env-grounded (16% total error) 6% critical

Critical errors 12–13% → 6%, total 40–47% → 16%: the simulator nearly drops out as a confound.

Prompt-only user: imagines the world from text — drifts, invents state, hallucinates.
Grounded user: reads and acts on real state — affordances offload cognition into the environment (situated cognition).

Realism bought reliability. The environment doesn't just challenge the agent — it makes the measurement cleaner.

If you take one thing from τ²
Don't prompt for a reliable simulator. Provide a world.
That was the paper. Then a year happened to the scores.
0 25 50 75 100 pass^1 (%) Apr ’25 Jul Oct Jan ’26 Apr Jul best reported score to date · taubench.com legacy-v1 board (incl. lab self-reports) 99τ²-telecom τ²-telecom 88τ-airline τ-airline 86τ-retail τ-retail Kimi K2 self-reportsJul 11 · the first lab on the τ²-telecom board 🤯 +30 in a monthAug 7 · GPT-5 model card and the race was on — Anthropic · Qwen · Gemini · DeepSeek…
What does this mean for τ²-bench? Models became incredibly better at long-horizon, coordinated tasks — which is what the benchmark measured. The instrument did its job.
Benchmarks don't retire. They gain dimensions.

The answer to saturation isn't more tasks — it's new dimensions: we expand the instrument, building on the same framework.

0 25 50 75 100 pass^1 (%) Apr ’25 Jul Oct Jan ’26 Apr Jul 99τ²-telecom 88τ-airline 86τ-retail 67τ³-voice τ³-voice 37τ³-knowledge τ³-knowledge New frontier new dimensions, same framework
τ³-voice
The same tasks, tools & policies — now a live, full-duplex phone call: noise, accents, interruptions.
67best voice pass^1 · text agents ≈ 91 on the same tasks
👋 Soham Ray is here — say hiarXiv:2603.13686
τ³-voice paper QR
τ³-knowledge
Same machinery — the 2-page policy becomes a 698-document knowledge base to search and reason over, turn after turn.
37.4%best frontier pass^1
👋 Quan Shi & Alexandra Zytek are here — say hiarXiv:2603.04370
τ³-knowledge paper QR
Use these! Artificial Analysis already runs them: τ³-voice → Speech Index τ³-knowledge → Productivity Index And dimensions compose: voice × knowledge — an open question no one has run yet. Come try it.
What kept it useful wasn't cleverness. It was maintenance.

A benchmark is a measuring instrument — it stays true only if it's used and maintained.

2024
τ-bench
retail · airline
single-control
2025 · this paper
τ²-bench
telecom
dual-control
2026
τ³-bench
voice · knowledge
community-fixed · versioned
Thank you, community — for the bug reports behind τ³'s task fixes, and for every leaderboard submission.
github.com/sierra-research/tau2-bench · taubench.com
code
Code & data
leaderboard
taubench.com

감사합니다

Thank you.
τ²-Bench · ICML 2026 · taubench.com
τ2-Bench · ICML 2026 Oral
1 / 21