OpenEnv

Fit for Purpose

Environments, benchmarks, and the metrology of AI agents

Victor Barres
Mercor Research Lab
Env is in the name.
But what is an environment for?
Part 1 of 4

An environment and its number

  1. 1An environment and its number
  2. 2Building the instrument
  3. 3Reading the measurement
  4. 4Living with an eval
Fit for Purpose2 / 19

One world, many environments

An environment is the world seen through an actor’s body: what it can sense, and what it can do.

owl a lookout squirrel a ladder bark beetle food ant a hunting ground fox a roof over its den forester timber Same for agents One customer-support operation, three environments. Chat agent sees text; acts with tools and messages Voice agent hears audio, timing, interruptions, noise Back-office agent sees records and policies; acts on databases After Jakob von Uexküll’s oak and its inhabitants (1934). In both, the body decides what the environment is.
Fit for Purpose3 / 19

A reward is not a measurement

Attach a task and a grader, and an environment produces a number. You can judge it by what it does to the model, or by what it tells you about the model.

environment, with a task and a grader attached task agent grader 0.73 one number, two jobs Reward a gym judged by what it does to the model learnable variance at the right difficulty hard to hack Measurement a benchmark judged by what it tells you about the model reliable valid for a stated purpose predicts deployment Fitness for purpose: a method is never valid in general, only for a stated purpose.
Fit for Purpose4 / 19

Homework and exams

Same materials, two assemblies. Homework adapts to the student. The exam can’t.

Same raw materials environment task task set scorer Curriculum homework adapts to the learner Score = learning signal Instrument exam fixed in advance Score = reading
Fit for Purpose5 / 19
Part 2 of 4

Building the instrument

  1. 1An environment and its number
  2. 2Building the instrument
  3. 3Reading the measurement
  4. 4Living with an eval
Fit for Purpose6 / 19

What bet are we making?

We want to predict what will happen in the world. Every environment bets on how its score carries over. Realism is the tempting bet.

lift Keeps: wing shape and the forces on it Drops: passengers, catering, the weather at the destination the world the environment homomorphism Realism, pushed to the limit: an isometry. Keep everything. Homomorphism: keep the relations you chose. Realistic with respect to what?
Fit for Purpose7 / 19

Four validity bets

Not a full typology. Different evaluations make different bets, and fail in different ways.

TheoryValidity rests ona theory ofwhat mattersExamplesARC-AGIFails whenthe theory is wrongSimulationValidity rests onpreserving the relationsthat matterExamplestau-bench*WebArenaOSWorldFails whenit’s only a simulacrumReal workValidity rests ondrawing the rightpieces of the worldExamplesSWE-benchGDPvalAPEX-Enterprise*Fails whenthe sample is skewed or staleA/BReal usersValidity rests onsplitting real usersat randomExamplesLMArenaFails whenthe crowd isn’t your users * Disclosure: I built tau-bench at Sierra, and APEX-Enterprise is from Mercor, where I work now. Examples are illustrations, not a survey.
Fit for Purpose8 / 19

We evaluate agents, not models

We train models, but we deploy and measure agents. The harness is how a model is adapted to the environment it has to deal with.

agent harness prompts, loop, memory, tools model interface MCP, orcomputer use? environment The harness adapts the model to the environment it faces. Environments assume what a good agent interface is: a matter of normalization across the industry. Questions every agent evaluation has to answer 1 Can you change the tools? Bring your own, or only use the ones provided? 2 Can you build a better interface? Switch from MCP to computer use, or the reverse? 3 Is it fair to swap the harness? Testing a model on a harness it wasn’t trained with. The eval doesn’t care how you got there. Say what’s in the box, and what it cost.
Fit for Purpose9 / 19

Only what can be graded

A measurement must be a situation we can grade. A reward can tolerate noise; a measurement can’t.

Reliability bounds validity.
We measure where the light is.
unambiguous policy reliable user simulator consistent grader clear blame expert judgment ambiguousrequests expertswhodisagree de-escalating anemotional customer upselling negotiation outboundsales calls policies open to interpretation
Fit for Purpose10 / 19

Beyond the average

Stress tests, must-pass cases, and counterexample search are measurements too, under stated conditions.

proof load deflection, measured Not representative. Still a measurement. It measures a different quantity: margin, failure load. Frequency How often does it succeed? Conformity Does it pass the cases we decided are critical? Existence Here is a situation where it fails. One counterexample refutes “never.” State the conditions of measurement in advance.
Fit for Purpose11 / 19
Part 3 of 4

Reading the measurement

  1. 1An environment and its number
  2. 2Building the instrument
  3. 3Reading the measurement
  4. 4Living with an eval
Fit for Purpose12 / 19

“How well should my agent do?”

A benchmark score is not a forecast. It departs from production in two opposite directions.

where the benchmark’s tasks come from refund password reset order status production traffic, by kind of case (illustrative) Easier than the world No ambiguity. Simulated users far simpler than real ones. You should ace it. Harder than production Sampled from the tail, to be interesting and to make models fail. Below 100% can be fine. Know your traffic. Map it. Reweight.
Fit for Purpose13 / 19

Read it, then experiment

The number is the least of what an eval produces. A reward is consumed by an optimizer. A measurement is read by people.

Read the trajectories The number says something changed. The traces say what. violated the refund policy asked for information it already had gave up after one tool error failure modes across runs (illustrative) policy errors redundant questions early stops Run the experiment Choose environments where tasks vary easily on the dimensions that matter. Control the agent acts alone A world Treatment the user acts on the world too A U world tau2: same tasks, one parameter changed alone coordinating cost of coordination From a score to a mechanism.
Fit for Purpose14 / 19
Part 4 of 4

Living with an eval

  1. 1An environment and its number
  2. 2Building the instrument
  3. 3Reading the measurement
  4. 4Living with an eval
Fit for Purpose15 / 19

A good eval may not train well

It may have the wrong shape for training, or what it measures may be best learned elsewhere.

The wrong shape tau-voice probes the whole agent. Training works on its parts. voice agent VADspeaker IDbackchannelencoderLLMTTS Architecture-agnostic: good for comparing systems, not for training any one of them. Better trained elsewhere The fastest way up an eval may not look like the eval. coding environments tool useplanning over many stepsstructured reasoningyour eval Training on code transfers well beyond code. Judge a training environment by what it does to the model, not by how much it looks like the eval.
Fit for Purpose16 / 19

A good eval is a maintained eval

Instruments drift. Benchmarks are software, not papers.

v1.0 v1.1 v1.2 v2.0 released Fixed 14 grader bugs found by people reading the traces Updated tool APIs the real tools changed underneath Retired saturated tasks re-calibrated against deployment traces Calibration Reference deployment traces Version v2.0 Maintained by a named team Who maintains it?
Fit for Purpose17 / 19

Before you trust the number

  1. Is this number a reward or a measurement?
  2. Realistic with respect to what?
  3. What’s in the measurand? Model, or model plus harness?
  4. Can it be graded? Can you easily vary its tasks?
  5. Who maintains it?

Never good in general. Fit for a purpose.

Fit for Purpose18 / 19
Thank you
Questions?
Victor Barres
Mercor Research Lab
Fit for Purpose
Environments, benchmarks, and the metrology of AI agents

Hesse’s analogy

Why an environment can predict anything at all, and what calibration actually does.

Properties of the world, as the environment relates to them Positive analogy Negative analogy Neutral analogy Known to be shared. Real tools and real APIs are identity, not resemblance. The foundation. Known to differ. No ambiguity, simpler users, clean policies. Defines the scope. Not yet checked. Does success with simulated customers carry over to real ones? The predictive bet. Calibration against deployment, through A/B tests or traces, moves properties out of the neutral zone.
Fit for Purpose: backupB1

What evaluation is for

Six jobs, each judged differently. Training sits beside them, judged on learnability.

FunctionWho it servesJudged on
PredictDeployersDoes the ranking match deployment outcomes?
CompareResearchers, regulatorsFairness, reproducibility, stability over time
PrescribeThe ecosystemDoes it encode what we actually value?
SignalBuilders, new entrantsDoes it show that a capability or market exists?
ExposeModel builders, data providersAre the failures real, reproducible, actionable?
LearnPractitioners, the curiousDoes it show how agents actually work and fail?
Fit for Purpose: backupB2

Everyone has an interest

Incentives don’t disqualify a benchmark. Methodology, not neutrality, is the protection.

Labswant to look goodStartupswant to prove a market existsAcademicswant citationsData companieswant to surface gaps Methodology a sampling procedure fixed in advance held-out sets, treated as held out transparency about what was excluded a method that can’t be quietly bent I work at a data company. Weigh this accordingly.
Fit for Purpose: backupB3

An A/B test is an instrument too

The real world is the arbiter. An A/B test is how we consult it, and it can be miscalibrated.

Offline eval fast, replayable, available early can miss: the real distribution, what idealization left out A/B test randomizes over real users can be miscalibrated: underpowered novelty effects proxy KPIs: resolution rate, NPS blind to rare, costly failures The world the arbiter Trust lands on the instrument closest to the world. It is earned, not automatic.
Fit for Purpose: backupB4

What an environment buys you

The world is its own best model. An environment buys you replay, and the two jobs need it for different reasons.

Replay Reset and run again. Training needs a resettable world. s₀ GRPO compares a group of rollouts from one state Randomize The A/B test. Evaluation needs a fair comparison. A/B Model A Model B Slow and needs users, but samples the real distribution.
Fit for Purpose: backupB5

Types, instances, and the tau family

New domains add values. New versions add axes. Every added axis has to stay gradeable.

Type entities, tools, policy, open parameters Instance one database, one customer, one request Task the instance, plus a goal and a check tau-bench an agent with tools and a policy; the user only talks tau2 the user acts on the world too tau3 search tau-voice audio: noise, accents, interruptions versions add axes airline retail telecom domains add values hyper-tau builds the agent that is tested below
Fit for Purpose: backupB6

Two hats on one head

Every eval designer wants to show capabilities and to find failures. The two pull task design apart.

Show it can Make it fail Showcase tasks chosen to illustrate a behavior Read alone, they overstate readiness. Failure-hunting tasks chosen to expose a gap Read alone, they understate readiness. Keep both hats. Label which tasks serve which purpose.
Fit for Purpose: backupB7

Arm, rock, or hammer?

We train models, but we deploy and evaluate agents: a model with a body. Arm, rock, or hammer? Every evaluation has to say which is which.

An arm part of the agent the harness: loop, memory, context A rock part of the world the systems the agent works on A hammer a tool an MCP server, a code interpreter Found, or brought? Tools the environment provides are part ofthe world, the same for every agent.Tools the agent brings are part ofwhat you’re measuring. found or brought Does it fit the body? A hammer is made for a hand. A different bodybrings an adapter: screenshots and clicksto use an interface built for people. The adapter is harness. adapter The eval doesn’t care how you got there. Say what’s in the box, and what it cost.
Speaker notes