Evaluating with $eval

What $eval does

$eval sends simulated users through your product and records what they did. It runs only when you invoke it in a session, and the simulation runs locally.

Running an evaluation

1. Name the version and the decision

   Tell it the exact build under test and what the result should help you decide.

2. Set personas and scenarios

   Each simulated user gets a persona, such as an impatient first-timer or an expert who skips onboarding, and a task they would plausibly do.

3. Give each user the real entry point

   Each user starts from a clean context and uses the actual product: a browser, a phone, a shell, a conversation, or a rendered artifact.

4. Read the evidence, then decide

   Screenshots, commands, traces and artifacts are kept. Then keep the change, revert it, or re-verify with held-out cases.

Reading the result

A run's result includes:

• Where users got stuck: the action or observation where the run broke down, with the trace kept for replay.
• What was observed and what was inferred, listed separately.
• The next task: each observed failure becomes a build task for the next round.

To find past runs, press ⌘K and open Eval runs.