Static demo · saved runs · nothing on this page calls an API

agent-eval

Describe an agent in a suite file: how to call it, what to ask it, what a good answer looks like. Get back a typed report and a statistically honest verdict, per check, on whether the thing you changed made it worse.

Loading saved runs.

How the framework decides

  1. It never learns how an agent works. It knows how to call one and nothing else. The moment it knew one agent's internals, it would only evaluate that agent.
  2. Wiring an agent in is a file, not a class. The entry point, the cases and the checks are data in a suite file. No adapter is written per agent.
  3. A score drop is not a regression until it survives resampling. Runs are paired case by case and the change in mean gets a bootstrap confidence interval. Per-check intervals are widened together, so ten checks don't mean ten chances at a false alarm.
  4. Runs are artifacts, not stdout. Every run is timestamped JSON you can diff and compare months later. This page renders exactly those files.