AgentProbe
Get started

REV v0.2.0 Unit D-01

API checking

Regressionsnotnoise

Planted flaws caught /// mock mode
9 / 9
False alarms /// simulation
0.9%
Attempts per case /// demo suite
10

Regression detection for LLM agents. Statistically corrected. Flakiness-aware.

Get startedTry the demo

Demo servers take about 30 s to wake

Repeat every case

Every case runs many times, not once.

Judge traces and tool calls

Judges read tool calls, not just text.

Gate the pull request

The check fails only on a real drop.

Proof

Measured 2026-10-02 /// mock mode

/support/v1 -> /support/v2

Regression
refund-outside-window10/10 -> 0/10 passedStable fail
P-value
5.4e-06
Threshold 0.0250 /// Fisher exact
Pass rate
86.7%
Was 98.9% /// CI unmeasured
False alarms
0.9%
95% CI 0.7–1.2 /// naive 70.6%
Planted flaws detected
9 / 9
4 demo agents /// controls 9 of 9

Mock mode: the demo agents are deterministic rule engines, so this proves the pipeline end to end, not detection on live models. The false-alarm rate is a seeded simulation of unchanged flaky agents (30 cases × 5 runs per case, 20% of cases flaky). How each number was measured

Install

CLI /// Run /// GitHub Action
pip install "agentprobe-core @ git+https://github.com/AlfredBateman/AgentProbe@v0.2.0#subdirectory=packages/core" "agentprobe @ git+https://github.com/AlfredBateman/AgentProbe@v0.2.0#subdirectory=packages/cli"
agentprobe run suites/regression.yaml
- uses: AlfredBateman/AgentProbe/action@v0.2.0

Install from the git URL: the PyPI name agentprobe belongs to an unrelated project. Quick start