You're probably measuring your AI agent wrong (and how a control arm fixes it)

The claim nobody backs

Every agent product says it makes the model better. Almost none prove it. Not because the agents don't improve — because the measurement is broken in a subtle, specific way.

Strategy loops are detectable without an LLM

A strategy loop is the same class of action failing repeatedly across distinct targets — the agent editing three files the same wrong way, or re-running the same failing fix. You don't need a model to notice this. A hard, auditable rule does it:

≥3 events of the same action class, hitting ≥2 distinct targets, most recent outcome = failure.

Deterministic, non-gameable. The class is derived from tool + error code + normalized message; the target (file/path) separates a genuine retry from a strategy loop. This is an anti-Goodhart guarantee: you can't weaken the rule to make your numbers look better.

The trap: selection bias

Once you can detect loops, the obvious thing is to intervene and measure. And the obvious measurement is wrong: the detector only fires on hard sessions, so "intervened" vs "not intervened" compares hard against easy. Your intervention looks harmful. You ship nothing, or worse, you rip out a feature that actually helped.

You cannot fix this after the fact with statistics. You fix it at collection.

The fix: a blind control arm

At the instant of detection — the one moment both groups are comparable — assign the loop to treated or holdout:

• treated: intervene (surface a provocation, let the agent reorganize).
• holdout: detect, measure, and stay silent. The intervention exists but is never delivered. Telling the agent would already be intervening, and would collapse the experiment.

Assignment is deterministic — hash(run | signature) — because detection can run more than once over a session; a random draw would put the same loop in different arms across runs and rot the sample.

Judge both arms with the same ruler

The treated arm has a natural success signal; the control has none. Comparing "the intervention's verdict" against "whatever happened in control" compares different instruments. So read the outcome from the event stream itself, on both arms: ≥3 same-class failures after detection → recurred; a success with the same tool on a target where it was failing → broke. That comparable outcome is the metric. The lift is treated-break-rate minus control-break-rate.

Why not just ask an LLM if it's looping?

An LLM gives a plausible guess, not a rule; it changes its answer run to run; it has no memory of last week; and running it on every action is absurdly expensive. Use cheap deterministic rules to watch continuously, and spend the model only at the rare, high-stakes moment a real loop is confirmed.

The hook

We built this on editor hooks: a config file plus a dependency-free Node script. Fail-open (never blocks the editor), dormant without a token, and it sends only action name, target, and outcome — never your code. On a confirmed loop it injects the provocation back into the conversation.

What we learned dogfooding it

Repeated read/search calls tripped the detector — false positives on navigation. A real strategy loop needs executive actions (edits, runs, commands), not the agent just looking around. So read-only tools no longer count as attempts. The thing measuring the thing caught itself.

Where we are

We're collecting data now, and — honest disclosure — we don't have lift numbers yet. We'll publish loop rates and measured lift with the sample size attached, never a ratio without an n. We're sharing the method now because the measurement design is the part worth stress-testing in public.