// the rules, before you argue with the verdicts

The method

Every verdict on this site comes from five yes/no questions. Here is exactly what each one means, who applied them, and where this whole approach is weak.

01

Who "we" are

One person and a set of AI research agents, working in the open. Every verdict was assigned by hand against the rules below; the trace research was done by agents reading vendor documentation, and every absence claim was then handed to a second agent whose only job was to overturn it — 7 of 13 flipped. Nobody paid for a verdict, no vendor was consulted before publication, and sponsoring the site does not move a score. There is no institution behind this and no claim of authority: the rules are public precisely so you can apply them yourself and disagree with the result.

02

The five questions, precisely

01 · Loops

Yes when: the product decides how many steps a task needs and keeps going until it judges the task done or stuck. The step count varies with the task.

No when: the flow is fixed by whoever built it — a chain, a graph, a scheduled trigger. A workflow that always runs A→B→C is not looping, however many nodes it has.

Edge case: a fixed pipeline that internally retries a step is still No. Retrying is not deciding.

02 · Chooses

Yes when: the product picks which tool to use, and in what order, at runtime, based on what it found so far.

No when: the tools and their order were wired up in advance by a human, even if a model fills in the arguments.

Edge case: picking between two branches of an if-statement someone else wrote is No. Deciding to run a search it was never told to run is Yes.

03 · Acts

Yes when: it changes state outside the conversation — writes a file, runs a command, submits a form, sends a message, opens a PR.

No when: the output is text for a human to act on. Generating a suggestion, however good, is not acting.

Edge case: we count producing an artifact the user then ships (a PR, a deployed app) as acting, and count "here is the code, paste it yourself" as not.

04 · Recovers

Yes when: it hits an error, works out something different to try, and continues — with no human in the loop.

No when: it stops and asks, or repeats the identical call and hopes.

Edge case: this is the axis we are least able to verify from documentation, and the one most likely to be wrong here. Where a session log exists, the leash tool measures it directly instead of guessing.

05 · Unsupervised

Yes when: a documented mode lets a full task complete without a human approving individual steps. Default or behind a named flag — both count, and we name the mode.

No when: every documented mode requires per-action confirmation, or no such mode is documented at all.

This rule was ambiguous until 2026-08-11. It previously said both "we score the shipped default" and "a mode behind a flag gets Yes" — two clauses that contradict each other, which is our fault and not the vendors'. It now turns on capability, and the shipped default is reported in the rationale instead of deciding the answer. The change flipped several verdicts; all of them are on the record.

Scoring: 4–5 yes = AGENT · 2–3 = WRAPPER WITH AMBITION · 0–1 = CRON JOB WITH EXTRA STEPS. The thresholds are a judgment call, not a discovery. The raw five answers sit next to every verdict so you can reject our cutoffs and keep the data.

03

Frameworks are judged differently

A framework is a toolkit for building an agent, so we score what the framework itself does when you run it out of the box — not what you can build with it. That distinction turns out to separate two products people lump together:

CrewAI ships the loop. Run crew.kickoff() and the framework's own reasoning loop runs, its own FileWriterTool writes to disk, and human_input defaults to false — so it scores as acting, looping and unsupervised. LangGraph ships the primitives: the cycles are edges you author, recursion_limit is only a cap, and RetryPolicy re-runs the identical attempt, which rule 04 excludes. It stays low, and that is not an insult — a hammer scores zero on carpentry.

04

Evidence, and what "trace" means

A trace is a step-by-step record of a real run — the individual tool calls, their results, the errors — complete enough that a third party could score it against the five questions. A demo video is not a trace. An outcome dashboard is not a trace. A chat transcript with no tool calls is not a trace.

Three axes, kept apart on purpose. Conflating them is how a rating turns into a vibe:

Capability — yes / no / uncertain

Does the shipped product satisfy the rule? Uncertain means the evidence genuinely cannot support either answer — it is not a polite "no". A missing public trace never turns a Yes into an Uncertain; it lowers the proof grade instead.

Proof — how directly it can be checked

TRACE VERIFIED we inspected an execution record that demonstrates the behaviour · OPERATOR INSPECTABLE you can run it and inspect the steps, but we scored no trace for this criterion · DOCS CLAIMED the vendor's documentation describes it and no inspectable record was scored · UNVERIFIED neither an adequate trace nor precise enough documentation was found.

Those four words are the only proof vocabulary on this site. Exactly one product is TRACE VERIFIED, because exactly one had logs we could read and score. That is not a gap we are hiding — it is the whole reason npx isitanagent exists. Run it and make yours the second.

Trace access — who can reach the record

PUBLIC the vendor publishes traces anyone can read · OPERATOR-OWNED whoever runs it holds the record and nothing is published · VENDOR-ONLY steps appear in a demo you cannot run · UNAVAILABLE no step record reaches anyone outside the vendor.

This is deliberately not a quality grade. A public trace and an operator-owned one can both be excellent evidence. Proof says how well we checked; access says who else could. Conflating the two is how "we can't see it" becomes "it doesn't work", which is the error this whole site was built to stop making.

Leash — a measured number

How many tool calls the agent completes before a human interrupts or redirects it. Measured from a real session, never asserted, and blank wherever nobody has measured it. It is the only number here that came from a machine rather than a judgement.

05

Ways this test can be wrong

the part we'd rather you read first
  1. The rubric is ours. "Agent" has no agreed definition. We picked five properties that seemed to carry the actual disagreement. A different five would reorder this table.
  2. Thresholds are arbitrary. Nothing physical happens at 4 out of 5. The bands exist to make the table readable, and they flatter products that clear them narrowly.
  3. Documentation lags shipping. Most verdicts read what a vendor wrote, not what the product did this morning. A product can fix an axis the week after we score it.
  4. Products are moving targets. Every verdict is a snapshot with a date on it. Treat an old date as a warning, not a result.
  5. Recovery is the weakest axis. It is nearly impossible to confirm from docs and easy to get wrong. Where we could measure it from a real log we did; where we could not, it is an assessment.
  6. Defaults versus capability. We score the shipped default. A vendor can reasonably argue their product is more autonomous than the box it ships in.
  7. Categories are unfair to categories. "AI SDRs" and "LinkedIn ghostwriters" are whole markets, judged on the typical product. Any specific vendor in them may be better than the row.
  8. The corpus behind the leash numbers is one person. n=42 sessions, one tool, one kind of work. It is a scale to read your own number against, not a benchmark.
  9. We are not neutral. The site is satire with a thesis. We think the word "agent" is being abused, and that belief shapes what we notice.
  10. Every correction so far went the same way. Sixteen criteria have been corrected since launch and every single one moved from No to Yes. Two readings fit: our hand-scoring was systematically harsh, or the re-ruling was systematically generous. We cannot tell you which from the inside, so we have published the rule, the evidence and every changed verdict and left the judgement to you. If you think the bar is now too low, that is a fair reading of the same data — say so with a challenge.

If a verdict is wrong, the fix is cheap: every row has a correction button, and overturned calls stay visible in the record rather than being quietly edited away.