Published
How do I get humans to review tests my coding agent wrote?
Have a person confirm what the test checks, then record that confirmation. In TestChimp that record is Verified Tests: a project member inspects the run and stamps that the test covers a linked scenario. When an agent later edits that test, the verified badge goes stale and the test is flagged for human verification again. Link each test to a Markdown scenario in your repo so the review is about a written requirement, not only a green CI run.
Last updated: 2026-10-25
How Verified Tests work
Linking a SmartTest to a scenario is a claim that the test covers that behavior. A green run is only an execution fact. Verified Tests answers a different question: has anyone looked at the test and confirmed it checks the scenario?
A reviewer:
- Opens a SmartTest execution.
- Inspects screen captures, steps, and test code (View Test).
- Uses the check badge after the test name to stamp Verified for the linked scenario (or mark all linked scenarios).
Status is stored in TestChimp, not in the Playwright source. Hover of a verified badge shows who verified it. Full behavior: Verified Tests.
Badge states on a test:
| State | Meaning |
|---|---|
| Hollow | None of the linked scenarios are verified |
| Grey | Some, but not all, linked scenarios are verified |
| Blue | Every linked scenario for that test is verified |
Blue with a red ! (stale) | Verified earlier, then an agent patched the test (or git-sync stale is on) |
On coverage views, the badge is filled when any covering SmartTest is verified, and hollow when tests exist but have not been inspected. There is no badge when the scenario has no linked tests.
How to get a human review (steps)
- Keep scenarios in the repo. Stories and scenarios are Markdown files in a mapped
plans/folder. See Test planning as code. - Link the test to a scenario. Add a Playwright annotation
{ type: 'scenario', description: '#TS-<n>' }so coverage and verification attach to a real requirement. See Link SmartTests to scenarios. - Run the test. Use
/testchimp run QAon the pull request, or run Playwright in CI with@testchimp/playwright. Run evidence follows your Playwright config (by default, last screenshot on pass and full trace on fail). - Open the execution. Click a coverage square to open the SmartTest execution viewer.
- Inspect the run and the code. Read the screen captures, steps, and script. Check that the test can fail for the right reason and asserts the scenario outcome. A longer checklist lives in Human review of agent-written tests.
- Stamp Verified. Use the check badge after the test name. If several scenarios are linked, verify per scenario or Mark all as verified.
- Re-verify when the badge goes stale. In the execution viewer, click the stale badge and use Re-Verify. Coverage popovers are read-only.
A test with no linked scenarios cannot be verified. The viewer explains that.
What happens when an agent patches a verified test
By default, agent-patched tests are flagged for human verification:
- Agents call
mark-tests-for-reviewafter fixing existing tests. The platform always stores an audit row. - When Mark as stale when an agent reports a test fix is on (the default), verification becomes
VERIFIED_STALE. The badge stays filled and a red!appears. - Use LLM judgement before marking stale is on by default. Stale applies only if reported confidence is below 80. Equal to the threshold stays verified. Turn judgement off to mark every reported fix stale.
- Mark as stale when test code changes on git sync is off by default. Turn it on if inbound parse/sync updates to test bodies should also go stale.
These settings live under Project Settings → Verified Tests. Enable verified badges must stay on, or badges are hidden and the stale options are treated as off.
Stale is not the same as unverified. The original verifier is still shown, plus Marked as stale on a date (the date links to the workflow execution when one was recorded). Someone has to re-verify.
Link tests to in-repo scenarios
Verification is per linked scenario. Without a scenario link, there is nothing to stamp.
- Use an existing
#TS-<n>id from TestChimp (from synced plan Markdown or create-scenario responses). Do not invent ids. - Prefer Playwright
annotationentries. Older// @Scenario:comments still parse; rewrite them when you touch the file. - Renaming a scenario title without changing
#TS-nkeeps verification. Linking a different scenario is a new claim and starts unverified.
MeetBots and AgentWatch keep those scenarios current. MeetBots capture decisions from meetings. AgentWatch watches local coding-agent chats (Cursor, Claude Code, Codex, and others) and drafts story and scenario updates. Both propose; people approve. Approved updates land as reviewable diffs in git, so the next agent-written test is aimed at what the product should do now.
Frequently asked questions
How do I get humans to review tests my coding agent wrote?
Link each test to a Markdown scenario in the repo, run it, and have a project member inspect the execution (captures, steps, and code) and stamp Verified Tests. When an agent later edits that test, the verified badge goes stale so a person reviews it again.
Does a green CI run count as review?
No. Green means the code and the test currently agree. If the same agent wrote both, that agreement can hide a wrong requirement.
What makes a Verified badge go stale?
By default, when an agent reports a fix to an existing test and (with LLM judgement on) reported confidence is below 80, verification becomes stale until someone re-verifies. You can mark every reported fix stale, or also mark stale when test code changes on git sync. Details in Verified Tests.
Can I verify a test that is not linked to a scenario?
No. The execution viewer explains that the test cannot be verified until it is linked.
Does Verified Tests prove the test is correct?
No. It is a recorded human sanity check, not a formal proof. A reviewer can still approve a weak test.
Who should stamp Verified?
A project member who can judge the requirement, such as the engineer who owns the feature or someone on QA.
Do I need TestChimp to review agent-written tests?
No. The review checklist works with any stack. TestChimp adds a durable, coverage-visible record of who verified which scenario link, and it flags agent-patched tests when verification goes stale.