Skip to main content

Published

QA for Coding Agents: What Claude Code, Cursor and Devin Cannot See

Short answer

Claude Code, Cursor and Devin can all write tests and run them, and you should let them. What a single agent session cannot see is the rest of the picture: which of its tests anyone has reviewed, which product behavior is still uncovered, whether older tests have drifted from the current requirements, and what to test next. QA for coding agents is the layer around the agent that answers those four questions.

Last updated: 2026-11-16

Do coding agents already do QA?​

Yes, within a session. Ask an agent to add a feature and a good one will add tests, run them, read the failures and fix its own mistakes. Agents can also drive a browser, for example through Playwright MCP, to check a flow by hand. This catches many bugs before a human sees the pull request.

The limit is scope. An agent works inside one task, in one session, with the context it was given. QA for a product is a standing question: across all changes, by all contributors, human and agent, is the product still correct? The answers live outside any single session.

What can an agent session not see?​

Four things. They are the same for Claude Code, Cursor, Devin and the rest, since the cause is the session boundary.

Reviewed tests. An agent can write twenty tests in a pull request. Whether anyone read them is a fact about your team, not about the agent. Green CI says the tests pass, not that they assert the right thing. See how to get humans to review tests your coding agent wrote.

Actual coverage. Line coverage tells you which code ran. It does not tell you which user scenarios have a test, or whether a given test would fail if the behavior broke. A new session starts without that map and tends to add tests near the code it touched. Test coverage metrics explains why the number alone misleads.

Drift. Requirements change, and an agent working on a ticket may update the code and the tests it can see. Tests elsewhere that encode the old rule keep passing or get quietly rewritten. Agents have also been documented changing a test to match the code; see agents weakening test assertions.

What to test next. An agent does what the prompt says. Nobody prompts it with "the refund flow has no end-to-end test and changed twice this month." Deciding where the next test is worth writing is a prioritization problem that needs data from outside the repo diff.

How do Claude Code, Cursor and Devin differ for QA?​

For this purpose, less than you would expect. All three can edit files, run shell commands and read test output. What differs is the control surface you get around them.

Claude Code and Devin's CLI both document lifecycle hooks: scripts that run at defined points. Devin's documentation lists events including PreToolUse, PostToolUse and Stop. A PreToolUse hook can block a tool call, a PostToolUse hook can run a check after an edit, and a Stop hook can block the agent from finishing and tell it to run the test suite. Devin's docs warn that blocking stop hooks can cause loops, so cap retries. Claude Code uses the same hook idea, configured in .claude/settings.json. Check the current documentation for Cursor and any other agent you use, since hook and rules support changes quickly.

Where hooks are not available, you still have the repo: branch protection, required status checks, and CODEOWNERS apply to every contributor whether human or agent.

What QA layer should a team put around agents?​

A short list that works for a team of 10 to 50 engineers with little or no dedicated QA:

  1. Write scenarios in the repo. Keep a plain-language list of what the product must do, as files next to the code. Agents read them. People review them. Tests link back to them.
  2. Make test changes visible. Review test diffs separately from product code, and require an owner's approval on test directories through CODEOWNERS.
  3. Gate merges on real checks. Run the suite in CI and make it a required status check. Run a quick smoke test subset on every pull request.
  4. Track coverage by scenario, not by line. Know which scenarios have a passing, reviewed test and which have none.
  5. Verify in the app, not only in code. Have the agent, or a person, run the changed flow in a real browser and keep the evidence with the pull request.
  6. Look at what users do. Use production behavior to find the flows that matter and are untested.

Each step either gives the agent better input or checks its output.

Should you stop letting agents write tests?​

No. Agent-written tests are a cheap way to get broad coverage, and the review and coverage steps above are what make them trustworthy. Keep the speed and add the view the session lacks. For the trust question in more depth, read how to trust AI-written tests.

How does TestChimp help?​

TestChimp sits across sessions and across agents. Scenarios live as Markdown in your repo, tests link to them, and coverage is shown per scenario. When an agent changes a test, TestChimp flags it for human verification, and TrueCoverage (Growth plan or trial) compares what users actually do in production with what your tests cover, so the next test to write is a ranked list and not a guess. It works alongside Claude Code, Cursor, Devin and other agents, using the same repo they already edit. The wider workflow is described in a QA workflow for agent-built apps.

Frequently asked questions

Can Claude Code, Cursor or Devin replace QA?

They can replace much of the test-writing and test-running work. They do not replace the cross-session view: which tests were reviewed, which scenarios are uncovered, and what to test next. Teams still need someone or something to own that.

Do I need different QA for Claude Code than for Cursor?

Mostly no. The gaps come from the session boundary and apply to every coding agent. What differs is the controls each tool offers, such as hooks, permissions and rules, so check its documentation.

How do I make an agent run tests before it says it is done?

Use a hook where your agent supports one. Devin's documentation, for example, describes a Stop hook that can block the agent from finishing and instruct it to run the test suite. Also make the suite a required status check in CI so the merge is gated whatever the agent does.

Who should review tests an agent wrote?

A person who understands the requirement, ideally the owner of that area. Assign test directories in CODEOWNERS so a review is requested automatically.

What should I test first in an agent-built app?

The flows where failure costs money or trust: sign-up and login, payments, data creation and deletion, and anything with permissions. Write them down as scenarios, then make sure each has a reviewed end-to-end test.

Add the QA layer your coding agents cannot see

Scenarios in your repo, tests linked to them, and flags when an agent changes one. 14-day free trial, no credit card.

Start a 14-day free trial · Book a demo