Why this question blew up on Hacker News​

On 19 October an Ask HN thread quoted a Meta engineer saying the code he ships now "feels like a black box" and asking how much he needs to understand before he's comfortable moving on. It passed 600 points and 700 comments, so plenty of people feel the same.

The replies split into roughly two camps. One says you must read the code before shipping it, because "this is less a matter of quality and more a matter of responsibility." The other says engineers have always shipped code they didn't fully read: principal engineers and engineering managers have relied on testing, design docs and release management for years.

Both camps are right about something. Responsibility doesn't move to the agent. And reading every line of a 1,000-line agent diff every day doesn't scale. Here's the middle path the most useful replies point to.

1. Decide where you still read every line​

Blast radius should decide how much you read. One commenter vibe codes the UI freely but still writes the SQL schema entirely by hand, because getting that wrong "tends to be far more catastrophic." Another puts it plainly: a stateless frontend needs little more than a visual check, while payments or client-server security deserve a real audit.

Make that list explicit for your codebase. Data migrations, auth, money movement, permissions and anything irreversible go on the "a human reads it" list. Copy changes and layout tweaks don't.

2. Own the design and let the agent fill it in​

The most upvoted reply in the thread comes from someone who doesn't let the LLM design the code at all. They lay out the abstractions and how they interact, and the agent implements the gaps. They might not know how each unit is written, but they know each unit's job and how the units connect.

That's enough understanding to debug, to review a diff at the boundary level, and to notice when the agent is quietly reshaping your architecture.

3. Write down what the product should do before the agent writes code​

If you can't read all the code, you need something else you can read. One commenter's workflow starts with interview-based design and spec: the agent asks about whatever it finds ambiguous before writing anything. Someone else boiled it down to "get really good at defining acceptance criterion."

Keep these as short user stories and scenarios in the repo, next to the code. A scenario like "a guest can't check out without a delivery address" is something a reviewer can understand in five seconds, and it gives tests something to point at.

4. Keep that description current as decisions happen​

The same commenter cites Peter Naur's idea that programming is building a theory of the system in your head. Writing code by hand was how we built that theory. With agents, a lot of the real product decisions now happen in agent chats and meetings instead: "guests must enter an address", "retry the webhook twice, then alert".

If those decisions never make it back into your stories and scenarios, the written description goes stale within weeks, and so does every check built on it. Make "update the scenario" part of finishing a change, or use tooling that picks those decisions up for you.

5. Don't let the agent grade its own work​

This is the sharpest worry in the thread. One commenter is uneasy when the AI writes both the code and the tests: "They might agree with each other and still miss the same thing." The replies have practical fixes:

• Split the roles: "The agent writing the code never writes the checks it's judged by", and those separate checks caught far more.
• Test the tests with mutation testing. One team added Stryker to CI and its first run surfaced hundreds of mutations the suite didn't catch.
• Ask for a failing test first. One reply argues that seeing a test fail before the fix goes in gives you a lot of confidence.

For a deeper checklist, see our guide on how to trust AI-written tests.

6. Measure coverage against intent, not test count​

An agent can write 400 tests that all hit the happy path. What you need to know is which of your scenarios have a test that would fail if the behaviour broke, which API operations and error responses were exercised at all, and whether the flows real users depend on are covered. A test count answers none of that. Track coverage against the scenarios from step 3, and against your API spec, and review the gaps rather than the totals.

7. Turn every escaped bug into a guardrail​

One of the most practical replies treats agents the way engineering management treats teams. Every time the agent ships a bug, ask how to rule out that whole class of bug. Their example: agents kept writing broken database migrations, so they added a migration testing framework that runs each migration over data in the old schema and checks the result. Do that repeatedly and trust is something you earn one bug class at a time.

8. Keep a named human on every release​

Someone still has to say "this is ready." As one commenter put it, you can't fire the agent when things go wrong. Before each release, a person should look at which scenarios changed, which have reviewed tests behind them, what's still uncovered, and what changed in the high blast radius areas from step 1. Then they sign off by name.

The catch: doing all of this by hand every week​

Each of those steps works. The trouble is keeping them going when an agent ships ten changes a day. Specs drift, nobody updates the scenarios, the suite fills with tests nobody reviewed, and "are we covered?" turns into a gut call again. Claude Code can write the tests. It won't tell you on its own whether the right things are tested or whether the description it tested against is still true.

How TestChimp makes this ongoing​

TestChimp runs the loop around your coding agents so the steps above happen on every change instead of when someone remembers.

The product description stays current. AgentWatch watches your team's coding agent sessions (Cursor, Claude Code, Codex and others), spots product decisions, and proposes updates to your stories and scenarios. MeetBots do the same for decisions made on Meet, Zoom, Teams or Webex calls. You approve each update, and it lands as Markdown in your repo. That covers steps 3 and 4.

QA runs against that description. Tests link back to scenarios, so every check points at a behaviour someone agreed on.

You see what's actually covered. Semantic coverage shows which parts of the product's scenario space have tests and where duplicates pile up. API contract coverage overlays real request and response payloads from test runs on your OpenAPI spec to show untested operations, fields and status codes. TrueCoverage is a Growth-plan feature that compares what real users do in production with what your tests exercise. Your coding agents get those gaps as work to close, which is step 6 running continuously.

Humans approve agent-written tests and releases. With Verified Tests, a person reviews a test against its scenario and stamps it as verified. If an agent later patches that test, the verification can be marked stale until someone checks it again. Release governance rolls test runs, issues and release checks into one readiness view per version, and CI can gate on the same data. That's step 8 with evidence attached.

You still own the design and the high blast radius code. TestChimp makes sure the behaviour you agreed on is written down, tested and visible every time the agent ships.

FAQ​

Do I need to read every line of AI-generated code before shipping?​

No, but someone needs to read the high blast radius parts: schema and migrations, auth, payments, permissions. For the rest, focus on understanding the design and checking behaviour against written scenarios with tests the coding agent didn't grade for itself.

How do I review code from Claude Code when the diffs are huge?​

Review at the boundaries. Check the diff against your design (did it add a dependency, change an interface, touch a high-risk module?), then check the behaviour against the scenarios it was meant to satisfy. Ask the agent to split large changes into smaller commits and to explain parts you don't follow, then confirm the explanation against the code.

Can I trust tests that the same AI wrote along with the code?​

Only after checking them. The code and tests can share the same blind spot. Have the tests derived from scenarios written before the code, review that each test can fail for the right reason, and add mutation testing or a separate checking agent where the risk is high.

What should a team check before releasing agent-written code?​

Which scenarios changed, which have a reviewed test behind them, which API operations and error paths were exercised, what real-user flows remain uncovered, and whether anything changed in high-risk areas. Then a named person signs off.

How do we keep specs current when decisions happen in agent chats?​

Treat "update the scenario" as part of done, or capture decisions automatically. TestChimp's AgentWatch proposes story and scenario updates from coding agent sessions, and MeetBots do the same from meetings, with a human approving each change.

Is TestChimp a replacement for Claude Code or Playwright?​

No. Claude Code and other coding agents write the code and the tests. TestChimp works alongside them: it keeps the product description current, measures QA coverage against it, feeds the gaps back to the agents, and gives humans approval over tests and releases.

Highlights: TestChimp now installs as a Claude Code plugin with the hosted MCP server built in, ExploreChimp runs on your own machine through your coding agent, and Git sync copes with large pushes and provider rate limits.

Install TestChimp as a Claude Code plugin​

The testchimp-skills repo is now a Claude Code plugin that bundles the skill and the hosted MCP server at https://mcp.testchimp.io/mcp (skill v1.0.64 to v1.0.73, 18 Oct).

/plugin marketplace add testchimphq/testchimp-skills
/plugin install testchimp@testchimp

Restart the session, run /mcp, choose plugin:testchimp:testchimp and sign in. You pick the TestChimp project on the consent screen, so there are no API keys or project IDs in your MCP config. Other changes in the same releases:

• Setup is split in two: one person runs /testchimp project-init once per project, and everyone else runs /testchimp init.
• Claude Code's / menu lists the core QA commands as /testchimp:<name> (for example /testchimp:run-qa), trimmed to 16 core commands in v1.0.73.
• The README documents a permissions rule (mcp__plugin_testchimp_testchimp__*) so TestChimp's MCP tools don't prompt on every call.
• Non-Claude hosts keep working with the same repo (npx skills add testchimphq/testchimp-skills, or a git clone into your agent's skills folder).

Safer runner credentials and MCP tool annotations (@testchimp/cli 0.1.92 and 0.1.93, 18 Oct)​

• 0.1.92 adds a title and the standard MCP hints (read-only, destructive, idempotent, open-world) to all 97 tools on tools/list, so agents and MCP clients can tell read-only tools from ones that change data.
• 0.1.93: agents on the hosted MCP server write the runner key to the gitignored .testchimp/mcp.json with a file-write tool, so the key never appears in a shell command. The local-only save-workspace-creds tool is hidden on the hosted (HTTP) server, and init status values use one consistent format.

ExploreChimp runs on your machine (1.0.15, 14 Oct)​

ExploreChimp UX analysis now runs locally through your coding agent instead of in TestChimp's cloud. The app gives you a ready-to-copy prompt from SmartTests and onboarding. The cloud-hosted exploration wizards and the older UI Test Studio and Test Tree Map screens are retired. Team invitations also got smoother: accepting an email invite keeps its verification, and re-inviting a removed teammate sends a fresh invite.

Git sync for big and busy repos (1.0.16 and 1.0.17, 14 Oct)​

• Pushes that touch many tests or plans, or create a new branch, sync reliably, with heavy updates processed in the background.
• When GitHub or GitLab rate limits are hit, sync now pauses and finishes after the limit resets instead of skipping files. A repo connected to several projects downloads each changed file once per push.
• New users can watch the TestChimp intro video from the dashboard.

Smaller fixes and docs (1.0.18, 17 Oct)​

• Smart Smoke selection through the agent API no longer fails with HTTP 400 when an agent sends testSuite or folderPath as a string.
• If a repo is connected but the plans or tests folder isn't mapped, Plans and Sync now say so and link to the mapping screen.
• With only the demo project, Quick Actions offers Create Project.
• New comparison pages: Meticulous alternatives, Autosana alternatives, TesterArmy alternatives and TestChimp vs Klarent.

Packages​

• @testchimp/playwright 0.3.0 (14 Oct) and 0.3.1 (16 Oct): now requires Node.js 20.9 or later. Dependency updates include axios 1.20.0, ai-wright 0.0.13 and Mobilewright 0.0.65.
• ai-wright 0.0.13 (14 Oct): axios bumped to ^1.20.0 for Dependabot security advisories.

Full product release notes: /changelog.

TL;DR: If Claude or Cursor already writes your tests, keep it that way. The hard part was never typing the test. It is knowing which tests matter, whether anyone checked them, and whether the release is ready. TestChimp measures your QA posture (requirement coverage, API contract coverage and real user behaviour coverage) so your coding agent knows exactly where the gaps are and can close them. Claude writes tests. TestChimp tells it which ones matter.

"Claude can write tests too, so why bother?"​

It's a fair question, and we hear it often. The answer starts with agreeing: yes, Claude can write tests, and it should. A coding agent with Playwright and a decent CLAUDE.md will produce a working end-to-end test in minutes.

But look at what happens a few months after a team starts doing this at scale. The suite has grown fast. CI is mostly green. And yet nobody on the team can answer a few basic questions:

• Are our requirements actually tested?
• How much of our API is exercised?
• Are the flows our users actually use covered?
• Which of these tests has a person ever looked at?
• What should we test next?
• Are we ready to ship this release?

None of these are test-writing problems. Writing more tests doesn't answer them. They are measurement problems, and what you can't measure, you can't improve.

What the agent can't see on its own​

A coding agent works from its context window: the code it's touching, the prompt, maybe a rules file. That is plenty for writing a test. It isn't enough to decide which tests are worth writing.

So the agent does the reasonable thing: it writes tests near whatever it built most recently. Coverage clusters. Important but unglamorous flows stay untested. And because the agent wrote both the code and the test, a misread requirement shows up in both, and the suite stays green.

Measure first: three questions a dev lead should be able to answer​

TestChimp gives your agent, and your team, three measures of what is actually covered.

1. Requirement coverage: are your requirements tested?​

Stories and scenarios live as markdown files in your repo, in a plans/ folder that syncs with TestChimp. Tests declare the scenario they cover with a Playwright annotation:

test('guest can cancel a booking', {
  annotation: [{ type: 'scenario', description: '#TS-214' }],
}, async ({ page }) => {
  // ...
});

Coverage rolls up per folder, release and environment, so you can see which scenarios have no tests, no recent runs or a failing latest run.

Requirements also go stale. Decisions now happen in meetings and in coding-agent chats, and rarely make it back into a spec. MeetBots capture what was agreed on calls, and AgentWatch spots product decisions made in Claude Code or Cursor sessions. Both propose story and scenario updates that you approve, so the plans your agent tests against stay current.

2. API contract coverage: how much of your product surface is tested?​

Connect your OpenAPI specs and run your tests with API capture on. TestChimp shows per-operation coverage, plus request field, response field and response code coverage. Traffic that hits undocumented paths or fields is flagged too. Each gap can go straight to your agent as a create-tests task.

3. Real user behaviour coverage: are the flows your users actually use tested?​

TrueCoverage captures the same user events in production and during test runs, on web, iOS and Android. Lay them over each other and you see which real user pathways have no test coverage, ranked by demand, duration and drop-off. That turns "what should we test next?" from a guess into a ranked list.

Then improve: let the agent close the gaps​

Measurement is only useful if it changes what the agent does next. That is where the TestChimp skill comes in. Install it in Claude Code, Cursor or another agent host, and you get workflows that turn the measures into work:

• /testchimp run QA on a PR: works out what the change affects, updates plans, writes or fixes tests for the scenarios in scope, and runs the related tests.
• /testchimp upkeep on a cadence: pulls the highest-value requirement and TrueCoverage gaps, fixes recently failing tests, and keeps instrumentation current.
• Smart-Smoke: impact analysis picks the tests related to a change and an optimal subset of a large suite, so every PR doesn't have to run everything.

Your agent still writes the tests. It just starts from a prioritised list of gaps instead of a blank page. The result is usually fewer, better-aimed tests rather than a bigger suite.

You can run these yourself in your agent, or let them run automatically on events like a failing CI batch or a release moving to ready.

Keep humans on the parts that need judgment​

Agent-written tests need a second pair of eyes, and that person shouldn't have to re-read the whole suite. In TestChimp, a person inspects the run (steps, screen captures and code) and marks a test verified. When an agent later patches a verified test, the test is flagged for human re-verification by default, so a quiet "fix" that loosens an assertion doesn't slip through.

Run evidence such as screenshots, traces and video follows your Playwright config, so you decide how much is captured.

Know whether the release is ready​

All of this comes together per release. Each release gets one clear picture of how ready it is to ship: test runs, requirement coverage, and verification status. Verified tests show a verified badge; unverified tests are visible. The gate can be checked programmatically, so CI can ask "is this release ready?" and get an answer based on evidence, not on a green tick.

What doesn't change​

• Your tests stay standard. Playwright on web, Mobilewright for native iOS and Android, in your own repo. npx playwright test still runs them.
• Your plans stay readable. Markdown with frontmatter, reviewed by pull request.
• Your agent stays your agent. TestChimp supplies the signals and the playbooks. Claude or Cursor does the work.

Who this is for​

If your team only just found out that Claude can write tests, enjoy it. You'll feel the problems above soon enough. If you are already shipping a lot of agent-written code and your suite has grown faster than your confidence in it, that is the point where measurement pays off.

Claude writes tests. TestChimp tells it which ones matter.

Frequently asked questions​

If Claude already writes our tests, what does TestChimp add?​

Measurement and direction. TestChimp shows requirement coverage, API contract coverage and real user behaviour coverage, then feeds the gaps to your agent through workflows like /testchimp run QA and /testchimp upkeep. Your agent writes the tests; TestChimp decides which ones are worth writing.

Do I have to change test frameworks?​

No. Tests are standard Playwright specs on web and Mobilewright specs on native mobile. Optional natural-language steps (ai.act, ai.verify, ai.extract) are available on web.

How do I know an agent-written test checks the right thing?​

Have a person inspect it and mark it verified in TestChimp. If an agent later edits a verified test, it is flagged for re-verification by default.

Does TestChimp support mobile apps?​

Yes. Native iOS and Android are supported through Mobilewright, and TrueCoverage works on web, iOS and Android when instrumented.

How do I get started?​

Connect your Git repo, map a tests folder and a plans folder, install the TestChimp skill in your agent, run /testchimp init, then run /testchimp run QA on your next PR.

Related docs​

• What is TestChimp?
• Requirement traceability
• API contract governance
• TrueCoverage
• Verified Tests
• Smart-Smoke
• Upkeep
• Mobile testing
• Release management
• Meet-Bots
• AgentWatch
• Workflows
• Previous article: Who checks the tests your AI agent wrote?

That sounds obvious, but it is the problem I keep hearing from teams that let Claude Code, Cursor or Codex write their end-to-end tests. A recent r/ClaudeCode thread put it well. The author gave the agent every advantage: data-testids in the markup, seed credentials in CLAUDE.md, an architecture doc and Playwright MCP. The suite still had:

• data-testids that exist nowhere in the app
• a query parameter the app never reads
• a test navigating to a route that does not exist
• test.skip() used every time a test needed data that wasn't already there
• after a feature was deleted entirely, one test stayed green, because its assertion checked for text anywhere on the page

The last one is the dangerous kind. Broken tests get noticed. Tests that pass while the feature is broken don't.

This post covers the checks we use to catch these, in rough order of cost. The Playwright snippets were run against @playwright/test 1.64.1.

Why agent-written tests pass for the wrong reasons​

These patterns show up whichever model writes the tests:

Agents tend to do these things because they are rewarded for getting the suite to go green. A comment in the r/Playwright thread on Test Agents describes the same thing as a "fix-run-fail-fix loop". If the only signal is green or red, green wins.

Check 1: Make the agent prove the test goes red first​

The cheapest check is the old TDD rule. A new test should fail before the change and pass after it. For a bug fix, ask the agent to run the test against the unfixed code and paste the failure output into the PR description.

You can enforce part of this in CI by running the new or changed spec files against the base branch:

# In a PR job (checkout with fetch-depth: 0): which spec files changed?
git fetch origin main
CHANGED=$(git diff --name-only origin/main...HEAD -- 'tests/**/*.spec.ts')
[ -z "$CHANGED" ] && exit 0  # no new or changed specs, nothing to check

# Check out the app code from main, keep the new tests
git checkout origin/main -- src/
npx playwright test $CHANGED --reporter=line && {
  echo "New tests pass on main. Do they test the change?"
  exit 1
}

This only fits tests written for a behaviour change. For tests that cover existing behaviour, you need the next check.

Check 2: Break the app on purpose with page.route​

You don't need a full mutation testing tool to see whether a test is sensitive to the thing it claims to cover. Playwright's network routing lets you corrupt one response and confirm the test fails.

Say the agent wrote this:

// tests/search.spec.ts
import { test, expect } from '@playwright/test';

test('search filters results by destination', async ({ page }) => {
  await page.goto('/search?destination=lisbon');
  await expect(page.getByText('Hotel Aurora')).toBeVisible();
});

Now write a "should fail" twin that removes the expected hotel from the API response:

// tests/sabotage/search.sabotage.spec.ts
import { test, expect } from '@playwright/test';

test('search test notices when results are wrong', async ({ page }) => {
  test.fail(true, 'sabotaged response: the assertion must not pass');

  await page.route('**/api/search*', async (route) => {
    const response = await route.fetch();
    const json = await response.json();
    json.results = json.results.filter((h: { name: string }) => h.name !== 'Hotel Aurora');
    await route.fulfill({ response, json });
  });

  const searched = page.waitForResponse('**/api/search*');
  await page.goto('/search?destination=lisbon');
  await searched; // make sure the sabotaged response actually arrived
  // Same assertion as the real test. We EXPECT this to fail.
  await expect(page.getByText('Hotel Aurora')).toBeVisible({ timeout: 3000 });
});

Playwright's test.fail() marks the test as expected to fail. It is reported as passed when it fails, and as an error when it unexpectedly passes.

If the sabotage test passes, your real test is too broad. In this example it passes when "Hotel Aurora" also appears in a "recently viewed" sidebar. That is exactly the failure from the Reddit thread.

Two traps we hit while testing this:

• Wait for the sabotaged response. Without await searched, the assertion can pass on the sidebar before the search results load. The test then ends while the route handler is still running, Playwright reports a "Test ended" error from route.fetch, and test.fail() counts that error as the expected failure. The sabotage twin looks fine while proving nothing.
• The real test has the same race. getByText('Hotel Aurora') can match the sidebar before results render, so the original test passes even when search is slow or broken. Scoping the assertion (next check) fixes both.

Don't write these for every test. Write them for the flows that make you money (signup, checkout, billing) and for any test that has never once failed in CI.

Check 3: Scope assertions to the part of the page that matters​

The fix for broad assertions is usually scoping. Playwright 1.64 added Locator.within(), which reads naturally:

const results = page.getByRole('region', { name: 'Search results' });
const hotel = page.getByRole('heading', { name: 'Hotel Aurora' });

await expect(hotel.within(results)).toBeVisible();

Chaining (results.getByRole(...)) works the same way on older versions. Either way, the sidebar can no longer make the test pass.

Better still, assert on the outcome instead of on the presence of text:

const cards = page.getByRole('article').within(results);
await expect(cards).toHaveCount(3);
await expect(cards.first()).toContainText('Lisbon');

Check 4: Catch the lazy patterns with lint, not review​

Some patterns should never reach a PR. eslint-plugin-playwright already has rules for most of them:

// eslint.config.mjs
import playwright from 'eslint-plugin-playwright';

export default [
  {
    ...playwright.configs['flat/recommended'],
    files: ['tests/**/*.ts'],
    rules: {
      ...playwright.configs['flat/recommended'].rules,
      'playwright/no-skipped-test': 'error',
      'playwright/expect-expect': 'error',
      'playwright/no-wait-for-timeout': 'error',
      'playwright/no-networkidle': 'error',
      'playwright/no-conditional-in-test': 'error',
      'playwright/no-force-option': 'error',
    },
  },
];

expect-expect catches tests with no assertions. no-skipped-test catches the "skip instead of seed" habit. no-networkidle matters because networkidle is itself a common source of flakiness. The Zenn team found this when their own agent skill recommended it (write-up).

Put the same rules in your CLAUDE.md, AGENTS.md or .cursor/rules so the agent avoids them in the first place. Keep lint as the backstop, because agents don't always follow instructions.

Check 5: Remove random failures so red means something​

Flaky tests teach a team to ignore red, and then the checks above don't matter. Three recent Playwright features help here:

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  retryStrategy: 'isolated', // 1.62: retries run at the end, one at a time
  use: { trace: 'retain-on-failure' },
});

# 1.64: random order surfaces tests that depend on each other
npx playwright test --shuffle
# reproduce with the printed seed
npx playwright test --shuffle 271828182

// 1.63: tests sharing a resource never run at the same time
test('update account settings', { lock: 'user-settings' }, async ({ page }) => {
  // ...
});

Sources: Playwright release notes for 1.62, 1.63 and 1.64. Treat a test that passes only on retry as a bug to fix.

Check 6: Seed data so there is nothing to skip​

Agents use test.skip() when the state they need isn't there. Give them a fixture that creates it through your API, so skipping is never the easy path:

// tests/fixtures.ts
import { test as base } from '@playwright/test';

type Booking = { id: string; propertyName: string };

export const test = base.extend<{ booking: Booking }>({
  booking: async ({ request }, use) => {
    const res = await request.post('/api/test/bookings', {
      data: { property: 'hotel-aurora', nights: 2 },
    });
    const booking = (await res.json()) as Booking;
    await use(booking);
    await request.delete(`/api/test/bookings/${booking.id}`);
  },
});
export { expect } from '@playwright/test';

Tell the agent in your rules file: "If a test needs data, use or add a fixture in tests/fixtures.ts. Never skip."

A checklist you can paste into your PR template​

### E2E tests in this PR
- [ ] Each new test was seen failing (output pasted, or the CI base-branch check ran)
- [ ] Assertions are scoped to a region/component, not the whole page
- [ ] Assertions check an outcome (data, count, state), not just that an element exists
- [ ] No test.skip, waitForTimeout, networkidle or force: true (lint passes)
- [ ] Test data comes from fixtures, not from whatever is in the DB
- [ ] Critical flows have a sabotage twin marked test.fail()
- [ ] Ran with --shuffle at least once

Where this leaves you​

None of this is new testing theory. What changed is the author: very fast, very eager to see green, and unaware of which parts of your page matter. The checks that scale best are the ones the agent can't skip: lint rules, the base-branch check and sabotage twins on your money paths.

We hit the same problem building TestChimp. That is why, in TestChimp, a test only gets a "verified" badge after a person on the team has inspected it, and an agent's later fix can flag that badge as stale for another look. You don't need a platform to get most of the value, though. The checklist above is a good start.

FAQ​

How do I know if an AI-generated test is useful?​

Break the behaviour it claims to cover and see if it fails. Corrupt the API response with page.route, remove the feature behind a flag, or run the test against the base branch. If it stays green, it is not testing what its name says.

Should I let an agent auto-fix failing Playwright tests?​

Only with guardrails. Allow fixes to locators and waits. Block any change that removes or loosens assertions, adds retries or adds test.skip, and require a human to review the diff. In the r/Playwright thread above, people found the healer agent useful for selector drift and weak on timing problems.

Is mutation testing practical for end-to-end tests?​

Full mutation testing is usually too slow for E2E suites. A cheaper middle ground is targeted fault injection: a few "sabotage" tests on critical flows that corrupt network responses and are marked test.fail().

Which lint rules catch the most common agent mistakes?​

From eslint-plugin-playwright: expect-expect, no-skipped-test, no-wait-for-timeout, no-networkidle, no-force-option and no-conditional-in-test.

What Playwright version do I need for Locator.within() and --shuffle?​

Playwright 1.64. Test locks need 1.63 and retryStrategy: 'isolated' needs 1.62. On older versions, chain locators and use --repeat-each to look for flakiness.

TL;DR: We shipped Bonobi. Every team member (developer, QA engineer, QA lead, PM) gets their own Bonobi, an AI QA bot that handles their share of the QA process in the background: updating requirements from coding-agent chats, writing E2E tests when a feature is done, fixing assigned issues, triaging failing batches, and reporting release health. TestChimp coordinates the swarm: it tracks your QA posture, routes each event to the right person's Bonobi, and keeps an audit trail of what every Bonobi did. Bonobi proposes, people approve. If you want QA to run with agents, mostly in the background while your devs build, this is how we think it should work. Docs: Bonobi · Install and use.

Coding got agents. QA got a longer queue.​

Coding agents changed how fast a team ships. A developer with Cursor or Claude Code can finish in an afternoon what used to take a sprint. QA didn't speed up at the same rate, so the work piles up in the same places it always did, only faster:

The common answer is "add an AI testing agent". That helps with one item on the list. It doesn't help with the rest, because these responsibilities aren't one person's job.

QA is a team sport​

Developers own tests for what they build. QA engineers own test runs and failures. Leads own release health. PMs own requirements and the go / no-go call. Every role holds a piece of quality, and every piece slips when that person is heads-down.

So we stopped asking "what should the QA agent do?" and asked "what would it look like if every role had its own QA agent?"

What is Bonobi?​

Bonobi is TestChimp's QA bot on Grok, one per team member. It acts as that person's QA counterpart on one project. It receives the events that matter to its owner (their git pushes, the issues and scenarios assigned to them, test batches on their branches, releases), proposes the next QA step, and runs it through a TestChimp workflow once the owner approves.

Each person installs their own Bonobi. During onboarding it asks for their role and pre-selects the work that role usually owns. Together, these Bonobis form a QA swarm that covers the whole team's quality responsibilities.

How does TestChimp coordinate the swarm?​

A swarm of agents without coordination is just more notifications. TestChimp is the coordination layer.

It knows your QA posture. Bonobis aren't guessing from code. They work from what TestChimp already tracks about your product: requirements as code (stories and scenarios in your repo), API contract coverage, real user behaviour from TrueCoverage, semantic coverage, releases, issues, test executions and verified tests.

It routes every event to the right Bonobi.

• Your pushes and your assignments go to your Bonobi. Your teammate's Bonobi never sees them.
• A failed E2E or k6 batch goes to the Bonobi of the latest author on that branch, so two Bonobis never race to fix the same failure.
• Release events go to every Bonobi subscribed to release health, each tuned to its owner's role.

It keeps delivery sane and auditable. Events are paced (grouped per person and type over a few minutes), queued while a Bonobi is offline, and acknowledged by that Bonobi. Every Bonobi has its own id, and TestChimp records what was sent to which Bonobi and which workflows each one ran.

What does each role's Bonobi do?​

The developer's Bonobi: tests and specs keep up with the code​

You push. Your Bonobi reads the commits and judges whether the feature looks done (WIP commits and bursts of pushes mean "wait"). When it does, it:

1. Asks AgentWatch for the requirement changes you decided in your coding-agent chats, and shows you the plan. You approve; the stories and scenarios in your repo update.
2. Proposes E2E tests for the changed behaviour, tied to those updated scenarios. You say "Go ahead"; it writes them and asks you to verify the runs.
3. Picks up issues assigned to you with a fix plan.

This is the hero flow, and the reason we built Bonobi: requirements updated from how the feature was actually decided, then tests written against them, without the developer stopping to context-switch into QA.

The QA engineer's Bonobi: no more chasing​

Scenarios assigned to you in a test run show up with a link and an offer to walk through them, record results, or automate them. Failing E2E batches on your branches arrive triaged (test needs an update, product bug, or flaky) with a proposed fix. k6 threshold breaches come with a baseline comparison. Turn on requirement updates and it drafts specs from the meetings you were in.

The QA lead's Bonobi: posture without a status meeting​

When a release is created or changes status, Bonobi sends a short heads-up: blockers, failing areas, open test runs, tests nobody has verified. Every week it sends a QA posture digest. When a release heads to done with gaps, it offers a release check.

The PM's Bonobi: go / no-go in three lines​

Release heads-ups written for a decision (blocking issues, failing priority scenarios, due date), a weekly health digest, and requirement updates proposed from meeting summaries, so what was agreed in the room becomes a story.

What is the developer experience like?​

The point is that QA happens next to the work, not after it.

No new dashboard to check. No QA ticket queue to babysit. You approve or say "hold off", and Bonobi takes it from there.

Does Bonobi act on its own?​

No, and that's deliberate.

• Propose, then wait. No story, scenario, test, issue, commit or local command happens without the owner's explicit approval in the conversation.
• Owner's permissions only. A Bonobi acts as its owner and nobody else. TestChimp's existing access controls apply.
• Existing workflows, not improvisation. Approved work runs through the same TestChimp workflows your coding agents use (author plans, create tests, fix issue, fix test failures, release check), each with its own plan and report.
• Your machine stays yours. Local steps (repo mapping, AgentWatch, local test runs) run on your computer only after you grant access, with per-command approval.
• Pause all from your Bonobi's settings page whenever you want quiet.

Agents do the legwork. People keep the judgment.

Why is this the best way to run QA with agents?​

There are a few ways teams try to put agents on QA today. Here's how they compare:

The first column is how most teams start, and it's why green tests stop meaning much. The second fixes testing for one team and leaves the rest of the responsibilities where they were. Bonobi gives every role its agent, and TestChimp gives the swarm a shared picture of quality and a way to coordinate.

Why this belongs in TestChimp​

We've argued that skills are SaaS distribution, that Meeting Bots should bring product intelligence into the call, and that AgentWatch should capture decisions where they now happen: in coding-agent chats.

Bonobi is what those pieces were for. The skill and CLI are the workflows each Bonobi runs. AgentWatch and Meeting Bots keep requirements current. Plans as code, coverage, releases and verified tests are the shared picture of quality. And the swarm makes sure every person on the team has a Bonobi working on their part of it.

A QA counterpart for every team member.

Get started​

1. Make sure your project's one-time setup is done (/testchimp project init). Your bot can walk you through it.
2. Add the Bonobi template in Grok.
3. Ask it to "set up my QA bot", authorize it for your project, and paste its webhook into User Settings → My Bots.
4. Pick your role. Let it map your local repo and pair AgentWatch if you want requirement updates.
5. Keep building. Approve the first proposal when it lands. Then get the rest of your team on it.

Full guide: Install and use Bonobi · Overview: Bonobi.

Frequently asked questions​

What is Bonobi?​

Bonobi is TestChimp's QA bot on Grok: an AI agent that acts as one team member's QA counterpart on one TestChimp project. It receives that person's project events (pushes, assigned issues and scenarios, test batch results, releases), proposes the next QA step, and runs a TestChimp workflow after the person approves.

How is a QA swarm different from a single AI testing agent?​

A single agent covers one role's work, usually test execution. Bonobi gives every team member their own QA counterpart, so requirement updates, test authoring, issue fixes, failure triage, manual test coordination and release health are all covered. TestChimp routes each event to the right person's Bonobi and tracks the shared QA posture.

Does Bonobi make changes without approval?​

No. Bonobi proposes and waits for its owner's explicit approval before any change, test run, commit or local command. Read-only lookups and event acknowledgements are the only things it does without asking.

Which roles get a Bonobi?​

Developers, QA engineers, QA leads and product managers. The role you pick pre-selects capabilities: developers get requirement updates, E2E authoring and issue fixes; QA engineers get E2E authoring, test batch fixes and manual test coordination; QA leads get QA posture, batch fixes and manual coordination; PMs get QA posture and requirement updates. You can change any of them.

How does Bonobi keep requirements current?​

From two sources. AgentWatch reads your local coding-agent chats and drafts story and scenario updates after a push, and Meeting Bots provide meeting summaries Bonobi can turn into requirement updates. You approve the plan and the stories and scenarios in your repo update.

Which platforms does Bonobi run on?​

Grok today, installed from the Bonobi template. Dots support is coming soon.

Will my teammates' Bonobis get my events?​

No. Pushes, issues and scenarios go only to the Bonobi of the person they belong to. Failed batches go to the latest author on the branch. Only release events go to every Bonobi subscribed to release health.

Related​

• Bonobi overview
• Install and use Bonobi
• AgentWatch
• Meeting Bots
• Verified Tests
• Who checks the tests your AI agent wrote?
• TestChimp workflows

TL;DR: TestChimp has two layers. The platform layer runs the QA loop and governance: it orchestrates agents working through product risks and coverage gaps, and holds verification status, execution history and coverage signals. The repo layer is yours: tests are standard Playwright specs and plans are markdown stories and scenarios, in your own git repo and synced both ways with TestChimp. If you leave, the repo and its full git history stay with you. Verified Tests status is the only thing that lives solely in TestChimp; nothing else is lost when you leave, because everything else is either in git or covered by TestChimp's full download of data not stored in git.

What does vendor lock-in look like in AI testing tools?​

Lock-in in testing tools rarely looks like a contract clause. It looks like one of these:

• Tests stored as records in a hosted tool. You can read them in the vendor's UI, but leaving means an export you did not test until you needed it.
• A proprietary test format or runner. The tests only execute inside the vendor's platform.
• Requirements and test cases in a database. Plans are reachable only through the vendor's UI or API, so your coding agents cannot read them and migration means a data export project.
• History that does not travel. Teams that have moved between test management tools report that imports keep case titles and steps but flatten the history (a practitioner comment, not a TestChimp claim).

AI test generation adds a new version of the same question: when an agent writes hundreds of tests for you, where do those tests live, and in what format? If the answer is "in the vendor's system", your investment sits on their side of the door.

TestChimp is built the other way around. Tests are standard Playwright (or Mobilewright for native mobile) files in your repo. Plans are markdown files with frontmatter in your repo. The platform adds workflow, verification and coverage intelligence on top of those files. Per the docs, it orchestrates a feedback loop so agents keep seeing product risk and coverage gaps and act on them, while your files stay the source of truth.

What lives in your repo?​

You choose the folder names. The tests and plans folders are mapped in the project's repository settings, and TestChimp writes small marker files (.testchimp-tests, .testchimp-plans) so agents can find the roots. Sync is two-way (bi-directional). Changes made in TestChimp reach your repo as pull requests or merge requests on branches prefixed testchimp-, so they go through your normal review, and commits to the mapped plans and tests folders come back into TestChimp through the provider webhooks. Plans are kept aligned with your default branch, while SmartTests sync branch by branch.

What lives in the platform instead:

• Verification status. Verified Tests badges and who verified what are stored in TestChimp, not in source. This is the only thing that exists solely in TestChimp.
• Data not stored in git. Execution history and coverage roll-ups (test-run results reported from CI, requirement coverage views, and release intelligence), coverage signals (TrueCoverage, which compares production user events with test runs; API contract coverage, which is API schema coverage built from captured API traffic; and semantic coverage, where stories, scenarios, tests and events are mapped by meaning in the Semantic Canvas), and workflow objects (issues, releases, test runs, and cloud meeting transcripts) are platform data rather than files in your repo. TestChimp supports a full download of all data that is not stored in git, so this data is not lost when you leave.
• Stable IDs. Story and scenario IDs are tracked in the platform and referenced in the markdown frontmatter, which is how status and traceability stay consistent.

This split is deliberate. What a test does and what the product should do are files you own. Who inspected which test is platform state (Verified Tests status), and the remaining platform-side data can be downloaded in full. Meet-Bots and AgentWatch feed the plans side: they propose story and scenario updates from meetings and coding-agent chats, and you approve them into the plans folder, so the files stay current.

Details: SmartTests in your codebase, Code repository overview, GitHub sync for SmartTests, Requirement planning as code, Export plans to Git.

What does leaving TestChimp look like?​

Concretely, in the order a team would do it:

1. Merge everything pending. Merge or close any open testchimp- pull requests so your default branch has the latest plans and tests. From here the repo is complete on its own.
2. Keep your files. The plans folder, the tests folder, the k6/ tree, your Playwright config and your CI pipeline are already yours, with full git history. Nothing needs to be exported for these.
3. Replace optional AI steps. If tests use ai.act, ai.verify or ai.extract, those steps run through the @testchimp/playwright plugin. Replace them with plain Playwright locators and assertions before you remove the plugin. Tests that never used AI steps are already plain Playwright.
4. Remove the integration wiring. Take the @testchimp/playwright reporter out of playwright.config, delete the TESTCHIMP_API_KEY secret from CI, and remove agent wiring you added (MCP config, the ChimpHands workflow file if you used it, and the TrueCoverage RUM SDK if you instrumented an app).
5. Disconnect the repository. Remove the GitHub App or the GitLab project access token and webhook. Stop allowing the testchimp- branch prefix if you had added a rule for it.
6. Download your platform-side data. TestChimp supports a full download of all data that is not stored in git. Verified Tests status is the only thing that lives solely in TestChimp; nothing else is lost when you leave.
7. Keep or clean up the markers. The marker files and the #TS-<n> annotations are harmless metadata. Plain Playwright ignores the annotations, and you can keep them as a readable link from each test to its scenario, or strip them.

Your tests keep running with npx playwright test, and your plans stay readable in any editor, Git host or agent. What you lose is the platform layer: the Verified Tests status and badges, and the hosted workflows and views. Apart from that status, your data is either in your repo or available through the full download.

Migrating in from TestRail or Jira​

Both imports are documented at Import from External Sources (Linear is supported too). Both start in Test Planning, Import / Export menu, then Import from... and the source you want.

Jira (user stories):

• What you provide: Jira URL, email, an Atlassian API token, the project key, and optionally a JQL query to narrow what is imported. Credentials are used for the import and are not stored.
• What you choose: which Jira issue type is treated as a user story (required) and, optionally, which is treated as a test scenario. Leave the scenario type blank if your test cases live in TestRail or another tool, and only stories are imported.
• What you get: stories become TestChimp user stories, and scenarios (if you chose a type) become TestChimp scenarios linked to their parent story. You choose file naming (external ID, TestChimp ID, or AI deduce) and folder structure (grouped by a Jira field, or AI deduce), and the wizard previews counts first.

TestRail (test cases):

• What you provide: TestRail URL, your login email, and an API key (API access must be enabled in TestRail under Admin > Site Settings > API). Credentials are used for the import and are not stored.
• What you get: each TestRail case becomes a TestChimp scenario, a markdown file. The wizard previews the case count first, lets you choose file naming (external ID, TestChimp ID, or AI deduce) and folder structure (TestRail section hierarchy or AI deduce), and runs as an async job with progress.
• Stories: the TestRail import brings in cases as scenarios only. If your stories are in Jira, import them first, and cases with matching Jira refs are linked to them automatically. Cases whose refs do not match an imported story are still imported, just without the link.

Both:

• Undo: Import / Export > Revert recent import deletes everything created by the most recent import.
• Trial limits: the docs list up to 25 imported user stories for Jira and up to 100 imported cases for TestRail on a free trial.

After the import, sync the new markdown to your repo with Sync to Git Repo, which raises a pull request you review and merge. Then link your automated tests to scenarios with the #TS-<n> annotation, and keep running the old system in parallel only until coverage overlaps. If you have an existing Playwright or other E2E suite instead, the /testchimp import workflow brings it in with one approved plan.

How does this compare with hosted test case stores?​

This compares the two models in general terms. It is not a feature-by-feature review of TestRail; check any specific vendor's current documentation.

Hosted stores have real strengths, especially for manual test case libraries and organisations with existing TMS mandates. If that is your situation, a hybrid is reasonable: import scenarios, sync the markdown to git, and keep the old system for historical audit while new work is git-native.

For plans and pricing, see testchimp.io/pricing. Meet-Bots usage is bundled with the Team plan (20 hours) and the Growth plan (80 hours), and AgentWatch is included in both plans. Team is $420 per month and Growth is $667 per month, billed annually (shown as the effective monthly price). For a side-by-side view, see TestChimp vs TestRail and Best TestRail alternative.

Frequently asked questions​

What is git-native test management?​

It means test plans and automated tests are files in your own Git repository, reviewed by pull request, and not records that exist only inside a vendor database. In TestChimp, plans are markdown stories and scenarios and tests are standard Playwright files.

Do my tests still run if I stop using TestChimp?​

SmartTests are ordinary Playwright spec files with no proprietary format, and npx playwright test runs them without a special runner. Optional AI steps and reporting rely on the @testchimp/playwright plugin, so expect to replace or remove those.

What data stays only in the TestChimp platform?​

Only Verified Tests status lives solely in TestChimp. Everything else is either in your git repo or covered by TestChimp's full download of data not stored in git, so nothing else is lost when you leave.

Can I migrate from TestRail or Jira to TestChimp?​

Yes. The TestRail import wizard turns each test case into a TestChimp scenario, and the Jira import wizard brings in user stories (and scenarios, if you pick a scenario issue type). Both preview the counts first and can be reverted after the import. The TestRail import brings in cases as scenarios only, so import Jira stories first to get the links.

Is there vendor lock-in with AI test generation tools?​

It depends on where the generated tests live and in what format. If they are standard framework files in your repo, you keep them when you leave; if they are stored as records in a hosted tool, check the export options before you commit.

How do plans and tests get into my repo?​

You connect GitHub or GitLab and map a tests folder and a plans folder. Sync is two-way: TestChimp raises pull requests or merge requests on branches prefixed testchimp- for changes made in the platform, which you review and merge like any other change, and commits to the mapped folders are pulled back into TestChimp.

Related docs​

• Import from TestRail, Jira and Linear
• Requirement planning as code
• Export plans to Git
• What is TestChimp?
• Code repository overview
• GitHub sync for SmartTests
• SmartTests in your codebase
• TestChimp vs TestRail
• Quick start
• Previous article: Who checks the tests your AI agent wrote?
• Plans and pricing: testchimp.io/pricing

TL;DR: When an AI agent writes the code and the tests, green proves little, because the agent is checking its own work. TestChimp runs the QA process as a feedback loop: it surfaces product risks and coverage gaps, agents work through them, and results flow back in. Governance signals (Verified Tests, TrueCoverage, semantic coverage and API contract coverage) show what is actually covered and who verified it. Meet-Bots and AgentWatch keep requirements current, so tests are written against up-to-date intent.

Why do agent-written green tests prove nothing?​

Because the agent grades its own work. If the model misreads a ticket, its tests misread it the same way, and the suite stays green while the product does the wrong thing. Practitioners describe this as the agent "marking its own homework".

Three failure modes show up once agents write both sides (they are described in the Verified Tests announcement):

Green is an execution fact. A scenario link is a mapping. Neither tells you that anyone checked whether the test verifies the behaviour. Before agents, "who wrote this test?" was a workable stand-in for "who checked it?". With agents writing tests at speed, that stand-in no longer holds.

The answer has three parts: a QA loop that keeps agents working on the real risks, governance signals that show what is covered and who verified it, and requirements that stay current so tests are aimed at the right target.

How does TestChimp run the QA loop?​

TestChimp is a QA workflow layer for agents. Per the docs, it orchestrates a tight feedback loop so coding agents such as Claude continuously understand product risk, coverage gaps and where users actually spend time, and then act on them. Humans stay in control of intent (what the product should do), and agents do the execution and upkeep against that intent.

In practice the loop looks like this:

1. Plans live in your repo. Stories and scenarios are markdown files in a mapped plans/ folder. Plans and tests sync both ways between TestChimp and your repo, and changes arrive as pull requests or merge requests you review.
2. Per PR, run QA. The /testchimp run QA workflow chains plan, environment, SmartTest and smoke steps, with optional exploratory and TrueCoverage steps, under one scope.
3. In CI, results are reported back. The @testchimp/playwright plugin reports runs and traces to TestChimp and tags user events with test identity for TrueCoverage.
4. On a cadence, upkeep. /testchimp upkeep closes coverage gaps found from requirements and TrueCoverage, explores high-signal paths and cleans up.

The web app (or TestChimp Studio on the desktop) is the control plane where the risk list is visible: uncovered scenarios, coverage gaps, bugs found by exploration and workflow executions. Details: What is TestChimp? and Upkeep.

Green run alone vs green run with governance signals​

The next sections cover each signal.

What is a Verified Test?​

A verified test in TestChimp is a SmartTest a project member has inspected and confirmed actually covers a linked scenario, not merely that it ran green or is annotated.

(A SmartTest is a standard Playwright test in your repo with scenario links added.) To verify one, a reviewer opens a test execution, looks at the screen captures and steps from the run, reads the test code with View Test, and then uses the check badge next to the test name. Status is stored in TestChimp, not in the Playwright source, so agents keep authoring tests while humans record that they inspected them.

The badge has three states on a test:

• Hollow: none of the linked scenarios are verified.
• Grey: some, but not all, linked scenarios are verified.
• Blue: every linked scenario for that test is verified.

On coverage views, each scenario shows a filled badge when a covering test has been verified, a hollow badge when tests exist but nobody has inspected them, and no badge when there are no linked tests. Hovering a verified badge shows who verified it. Un-verifying is recorded as "manually unverified", so the audit trail stays.

Two honest details matter for governance:

• Stale markers. A blue badge with a red "!" means the test was verified, but an agent later patched it. By default, when an agent reports a fix to an existing test, the badge goes stale until someone re-verifies it. This can be changed in project settings.
• Inspection is deliberate. Clicking the coverage-side badge does not verify anything. It points you to the execution viewer, so verification stays an inspection action and not a bulk paint on a dashboard.

Details: Verified Tests and Link tests to scenarios.

What other signals show what is actually covered?​

Verified Tests answer "did a person check this test?". Three more signals answer "what is covered at all?":

• API contract coverage (API schema coverage). Connect your OpenAPI specs from git, run SmartTests with API capture enabled, and TestChimp shows per-operation coverage plus request, query and response field and response-code coverage. Traffic that hits paths or fields the spec never declared is marked as undocumented. Each gap can become an issue, a create-tests prompt for your agent, or a deliberate ignore. See API contract governance.
• TrueCoverage. The same user-event taxonomy is overlaid from production (via RUM SDKs) and from test runs, so you see which real user pathways have no test coverage, with metrics such as demand, duration, drop-off and depth. See TrueCoverage.
• Semantic coverage. Stories, scenarios, SmartTests, issues and TrueCoverage events are placed in one embedding space in the Semantic Canvas, so you can see what is close in meaning, whether it has a linked test, and which tests are near-duplicates. See Semantic Canvas.

Together with requirement coverage (next sections), these show where coverage is real, where it is thin, and where it is duplicated.

How do requirements stay current? (Meet-Bots, AgentWatch)​

Tests can only be aimed at the right target if the requirements do not rot. Much of what a team knows is decided out loud or in chat, then never written down. Meet-Bots and AgentWatch keep the QA surface current from the two places those decisions now happen, so the tests agents write are written against up-to-date requirements:

• Meet-Bots join Google Meet, Zoom, Microsoft Teams and Webex calls, transcribe once, and keep the meeting notes as tribal knowledge. After the call, the transcript can be handed to a workflow that drafts user stories and scenarios from what the team agreed. Team members can also @mention the bot for answers grounded in the stories, scenarios and test results. See Meet-Bots and Meetings, PII, and follow-ups.
• AgentWatch (in TestChimp Studio) watches local coding-agent sessions such as Cursor and Claude Code, spots when business rules or UX expectations were decided in chat, and proposes story and scenario updates. You review and approve the proposed plan, the plans/ folder updates, and you raise a PR. See AgentWatch and Enable and configure AgentWatch.

Both mechanisms propose; people approve. Stories and scenarios are markdown files with frontmatter in your Git repo (see Requirement planning), and plans sync both ways between TestChimp and the repo, so every requirement change is a reviewable diff. The tests are then authored and maintained against those scenarios, for example by the create-tests workflow, which writes tests for requirements or API coverage gaps. See Author plans and Create tests.

How does traceability connect tests to scenarios?​

Traceability is code-native. A test declares which scenario it covers with a Playwright annotation on the test:

test('successful login', {
  annotation: [{ type: 'scenario', description: '#TS-101' }],
}, async ({ page }) => {
  // ...
});

TestChimp links the test to the scenario by its #TS-<n> ordinal and tracks execution results from CI runs (via the @testchimp/playwright reporter). One test can list several scenarios. In Test Planning, the Insights tab shows requirement coverage for any folder, filterable by environment, release and time range, including scenarios with no linked tests, scenarios with no recent executions, and failing scenarios. Renaming a scenario does not reset verification as long as its ordinal stays the same; linking a different scenario is a new claim and starts unverified.

Put together, the chain is: scenario (kept current) -> linked test -> execution -> human verification. Each link is visible, and the last one has a named person and a time.

Details: Requirement traceability and Traditional traceability vs TestChimp.

What should a team review before release?​

A short checklist that maps to what the platform shows:

1. Uncovered scenarios. Any high-priority scenario with no linked test (Insights, Requirement Coverage, filtered to the release scope).
2. Failing or unexecuted scenarios. Scenarios that have tests but no recent execution, or whose latest run is red.
3. Hollow badges. Scenarios with tests that nobody has inspected. Decide which ones need a human look before this release.
4. Stale badges. Tests verified earlier that an agent has since patched.
5. API and user-behaviour gaps. Uncovered API operations or fields, and production events with no test coverage in TrueCoverage.
6. Requirement drift. Pending AgentWatch plans or meeting follow-ups that have not been reviewed yet, so the scenarios may not reflect what was decided.
7. Release evidence. The release view brings together test runs, requirement coverage, ExploreChimp findings and release checks; a gate can also be queried programmatically with an API key.

See Release management, Release intelligence and Programmatic release gating.

What does this not solve?​

• Verification is a human sanity check, not a proof of correctness. A reviewer can still approve a weak test.
• The loop assumes the scenarios are right. TestChimp helps keep them current and can score their quality (see Requirement quality governance), but a wrong requirement stays wrong until someone notices.
• Capture depends on the team using the bots and enabling AgentWatch. Decisions made outside meetings and agent sessions are not seen.
• Coverage signals depend on setup: API capture is off by default, and TrueCoverage needs the RUM SDK instrumented in your app.

Frequently asked questions​

How do I trust tests written by an AI agent?​

Do not rely on a green run alone. Look at what is actually covered (linked scenarios, API contract coverage, TrueCoverage, semantic coverage), and have a person inspect the run and the code and record that the test verifies the scenario.

Who should review AI-generated tests?​

A project member who can judge the requirement, such as a QA lead or the engineer who owns the feature. In TestChimp they open the test execution, inspect the screen captures, steps and code, and mark the test as verified.

Is a passing test the same as a verified test?​

No. A passing test is an execution fact, and a scenario link is a coverage claim, while a verified test has been inspected by a person who confirmed it covers the linked scenario.

What happens to the Verified badge when an agent edits the test later?​

By default the badge shows a stale marker when an agent reports a fix to the test, so someone can re-verify it. Teams can change this in project settings.

What does TestChimp do beyond verifying tests?​

It runs the QA loop: it orchestrates a feedback loop so agents keep seeing product risk and coverage gaps and act on them, while people own intent. Meet-Bots and AgentWatch keep requirements current, so tests are written against up-to-date requirements.

What coverage signals does TestChimp show for AI-written tests?​

Requirement coverage by folder, environment, release and time range; API contract coverage for OpenAPI operations and fields; TrueCoverage for production user events versus test runs; and semantic coverage through the Semantic Canvas. Verified Tests adds who inspected each test.

Related docs​

• What is TestChimp?
• Verified Tests (product guide)
• Verified Tests: When Agents Write the Coverage, Trust but Verify (announcement)
• Link tests to scenarios
• Requirement traceability
• API contract governance
• TrueCoverage
• Semantic Canvas
• Requirement planning as code
• Meet-Bots
• AgentWatch
• Release management
• Changelog
• Next article: Exit-safe by design: your tests and plans live in your git repo
• Plans and pricing: testchimp.io/pricing

The tribal knowledge problem did not go away​

Tribal knowledge—the discussion with a colleague over coffee—was often the hardest part of the SDLC to capture. Specs, tickets, and test plans lag behind what the team actually agreed.

Whether you like it or not, with the transition to agents those conversations have been repeated almost verbatim with coding agents: same product rules, same UX expectations, same edge-case constraints—usually a bit better filtered. That makes agent threads a near-perfect capture point for the knowledge that used to evaporate in hallways.

Meeting tools still miss the desk. Meeting Bots capture what is said in the call. AgentWatch captures what is decided while you build.

What AgentWatch does​

AgentWatch watches your agent interactions and identifies when:

• new business rules are decided
• UX expectations are finalized
• finer-grained constraints get defined

…then proposes updates to user stories, and adds or updates scenarios—so your QA surface stays aligned with product reality across all your developers, not only the person who had the chat.

Core loop:

1. Work normally in your coding agent
2. Studio’s watch daemon indexes sessions for mapped projects
3. On cadence, high-signal decisions are filtered and batched
4. You get a plan → approve → stories/scenarios update → raise a PR
5. Downstream QA workflows course-correct from the updated surface

Full product walkthrough: AgentWatch docs. Setup: How to enable.

Why this became feasible now: Jev as the filter​

Agent conversations are verbose. Most chunks are tooling noise, not durable product guidance—requirements, expectations, acceptance-worthy constraints.

That is an awkward job for a full LLM on every chunk, for every developer, all day. It is a natural job for Jev—TypeSafe’s System One model built to answer typed decision questions with calibrated scores, not to write essays. Talk about timing: AgentWatch needs high-volume, low-latency “is this product-relevant?” judgments. Jev fits.

One practical limit of Jev is the tighter context window (~32K for state + the longest question). Stuffing all stories and scenarios into every classify call is infeasible. This is where TestChimp’s shared-space semantic embedding canvas earns its keep:

1. Query the closest story / scenario neighbours to the conversation chunk (Semantic Canvas is how humans explore that same space)
2. Feed those neighbours—plus chat summary and chunk—to Jev
3. Jev scores likelihood of impact per related story/scenario, and whether a new story should be authored
4. Only the filtered hits go to an LLM to suggest updates and draft new stories
5. You get a PR-shaped plan; you confirm; QA workflows follow

Jev is a means. The product is AgentWatch: tribal knowledge → governed QA surface.

Where AgentWatch sits among agent-history tools​

A wave of local tools now index coding-agent sessions so you can search or recall past threads—useful when you ask “what did we decide about auth last month?” Examples people already search for:

• AgentsView — multi-agent session browser / index (AgentWatch uses it as the watch layer)
• aise — ultra-fast local session search + MCP
• Callimachus — hybrid keyword + semantic recall over agent history
• Prism — persistent session memory and knowledge graph for agents

Those tools help you find the thread. AgentWatch’s bet is different: when a decision should change what QA monitors, it should land in stories and scenarios—versioned, reviewable, and wired into the same workflows and QA Brain you already run—not stay buried in a searchable chat archive.

The DevX we optimized for​

No new ritual. No “paste this transcript into TestChimp.”

Configure cadence, which agents to watch, and whether add/update is allowed for stories and scenarios—details in the how-to.

Why this belongs in TestChimp​

We already argued that skills are SaaS distribution, that Studio collapses the stitching tax on the desktop, and that Meeting Bots put product intelligence in the call.

Agent chats were the remaining tribal channel: rich decisions, thin structured memory, specs that lag the product.

AgentWatch closes that channel—so the hours your team already spends directing agents become fuel for a living QA surface, not another forgotten scrollback.

Get started​

1. Install TestChimp Studio and map a local folder to your project.
2. Open Settings → AgentWatch → enable.
3. Import or bootstrap a QA baseline if you do not have one yet.
4. Keep working in your agents. Review the first plan when it lands. Raise the PR.

Docs: AgentWatch · How to configure · Semantic Canvas · Meeting Bots.

Frequently asked questions​

Which agents are supported?
Cursor, Claude Code, Codex, and 20+ others via the pinned AgentsView provider set. Pick which ones to watch in Settings.

Does every chat create a plan?
No. Jev filters for decided, verifiable product guidance. Implementation chatter is ignored by design.

Do I need a baseline first?
Yes for day-to-day sync—roughly ≥ 10 stories or ≥ 20 scenarios—or bootstrap / import first. See baseline.

Is this the same as searching my agent history?
No. Search tools help you recall threads. AgentWatch proposes governed updates to the QA surface your team shares.

The note-taker gap​

Meeting tools got good at recording. They did not get good at knowing your product—or at keeping what the team said from evaporating after the call.

If you already run TestChimp, the expensive part is done: plans as code, coverage, issues, releases, knowledge bases, observability, API schemas, QA Brain. The meeting was the last place that context stayed outside the room—and the last place team knowledge got said once and lost.

Meeting Bots close that gap.

What Meeting Bots are​

A Meeting Bot is not a second product. It is a way for TestChimp Agent to sit in the call with the same control plane your agents and humans already use.

Core loop:

1. Capture tribal knowledge — detailed notes organized and retained in cloud; optional PII redaction + summarize; knowledge can be fed into later workflows
2. Ask in-call — @testchimp or @QABot (or just ask for help) answers with product / SDLC context
3. Assign — capture tasks and issues from the discussion, not from post-hoc archaeology
4. Follow up — after the call, turn the conversation into workflows (author the stories and scenarios you just agreed, scope tests, file issues)

Concepts (full walkthrough in docs):

Capture tribal knowledge​

Alignment that lives only in the room dies when the call ends. Meeting Bots treat the conversation as first-class product memory: transcripts are organized, retained, and available for redact / summarize and for agent handoff—so what your team shares can be captured, codified, and reused, not trapped in chat scrollback or someone’s private notes.

That is the foundation for everything else: live answers, task assignment, and post-meeting workflows all consume the same durable meeting context.

One join, many personalities​

Here is the part people get wrong about “AI in meetings”: they assume every personality means another recorder and another bill.

In TestChimp, one visible agent joins and one transcript is produced for the meeting. You can still attach multiple custom bots—FDE assist, PM critique, pre-sales, support—and @mention the right lens. Transcription is per meeting, not per bot. Adding three personalities does not triple the join meter.

Policy files live where the rest of your agent governance lives:

plans/knowledge/policies/meetbots/<BotName>.policy.md

Version them. Review them. Treat brainstorming agents like any other playbook—not a one-off system prompt someone typed into a vendor dashboard.

Billing stays boring on purpose: $1 / hour of join time, plus ChimpHands credits when you actually invoke the LLM (@testchimp / @BotName, or transcript PII / summarize processing). Passive transcription does not burn credits every spoken second. Details: Billing.

Beyond QA​

The use cases go well beyond QA.

TestChimp already has deep product-context awareness across user stories, scenarios, tests, executions, releases, KBs, issues, observability metrics, API schemas, and more. Meeting Bots put an assistant with that context into the conversation—ready when needed.

Imagine that assistant in:

• Pre-sales calls — product truth without improvising capabilities you do not have
• Customer support conversations — grounded in real issues, releases, and scenarios
• FDE interactions — implementation help against plans and APIs, not vibes

Define custom bots with bespoke personalities and objectives, then bring the right assistant(s) into each meeting. Docs: Custom bots.

Post-meeting is where the loop compounds​

Live answers are useful. Governed follow-through is the unlock—especially for QA workflows built from what the team agreed.

From a meeting detail page you hand the transcript to ChimpHands with a draft like:

/testchimp referring the meeting <id> as context, do the following :
author user stories for the behaviour we agreed, with scenarios

Same catalog workflows you already run for author-plans, fix-issue, upkeep—except the meeting is first-class context instead of a forgotten Google Doc.

That is the difference between “we recorded the alignment” and “the alignment became stories, scenarios, and tests in the repo.”

Why this belongs in TestChimp​

We have argued that skills are SaaS distribution, that automations close the routing gap, and that Studio collapses the stitching tax on the desktop.

Meetings were still a parallel universe: rich talk, thin product memory, tribal knowledge that never got written down.

Meeting Bots put product intelligence in the call—so the hour you already spend aligning becomes fuel for delivery, support, and sales loops, not another artifact to reconcile later.

Your meetings—now part of your product intelligence.

Get started​

1. Open Meeting Bots in the app (capability on eligible plans).
2. Connect a calendar.
3. Join with QABot (or create a custom personality for sales, support, or FDE).
4. @testchimp a product question mid-call.
5. Afterward, open the meeting → ChimpHands → turn the agreement into work.

Full product docs: Meeting Bots · Custom bots · Meetings & PII · Billing.

Frequently asked questions​

Do I need a calendar?
Strongly recommended. Calendar-linked joins improve upcoming-event pickers and team @mention guards when participant emails are available. You can still paste a Meet / Zoom / Teams / Webex URL for transcription-only joins.

Who can @mention the bot?
Org members only (email preferred). Externals get a denial—your product brain stays inside the team.

Does every spoken minute cost ChimpHands credits?
No. Join time is the $1/hour meter. Credits apply when you @mention for an LLM answer (or when LLM-backed redact / summarize runs).

Will adding more bots double my bill?
No for transcription / join time. Multiple personalities share one meeting session and one transcript.

The important part is not another dashboard. TestChimp’s QA workflows can now use the same telemetry to decide what to test next:

• Uncovered, high-volume operations move up the functional coverage queue.
• Uncovered operations with high error exposure move up faster.
• High-p95 or high-p99 operations become candidates for performance-test creation and upkeep.

Production telemetry becomes a prioritization signal for QA—not a substitute for tests, and not an excuse to copy production traffic into a load profile.

Product guide: Observability in API Contract Governance.

Most QA backlogs know what is uncovered. They do not know what matters.​

OpenAPI coverage can tell you that 80 operations have weak coverage. Execution history can tell you that a test failed yesterday. A requirements graph can tell you which scenarios the team marked high priority.

All useful. Still incomplete.

If one uncovered endpoint handles 40% of production requests and another is called twice a month, they are not the same risk. If a mapped operation is returning errors on meaningful traffic, “we should test this someday” is no longer an honest prioritization. If p95 latency is already elevated, a green functional check does not answer whether the path stays healthy under concurrency or real tenant volume.

The usual QA process flattens these into one queue:

Telemetry-driven QA closes that context gap. It uses production-like runtime evidence to optimize the order in which the QA process spends human attention, agent tokens, test runtime, and performance-environment budget.

What TestChimp now ingests​

TestChimp performs read-only aggregate queries against the configured observability provider. It does not copy raw traces, logs, request bodies, credentials, or customer payloads into the QA context.

The flow is:

1. Discover observed routes from the previous 24 hours.
2. Map method + route templates onto operations in the selected OpenAPI service.
3. Surface unmatched runtime routes as Unmapped contract drift.
4. Collect aggregate metrics for each completed hour.
5. Store a complete prior-UTC-day summary with request count, average RPM, error count and rate, HTTP status classes, and p50 / p95 / p99 latency.
6. Show the latest finalized daily summary in the APIs list, with the latest persisted hourly result as a fallback while daily data is not yet available.

That gives each mapped API operation two distinct evidence sets:

• Contract coverage — which operations, fields, and response codes SmartTests or API tests actually exercised.
• Runtime observability — how much traffic reached the operation, how often it failed, and how its latency was distributed.

Observability does not make an endpoint covered. Coverage does not prove the endpoint is important. Put them together and the QA queue gets much smarter.

Setup guides: Observability integrations · Datadog · Honeycomb · CloudWatch · Google Cloud Monitoring.

How should telemetry prioritize functional API coverage?​

Start with the coverage gap. Then use telemetry to order the queue.

1. Missing coverage is the eligibility signal​

Find operations with no covering tests or weak request-field, response-field, and response-code coverage. Intentionally ignored gaps stay out of the active queue.

2. Request volume is the impact signal​

Among uncovered operations, prioritize high request count or RPM. A regression on a hot operation has a larger blast radius than the same bug on a dormant path.

This does not mean “only test popular endpoints.” Authentication, billing, destructive actions, and compliance paths can remain critical at low volume. Telemetry improves ordering; it does not replace product judgment.

3. Error exposure is the urgency signal​

Raise operations with high error rate, high absolute error count, or meaningful 5xx traffic. Then inspect the contract detail:

• Is the failing response code represented in the OpenAPI spec?
• Does any SmartTest exercise it?
• Are error-body fields and recovery behavior asserted?
• Does the operation have an auth, role, validation, or state-transition branch that the happy path never reaches?

The goal is not to write one API test per metric. Prefer extending an existing UI or API test that already reaches the operation and can exercise the missing branch naturally.

4. Business criticality breaks ties​

Volume and errors are evidence, not governance. Use scenario priority, release scope, semantic coverage, incident history, and backend branch complexity alongside telemetry.

A practical ordering is:

missing coverage
  + request volume / error exposure
  + business criticality
  + distinct branch complexity

That is how /testchimp create tests, /testchimp run QA, and /testchimp upkeep now reason about API coverage when observability is available.

How should latency telemetry prioritize performance testing?​

Functional testing asks whether the operation behaves correctly. Performance testing asks whether the journey remains healthy under an approved workload and dataset.

Use runtime p95 and p99 latency to identify:

• A slow operation with no corresponding k6 journey.
• A hot operation whose performance coverage exists but has gone stale.
• A high-error operation where latency, queueing, timeouts, or dependency behavior may be contributing.
• A list, search, report, export, or history path that may need a volume test against a large seeded dataset—not merely more concurrent users.

The workflow mapping is deliberate:

• /testchimp create-perf-tests authors a scenario-linked journey when a high-value operation has no useful performance representation.
• /testchimp upkeep-perf reviews stale journeys, baselines, datasets, dependency mocks, and composite membership—with slow, high-volume, and high-error operations first.
• /testchimp run-perf-tests can use the signal to choose among related journeys for a PR or release scope.

This is performance-test prioritization, not automatic capacity planning.

Production request rates do not become k6 VUs. Production p95 does not silently become a threshold. Absolute concurrency, duration, dataset cardinality, SLOs, and pass/fail limits still come from your capacity model and approved policy.

And production p95 is not directly comparable to test p95 unless environment, workload, time window, dataset, code/config, and dependency behavior genuinely match. Otherwise the comparison is directional context—not a regression claim.

Full workflow guide: Performance Testing Workflows.

Agents can consume the same evidence​

The APIs UI is not the end of the data path.

TestChimp’s API-operation list and detail APIs return observability mapping state plus the latest persisted runtime summary. The TestChimp CLI and MCP tools expose those same responses to agents:

list-api-operations
get-api-operation-detail

The summary includes its time window and sync status alongside request volume, errors, status classes, and latency percentiles. That matters because agents need to reason about data quality, not merely sort numbers.

Our skill playbooks now instruct QA workflows to:

1. Fetch API contract coverage and runtime observations together.
2. Check the observation window and sync status.
3. Treat absent, stale, partial, no-data, or query-failed telemetry as unknown, not zero.
4. Prioritize uncovered high-volume and high-error operations for functional coverage.
5. Prioritize high-p95 / high-p99 operations for performance creation or upkeep.
6. Keep production observations separate from test thresholds and baseline comparisons.

This is the useful version of “AI-powered testing”: not a model generating more scripts, but an agent allocating QA effort from evidence.

The optimization target is the QA process​

Observability platforms already help teams operate production. We are using the same aggregate evidence for a different question:

Given limited QA time and runtime, which uncovered behavior should we protect next?

That creates a tighter feedback loop:

OpenAPI contract
      +
test coverage
      +
runtime volume, errors, and latency
      ↓
prioritized QA workflow
      ↓
tests, fixtures, performance journeys, or accountable issues
      ↓
new execution evidence

The point is not to maximize test count. It is to optimize the QA process around expected impact.

This sits alongside TrueCoverage:

• TrueCoverage uses semantic product events to show which user journeys, transitions, and world-state slices deserve protection.
• API observability uses route-level volume, errors, and latency to show which contract operations deserve functional and performance attention.

Requirements say what should matter. Tests say what was proven. Telemetry says where the product is carrying real load and risk. Agentic QA gets better when it can reason across all three.

Frequently asked questions​

What is telemetry-driven QA?​

Telemetry-driven QA uses runtime signals such as request volume, error rate, and latency percentiles to prioritize test creation, maintenance, and execution. It optimizes which QA work happens first; it does not replace requirements, tests, or human risk policy.

Does TestChimp ingest raw traces or request payloads?​

No. TestChimp runs read-only aggregate queries and stores route mappings plus hourly and daily metric summaries. Raw provider events, logs, traces, and request bodies are not copied into TestChimp.

How does observability improve API test coverage?​

Among uncovered API operations, TestChimp workflows prioritize fresh high-volume and high-error operations, then inspect field and response-code gaps to extend an existing test or author focused coverage.

How does observability improve performance testing?​

p95 and p99 latency identify operations that need a k6 journey or performance upkeep. Request volume and error exposure strengthen that priority. Telemetry selects what to test; your policy still defines VUs, duration, datasets, and thresholds.

What happens when an operation has no observability data?​

It is unknown, not zero traffic. The workflow falls back to business criticality, scenario priority, code/release impact, semantic coverage, and execution history.

Can production latency be used as a k6 threshold?​

Not automatically. Production and test measurements usually differ in environment, workload, data, dependencies, and time window. Set k6 thresholds from explicit SLOs or an approved capacity policy, and compare runs only across compatible dimensions.

Try it​

1. Configure API Contract Governance with your OpenAPI root.
2. Connect an observability provider.
3. Map its service/resource to the matching OpenAPI service and run Sync now.
4. Open APIs and sort/filter coverage alongside RPM, error rate, and p95 latency.
5. Use Create test, /testchimp create tests, or /testchimp upkeep for high-impact coverage gaps.
6. Use /testchimp create-perf-tests or /testchimp upkeep-perf for slow, hot, or error-prone operations that lack current performance evidence.

Start here:

• Observability in API Contract Governance
• API Contract Governance
• Operations list
• Create Tests
• Create Performance Tests
• Upkeep Performance Tests
• TestChimp skill

Do not let the test backlog decide its own order. Let evidence tell the QA process where risk is concentrated—then make the work accountable.

Product page: testchimp.io/studio. Docs: TestChimp Studio · Download.

The stitching tax​

Agentic QA is working. Teams already run /testchimp workflows, keep plans in Git, gate releases on evidence, and let automations start ChimpHands when something changes.

The remaining bottleneck is the workspace. Delivery still looks like this:

Agents compressed doing the work. They did not invent a control pane that sits on your machine next to the repo.

Studio is that pane—in the convenience of a desktop app.

What is TestChimp Studio?​

TestChimp Studio is TestChimp’s companion desktop application. It is not a second product with a second source of truth. It is the same control plane as the web app, plus a local OpenCode agent runtime, so planning and execution share one window.

You can:

• Plan requirements — organize stories and scenarios, then hand them to agents without leaving the workspace (requirement planning).
• Run agentic QA — create tests, upkeep the suite, explore UX, implement a story—catalog workflows with full local checkout context.
• Govern the release — test runs, checks, and readiness on the version you are about to push (release governance).
• Track issues — agent-found and human-found, tied back to plans and executions.
• See executions — E2E, API, performance, security, and UX evidence without a dashboard scavenger hunt.
• Govern coverage — requirements, API schemas, real-user behaviour (TrueCoverage), and the gaps you should close next.
• Explore the QA knowledge-base — stories, scenarios, tests, releases, issues, and how they relate (QA Brain / InfiniTrace).

The desktop binary is the shell + agent. The product UI still loads from your App URL and ships independently—so Studio does not freeze the product on whatever build you installed last month.

Full walkthrough: TestChimp Studio.

One workspace, the whole delivery loop​

If you have been living in the TestChimp loop, the pieces already exist as separate surfaces. Studio is the place they stop competing for focus.

Requirement (story / scenario)
        ↓
Agentic workflow  (/testchimp create-tests, run QA, upkeep, …)
        ↓
Executions  (SmartTests, API, k6, ExploreChimp, security)
        ↓
Coverage + issues  (plans, schemas, RUM, QA Brain)
        ↓
Release  (evidence, checks, ship / don’t ship)

QA can run autonomously—local agent now, cloud ChimpHands and automations when you want CI to do it overnight—while you keep complete transparency into posture and how it aligns with product intent.

That is the difference between “we have an agent” and “we have a delivery workspace.”

Your agent, your keys, your machine​

Studio is built on OpenCode. Settings → LLM is explicit about who pays for tokens:

Studio starts one OpenCode serve for the active workspace and keeps the TestChimp skill current. You are not pasting mcp.json into a third IDE to get the catalog. You also are not required to burn a Cursor or Claude seat for every /testchimp upkeep you run at the desk.

Cloud ChimpHands stays the right executor for async CI. Studio is the right executor for interactive delivery on the laptop that already has the branch.

Why this matters in the agentic era​

We have argued that skills are SaaS distribution—the playbook travels with the agent. That automations close the routing gap. That requirements need governance before agents spend tokens. That release governance is the ship decision, not a spreadsheet.

Studio closes the attention gap.

Without it, the platform is still a tab, the agent is still another tab, and the repo is still a third window. With it, the control pane and the agent share a desktop—requirement planning all the way to a confident release push.

That is how you boil the QA lake without boiling your context-switching budget.

Frequently asked questions​

What is TestChimp Studio?​

TestChimp Studio is TestChimp’s companion desktop app—an IDE for software delivery. It consolidates requirement planning, agentic QA workflows, release governance, issues, test executions, coverage insights, and QA knowledge-base exploration in one workspace on your machine.

How do I download TestChimp Studio?​

Use the official Download TestChimp Studio page. Certified installers are on GitHub Releases for macOS, Windows, and Linux. Cross-platform: npx @testchimp/studio.

Does Studio replace the TestChimp web app?​

No. Studio loads the same product UI from your configured App URL. The web app remains the shared control plane and the URL you send a teammate. Studio adds the local shell and agent runtime.

How is Studio different from Cursor or Claude Code?​

Those IDEs are excellent for implementation. Studio is a delivery workspace: plans, releases, executions, coverage, and catalog /testchimp workflows sit next to the local agent. You can still use Cursor/Claude for feature work; Studio is where QA posture and product intent stay visible.

Can I use my own LLM or a local model?​

Yes. Settings → LLM supports ChimpHands, BYO API keys (OpenAI, Anthropic, ChatGPT OAuth, models.dev providers), and a custom base URL for local or self-hosted models. BYO and custom paths do not consume ChimpHands credits.

Is TestChimp Studio available on macOS, Windows, and Linux?​

Yes. Certified installers ship for those platforms, and npx @testchimp/studio runs on all three. See Download.

Do I need to re-map my project if I switch between npx and the installer?​

No. Certified builds and npm paths share ~/.testchimp and the same Electron userData.

Try it​

1. Open Download TestChimp Studio
2. Install via your platform’s certified build—or run npx @testchimp/studio
3. Sign in and map a local project folder
4. Open a story → create tests, or open Releases and look at the version you are about to ship
5. Set Settings → LLM to ChimpHands, BYO, or local

Start here:

• TestChimp Studio overview
• Download & install
• Studio updates
• Workflows
• ChimpHands
• Release management
• testchimp.io/studio

Your control pane for software delivery—now in the convenience of a desktop app.

Further reading​

TestChimp

• TestChimp Studio
• Download TestChimp Studio
• What is TestChimp?
• Workflows
• ChimpHands
• Requirement planning
• Release management
• TrueCoverage
• InfiniTrace

Related posts

• Meet ChimpHands
• Put QA on Auto-Pilot with Workflow Automations
• Release governance
• Test planning as code
• SKILLs as SaaS distribution
• Boiling the QA lake

Concepts

• OpenCode
• TestChimp skill
• Agent Skills

Full docs: ChimpHands.

The wrong agent for the job (and the wrong bill)​

Agentic QA is working. Teams run /testchimp fix-issue, /testchimp upkeep, /testchimp author-plans—and Automations route those workflows when issues, batches, and releases change.

But the executor still matters:

We built ChimpHands to be the native answer: QA workloads on your CI, orchestrated by TestChimp, metered for QA, traceable at the workflow level.

What is ChimpHands?​

ChimpHands is TestChimp's own cloud agent for QA—not a generic coding-agent integration you bolt on later.

• Runs on your repo's GitHub Actions runner — not on TestChimp VMs. The agent checks out your branch, uses your tree, and opens PRs from CI.
• Full code context — same environment your SmartTests and policies already live in.
• Same workflow catalog — fix-issue, implement, upkeep, author-plans, instrument-truecoverage, run-qa, and the rest. One skill, one MCP, one workflow-execution-id timeline.
• OpenCode + TestChimp skill — each job installs OpenCode and @testchimp/cli, clones testchimp-skills, and runs testchimp chimphands serve.
• Cost-optimized models — LLM traffic routes through GPT 5.6-range models suited to playbook-driven tool use, not always the priciest frontier tier.

Think: the QA agent your platform already knows how to schedule, audit, and budget.

Platform context and IDE-agent ergonomics—without the wiring tax​

BYO cloud agents mean committing mcp.json, configuring MCP in Cursor/Copilot dashboards, and pasting issue ids into prompts. ChimpHands ships differently:

Packaged on CI: every job installs OpenCode, @testchimp/cli@latest, and clones testchimp-skills. You set TESTCHIMP_API_KEY once as a GitHub secret—not per-agent MCP setup.

Platform entities in chat: the composer + menu attaches removable tags for stories, scenarios, issues, tests, releases, and automation batches—searchable pickers, stable ref: lines in the outbound prompt (ordinal ids where applicable). Handoffs from Plans and Issues can pre-fill that context.

Repo + file parity with typical agents:

So you get the "attach the failing test batch and lines 42–88 of checkout.spec.ts" workflow—native to TestChimp, not duct-taped through a generic coding-agent product.

Details: Chat with platform and repo context.

How it runs (one-click, then forget)​

Setup is deliberately boring—in a good way:

1. Connect the TestChimp GitHub App and map your repository.
2. Open ChimpHands in the sidebar → install .github/workflows/chimphands.yml (direct commit or PR if branch rules require it).
3. Add TESTCHIMP_API_KEY as a GitHub Actions secret.

After that, ChimpHands sessions start from chat, Plans (Spec out / Scope out / Implement / Test), Issues, or Automations when you pick ChimpHands as the invocation strategy.

Trigger (automation / chat / Plans handoff)
        ↓
ChimpHands session + workflow execution
        ↓
GitHub Actions: checkout → OpenCode → TestChimp skill → /testchimp …
        ↓
Tunnel ↔ TestChimp UI (stream tools, diffs, messages)
        ↓
Pull request + Done / Failed in Workflow Executions

You can always drop to the GitHub Actions run for raw logs. The product UI is for humans; CI is the source of truth for the run.

Mechanics: How ChimpHands runs.

Async → in-convo without losing the thread​

Most cloud agents force a tradeoff:

• Async: great for overnight upkeep, terrible when the agent hits an ambiguous requirement at 2am.
• Interactive: great for pairing, terrible for "fix every failed batch without me watching."

ChimpHands bridges both on one session id:

External agents (labelled issues, separate cloud UIs) usually break continuity here—you re-paste context or start over. ChimpHands keeps one session, one workflow execution, one Actions run.

That's the seamless human-in-the-loop story we wanted for policy-traceable workflows: attend when it matters, ignore when it doesn't—without losing the audit trail.

Why separate QA agent economics​

ChimpHands metered usage enables you to have your QA workloads isolated from dev-agent subscriptions.

Your Claude Code and Cursor seats stay for implementation. ChimpHands credits cover the long tail:

• Fix failing SmartTests after CI
• Windowed upkeep digests
• author-plans with full repo context
• TrueCoverage instrumentation and gap fixes
• Release run-qa composites

You set a QA budget per billing cycle. The platform enforces gates before metered runs and shows remaining credits in-product. Within that budget, ChimpHands aims for the best feasible protection—high-signal automations first, predictable spend, fail-closed when credits run out.

Value prop deep dive: Why ChimpHands.

Automations, now native​

When we shipped Workflow Automations, the missing piece was a first-class executor that didn't require you to operate another agent product.

ChimpHands is now the recommended invocation strategy:

Configure once under Workflows → Automations, choose ChimpHands, tune aggregation and gates. Same workflow-execution-id reporting as local /testchimp.

Recipes: Typical automation setups · Best practices.

ChimpHands vs bring-your-own cloud agents​

Bring-your-own paths remain: Cloud agents · Setting up cloud agents.

Why this matters in the agentic era​

We've argued that skills are SaaS distribution—the playbook travels with the agent. That automations close the routing gap—events in, workflow executions out. That requirements need governance before agents spend tokens.

ChimpHands closes the executor gap: a cloud agent that is of TestChimp, on your CI, for QA economics.

Local Cursor / Claude for the interactive dev loop. ChimpHands for the overnight fix, the failed batch, the ready story, the release gate—and for jumping into the same session when the agent has a question.

That's how you boil the QA lake without standing next to every kettle or duct-taping five agent products together.

Frequently asked questions​

What is ChimpHands?​

ChimpHands is TestChimp's native QA cloud agent. It runs catalog /testchimp workflows on your GitHub Actions runner with full repository context, OpenCode, and the TestChimp skill—reachable from automations, Plans, Issues, and in-product chat.

Where does ChimpHands run?​

On your GitHub Actions infrastructure. TestChimp dispatches .github/workflows/chimphands.yml; the job checks out your branch and executes the workflow there. TestChimp orchestrates sessions and streams UI updates over a tunnel—it does not host your source code.

How is ChimpHands different from Cursor or Claude for QA?​

ChimpHands isolates QA spend from dev-agent subscriptions, uses cost-optimized GPT 5.6-range models for playbook-driven work, and ties every run to workflow executions in TestChimp. Use Cursor/Claude for IDE pair-programming; use ChimpHands for async QA, automations, and repo-context plan authoring.

How does human-in-the-loop work?​

Automations can run fully async. If the agent needs clarification, open the same ChimpHands session in TestChimp chat—your reply reaches the CI runtime and the turn continues without a new conversation or copy-pasted context.

How do I set up ChimpHands?​

Connect the TestChimp GitHub App, map the repo, install the ChimpHands workflow from the ChimpHands page, and add TESTCHIMP_API_KEY as a GitHub Actions secret. Step-by-step: ChimpHands intro.

Does ChimpHands replace local /testchimp?​

No. Local agents with the TestChimp skill remain ideal for fast iteration and heavy IDE work. ChimpHands is the native cloud path—especially for automations and fire-and-forget QA runs.

Can I still use GitHub Issue or Webhook automations?​

Yes. ChimpHands is the recommended default. Via GitHub Issue and Webhook stay available for bring-your-own agent stacks.

Do I need to commit mcp.json for ChimpHands?​

No. The Actions job installs the TestChimp skill and @testchimp/cli on every run. You only need TESTCHIMP_API_KEY as a GitHub secret—unlike BYO cloud agents that require per-host MCP configuration.

Can I attach platform entities and file ranges in chat?​

Yes. Tag stories, scenarios, issues, tests, releases, and automation batches from the composer + menu; add repo line ranges or whole files from the diff pane or explorer; upload local files as artifacts. Edit files collaboratively in the side pane when the runtime allows.

Try it​

1. Open ChimpHands in your TestChimp project sidebar
2. Connect GitHub and Install the workflow (direct or via PR)
3. Add TESTCHIMP_API_KEY to GitHub Actions secrets
4. Start a chat session—or create an automation with ChimpHands as invocation strategy
5. Watch Executions → Workflow Executions (and join chat if the agent asks)

Start here:

• ChimpHands overview
• Why ChimpHands
• Best practices
• Automations
• Workflows catalog

When QA work needs doing, the agent should already be on your CI—not eating your dev subscription or waiting for you to wire another product.

Further reading​

TestChimp

• ChimpHands
• Why ChimpHands
• ChimpHands best practices
• Automations
• Cloud agents
• Fix issue
• Upkeep

Related posts

• Put QA on Auto-Pilot with Workflow Automations
• Policy-traceable workflows
• SKILLs as SaaS distribution
• Test planning as code
• Boiling the QA lake

Concepts

• OpenCode
• TestChimp skill
• Agent Skills

Full product guide: Verified Tests.

The missing piece is accountability​

Agents are very good at getting a path green.

The SmartTest runs. The scenario is linked. Requirement coverage looks healthier than last sprint.

That is not the same as: this test actually verifies the behaviour the scenario describes.

Has anyone actually looked at the test?

A scenario annotation is cheap to add—and agents add them at speed. They can link the right scenario to a script that asserts a toast, skips the important check, or walks a cousin flow that happens to share a button. The dashboard still counts it as coverage.

We used to paper over that with implicit knowledge: the person who wrote the test was the person who knew what it proved. Agents broke that assumption. Volume went up. Inspection did not.

Green is an execution fact. Linked is a mapping. Verified is the accountability layer: a person has sanity-checked that the covering test truly verifies the scenario.

That is the problem Verified Tests solves.

Why this is necessary in the agentic era​

Before agents, test authoring was slow enough that “who wrote this?” was a reasonable proxy for “who checked this?”

That proxy is dead.

Agents implement. Agents author SmartTests. Agents attach scenario ids because the plan files told them to. You still need a durable record that a person opened the execution, looked at what actually happened, read the script, and agreed: yes, this test verifies that scenario.

Otherwise requirement traceability is a spreadsheet of claims with nicer UI. Coverage insights tell you what ran. They do not tell you whether the test is the right test.

Introducing Verified Tests​

TestChimp now lets you Verify tests in the platform—without bouncing out to a repo or a recording elsewhere.

Open a SmartTest execution and you already have what you need to make an informed decision:

• Screen captures from the run
• Steps the test actually took
• The code, via View Test, so you can read the script next to the evidence

Then use the check badge next to the SmartTest name. Same visual language as a verify badge elsewhere on the internet, because the job is the same: this was inspected.

Three states on the test:

• Hollow — none of the linked scenarios are verified
• Grey — some, not all
• Blue — every linked scenario for that test is verified

For one scenario, you confirm. For several, you verify per scenario—or Mark all as verified. Hover a filled badge and you see who verified it.

Un-verify is deliberate: it records manually unverified, not “never happened.” The audit trail stays.

Status lives in TestChimp, not in source. Agents can keep authoring Playwright. Humans stamp the claim when they have actually looked.

A Verified Badge on scenario coverage​

Plans, requirement coverage, and test runs already show recent execution results per scenario.

You now also see a Verified Badge on each scenario—an extra layer of assurance that the product is being tested properly, not only that something ran green.

• Filled — a user has sanity-checked a covering test for that scenario
• Hollow — the scenario has tests, but nobody has inspected them yet
• Omitted — no linked tests, so there is nothing to verify

Clicking the coverage-side badge does not flip the bit. It tells you how: open an execution, inspect the run, then use the check badge next to the test name. Verification is an inspection action, not a bulk paint on a dashboard.

That is the point. If it were too easy, we would have rebuilt the original lie at a larger scale.

How this fits the rest of TestChimp​

If you’ve been following along:

• Test Planning as Code puts stories and scenarios in Git so agents have a real task
• Scenario annotations declare which SmartTest claims which scenario
• Requirement traceability rolls execution results up the plan
• Verified Tests records that a human checked the claim—not just that CI was green

Planned reality → linked automation → inspected coverage → release confidence. The middle of that chain was the hole once agents started writing both sides.

Frequently asked questions​

What is a verified test?​

A verified test in TestChimp is a SmartTest a project member has inspected and confirmed actually covers a linked scenario. It is more than a green run or a scenario annotation: someone looked at the screen captures, steps, and code, and stamped the claim.

How is verified different from linking a test to a scenario?​

Linking (a Playwright annotation with type: 'scenario' and #TS-<n>) is a coverage claim. Verification is a human sanity check of that claim. You can have linked, green, and still unverified.

Where do I verify a SmartTest?​

Open a SmartTest execution (click a coverage square). Inspect the screen captures, steps, and test code, then use the check badge after the test name. Coverage-train badges are read-only hints.

What does the Verified Badge on a scenario mean?​

A filled badge means at least one covering SmartTest has been manually verified for that scenario. Hollow means tests exist but have not been sanity-checked yet. No linked tests → no badge.

Does renaming a scenario reset verification?​

No, as long as the scenario ordinal (#TS-n) stays the same. Linking a different scenario is a new claim and starts unverified.

Can I un-verify?​

Yes. Confirming un-verify sets the link to manually unverified (distinct from never verified). Who/when hover applies to the verified state.

What if the test has no linked scenarios?​

It cannot be marked verified. There is no scenario claim to inspect. Link the scenario first.

When agents run your SDLC, trust—but verify.

Docs: Verified Tests · Link tests to scenarios · Requirement traceability.

A user can check out. The API returns 200. The SmartTest is green. You merge.

Then ten concurrent checkouts queue behind one chatty query. Or the reports page, which was snappy with three invoices in the seed DB, falls over on a tenant that actually uses the product.

That gap is not a mystery. Agents optimize for the path in front of them—one user, empty-ish data, mocked collaborators that return in 0 ms. Concurrency and data volume are different questions. Functional tests do not answer them.

So we shipped performance testing as a first-class TestChimp surface: k6 in your repo, agentic workflows that author the right journeys (and keep composites honest), related runs after a PR or a release, and an Executions view that compares this run to a prior one.

Full product guide: Performance Testing.

The questions that actually matter​

Not “did k6 print a chart.” The questions product and eng already have:

We keep load and volume as separate axes. “Make it heavier” by turning both knobs hides which one broke.

Agents author journeys. You still own capacity.​

/testchimp create-perf-tests does not invent a k6 file from vibes.

It ranks real scenarios (priority, semantic coverage, get-requirement-coverage --include-perf). It uses redacted REAL E2E interaction shapes—method, path template, schema, status class, timing distribution—not cookies, tokens, or production bodies. When TrueCoverage is mature, relative demand helps order the queue and suggest composite weights.

What it will not do: copy a production RPS into load.js. TestChimp telemetry tells you what is worth testing. You (or run-perf-tests.policy.md) still set VUs, duration, and dataset size. Smoke is the default while authoring. Load/volume wait on an explicit capacity decision.

Composites are a weighted mix of journeys—“typical overall load,” not isolated degrade detection. Adding a journey to a composite is always a prompted approval. Silent membership is how you accidentally change the mix and then argue about the chart.

Outbound deps (payments, email, LLMs, partner APIs) get harness mocks with realistic latency. Hitting Stripe in a load test is expensive and flaky. Stubbing it at 0 ms is worse: you never see pool exhaustion. Either outcome is false confidence.

Folder layout, metadata, and wrappers: How performance tests are organized.

Related runs after a PR. Release range when you cut.​

Perf load/volume does not execute inside /testchimp test. Functional smoke should stay fast. k6 execution is /testchimp run-perf-tests or CI k6/scripts/run.sh. Run QA does write plans/smart-smoke/<branch>/related-perf-tests.json when k6/journeys exists, so the next CI --impacted run picks up the affected journeys.

On a PR, we select related journeys from the change set—scenarios, operations, path templates—then run them through k6/scripts/run.sh --impacted. Bare k6 run is not a TestChimp run: no ingest, no Executions charts.

On a release, the prompt from the release page is:

/testchimp run performance tests for release 1.2.0

The git range is prior SHA → cut SHA, not whatever happens to be checked out. Ingest stamps TESTCHIMP_RELEASE so the release panel and Executions list light up. If existing journeys do not cover that range, the agent asks whether to author (nested create-perf-tests on the same range) instead of silently inventing a suite or pretending coverage exists.

/testchimp upkeep-perf is the long loop: stale scenario links, drifted contracts, composite membership, thresholds that should not be quietly weakened to make compare green.

/testchimp init-perf scaffolds k6/ once. /testchimp import-perf-tests brings Locust/JMeter/Gatling/Artillery/k6-from-elsewhere into that tree. Source VU counts stay out of load profiles until you approve capacity.

Workflow walkthrough: Performance testing workflows.

Results you can overlay, not a one-off HTML report​

Wrapper runs ingest a summary and attach downsampled timeseries (p95, fail rate, VUs, and the rest of the k6 dump). Executions → Performance Tests lists them—optionally grouped by batch. Open a run for threshold, p95, fail rate, duration, metric-over-time, and VUs.

Then click Add Comparison Run. Recommended candidates share the comparison keys (environment, profile, dataset, LLM mode, mock/latency profile). The overlay re-bases both series on elapsed time so you are not comparing Tuesday 4pm wall clock to Wednesday 9am.

That is the degrade question: same kind of run, previous comparable result, is this worse? Green against a loose threshold can still be a regression versus last Tuesday. CLI compare-perf-to-baseline is the same contract for agents and CI—it exits nonzero when comparison.regressed is true. A mismatched baseline is incomparable, not a pass.

Viewing results.

How this fits the rest of TestChimp​

If you’ve been following along:

• SmartTests (now grouped as Functional Testing in the docs) prove the path works for a user
• Smart Smoke keeps that suite runnable in a CI budget
• TrueCoverage says which journeys real people actually take
• Performance testing asks whether those journeys still hold when many people take them—or when the tenant already has history

Planned reality → functionally tested reality → load/volume-tested reality → production reality. The middle was the hole for teams whose agents ship faster than anyone can stare at an EXPLAIN plan.

Frequently asked questions​

What is TestChimp performance testing?​

Grafana k6 scripts in your mapped tests folder (k6/journeys and k6/composites), authored and selected by agentic workflows, ingested into Executions so you can chart a run and overlay a comparable prior run.

Do I need this if my SmartTests are green?​

Yes if you care about concurrency or tenant data volume. Agents (and humans) routinely ship designs that are correct for one user on a small seed database and fail under the conditions production actually creates.

Does /testchimp test run k6?​

No. Keep functional PR smoke fast. Use /testchimp run-perf-tests (or a dedicated CI job that calls the same wrappers).

Will TestChimp pick my VU count from production traffic?​

No. TrueCoverage and interaction timings are relative signals—what to test, how to weight a mix. Absolute VUs, RPS, duration, and dataset size come from policy or an explicit approval.

How do I see if we got slower?​

Open the run → Add Comparison Run → pick a recommended candidate. Or compare-perf-to-baseline in CI. Compare only matching environment / profile / dataset / mock profile.

Docs: Intro · Organization · Workflows · Results.

Concept: What is load testing? · Volume testing · k6 vs JMeter vs Locust · Compare across releases.

The trouble starts when the regression suite grows faster than your CI budget. That is the problem Smart Smoke solves.

Your agents have been writing tests for a few months, and now you have 500+ E2E tests taking an eternity to run. Sounds familiar?

Say hello to Smart Smoke.

With agents authoring tests en masse, the problem has changed. Coverage gaps close fast—but:

• Your CI bill keeps growing as you run the full suite on every PR
• Worse, you wait hours before knowing if anything broke

Today, smoke suites are often manually managed. Tag lists. Folder filters. A @smoke set someone curated last quarter. That leaves a lot of useful signals on the table—especially as test suites grow at an unprecedented pace.

What really interested me here—as a lover of algorithms—was that this is essentially a classic optimization problem:

Given N minutes, how do you maximize semantic coverage + PR impact?

First: find what the PR can break​

An agent identifies the tests relevant to the change—impacted scenarios from your plans, linked SmartTests via scenario annotations, plus anything newly authored on the branch. That related set lands in plans/smart-smoke/<branch>/related-tests.json.

That’s the seed. Then comes the fun part: covering as much ground as possible.

Paint the canvas within a fixed budget​

Imagine your tests laid out as nodes on a semantic canvas. The problem becomes:

Paint the canvas by selecting test nodes to maximize the covered spread—within a fixed time budget.

We don’t just grab the nearest cluster. We iteratively pick the next test that adds the most new ground—so selection spreads across the plane instead of camping in one corner.

Then weigh the terrain​

Pure geometry isn’t enough. We weigh the terrain using signals like scenario priority, recency, test stability, execution time, historical failures—so packing prefers tests that are both informative and practical.

Seeds always include related tests, tagged smoke (e.g. @smoke), and newly authored branch tests. Packing fills whatever budget remains—until the bin is full.

The result: the best subset of tests to run as smoke for a given PR—giving you the most confidence within a fixed time budget.

Less CI cost. Less waiting. More confidence.

Same Playwright command​

Smart Smoke isn’t a new runner. You keep your SmartTests and the same npx playwright test—opt in per run:

export TESTCHIMP_SMART_SMOKE_ENABLED=true
# optional: time budget, suite %, tags, related-tests-only
npx playwright test

Non-selected tests skip with reason smart-smoke (distinct from an explicit test.skip).

It plugs into /testchimp test as Phase 5, or standalone via /testchimp run smart smoke.

Full reference: Smart Smoke Runs · How it works · Configuration.

Plugin: @testchimp/playwright (≥ 0.2.20).

Related QA guides​

• Playwright alternatives compared
• Cypress vs Playwright
• Stripe test cards for Playwright E2E
• Requirements traceability matrix (RTM) template
• Test plan template in Markdown

Smoke responsibly.

The contract nobody audits​

Your OpenAPI (or Swagger) file is not documentation fluff. It is the business contract of the product: methods, paths, request shapes, response codes, and fields clients depend on.

UI E2E already touches that contract. Checkout hits POST /orders. Declined payment hits POST /payments. A rare enum or a 422 body? Often never.

Without a contract lens, teams hit the same failure modes:

API contract governance is not a separate API-testing product. It is the audit and triage layer that turns traffic your SmartTests already generate into accountable coverage of the contract.

Full product guide: API Contract Governance.

How it works​

Three inputs. One surface.

1. Discover the contract — Configure OpenAPI root path(s) from your connected repo (APIs → Configure). TestChimp parses them into services and operations.
2. Observe traffic — Run SmartTests with TESTCHIMP_ENABLE_API_CAPTURE=1 (and @testchimp/playwright ≥ 0.2.17). Matching HTTP interactions upload with the batch; coverage denorm runs after—not on the hot ingest path.
3. Govern gaps — List view for the portfolio; detail view for schema fields and response codes. From any hole: file an issue, copy an agent prompt to create/update tests, or ignore the gap when it is intentional.

Capture is off by default so ordinary runs stay light. Opt in when you want Operations coverage—same pattern as other optional runtime features. Setup detail: Set up API contract governance and the runtime plugin.

From endpoint scores to schema fields​

The APIs list is the portfolio view: service picker, coverage-band summary, method/path, spec-defined vs observed-only, coverage score, latency, covering-test chips.

Click a row and you get the interesting part—field-level truth:

• Request schema and query parameters with covering tests per field
• Response schemas nested by status code
• undocumented tags when traffic (or denorm) sees a field or code the OpenAPI root never declared
• Empty covering tests when the contract exists but automation never exercised it

That is the difference between “we hit /orders” and “we never sent delivery.type = same-day, and nobody owns that hole.”

List and detail walkthroughs: Operations list · Operation detail.

Governance that sticks: issue, test, or ignore​

Coverage without triage is another dashboard nobody opens.

Wherever coverage is missing—operation, field, or response code—TestChimp offers the same decision set:

Agents already understand operation-id / field / response-code scopes. The UI just stops the gap from living only in someone’s head.

How this fits the rest of TestChimp​

If you’ve followed us:

• Requirement quality governance hardens the plan contract before agents spend tokens
• Release governance hardens the ship decision for a version
• API contract governance hardens what your automation proved against the HTTP contract—the surface clients and integrations actually depend on

SmartTests remain the traffic source. Issues remain the backlog. Create-tests remains the authoring path. The APIs page is where contract debt becomes visible and governable—next to TrueCoverage for behaviour-aligned RUM, not instead of it.

Planned reality → tested reality → production reality. The API surface was the missing middle for teams whose “coverage” stopped at green UI journeys.

Frequently asked questions​

What is API contract governance in TestChimp?​

API contract governance maps OpenAPI operations to real and mocked SmartTest traffic so you can see endpoint and schema-field coverage, then triage gaps with create issue, create test, or ignore gap.

Do I need dedicated API tests?​

No. Coverage is inferred from HTTP traffic your SmartTests already generate when capture is enabled. You can still author dedicated API specs; the governance surface tracks both.

Why is my coverage empty after configuring OpenAPI?​

Parsing the spec discovers operations. Scores and covering tests need runs with TESTCHIMP_ENABLE_API_CAPTURE=1. Capture is off by default on purpose.

What does Ignore gap do?​

It marks an uncovered operation, field, or response code as intentionally ungoverned for now—without creating an issue or a test—so teams can acknowledge debt without false urgency.

Spec defined No vs undocumented—what’s the difference?​

Spec defined No on an operation means the whole method/path is missing from the OpenAPI root (observed in traffic only). An undocumented tag on a field or response code means that leaf was seen but not declared—classic contract drift.

Try it​

1. Connect git and open APIs → Configure — select your OpenAPI root file(s)
2. Wait for parse/sync; pick the service in the list
3. Run SmartTests with TESTCHIMP_ENABLE_API_CAPTURE=1
4. Refresh APIs — open a low-score or uncovered operation
5. Create issue, Create test, or Ignore gap on the holes that matter

Start here:

• API Contract Governance
• Setup
• List view
• Detail view
• Playwright runtime plugin
• TestChimp skill

Know the contract. Cover the edge cases—before unaudited branches ship.

Further reading​

TestChimp

• API Contract Governance
• Set up API contract governance
• SmartTests runtime plugin
• Create tests
• Issue management

Related posts

• Release Governance: Ship When the Evidence Says So
• Requirement quality governance (DeFOSPAM)
• TrueCoverage
• Boiling the QA lake

Explicit links aren’t enough​

Last week we shipped InfiniTrace—walk the structural edges of your QA graph board by board.

That answers:

“What is linked to this story / test / event?”

Most of the industry stops there. Jira, TestRail, CI folders, coverage tools: if the edge wasn’t created, the relationship doesn’t exist. Keyword search helps only when titles share tokens. Paraphrases, cross-type near-misses, and quiet duplicates fall through the cracks—especially when agents author plans and tests at speed.

The sibling question traditional tools never answer:

“What looks like this—even if nobody linked it yet?”

Or:

“Which entities belong to the same conceptual neighborhood?”

Those need meaning, not another list sorted by updated-at.

TestChimp already maps every QA entity into a shared embedding space—beyond explicit links, into semantic proximity. Semantic Canvas is the surface that lets you see and navigate that space, next to InfiniTrace in QA Brain.

Introducing Semantic Canvas​

Semantic Canvas is a spatial map of your project’s QA knowledge—laid out by conceptual similarity.

Open QA Brain → Semantic Canvas tab (/brain?tab=semantic-canvas).

1. Add the entity types you care about—Stories, Scenarios, Tests, Issues, Events
2. Nodes land on a UMAP canvas: high-dimensional embeddings projected so local neighborhoods reflect cosine similarity
3. Pan, zoom, filter, search (#US-12, titles, …)
4. Click a node → the detail sidebar fills with cosine-ranked peers
5. Link, mark distinct, create a duplicate-cleanup issue, or open details—in place

Deep links encode types, filters, border modes, and selection in ?locator= (with tab=semantic-canvas). “Look at this cluster” is a URL, not a screenshot.

Docs: Semantic Canvas.

What you can actually do with it​

A few use cases that show why a meaning map beats link-chasing alone:

Plan neighborhoods with live automation health​

Load Stories + Scenarios + Tests, paint test borders with latest execution status, and scan the map: requirement clusters with failing automation sitting right next to the plans they should cover—without walking each story’s link tree.

Duplicate / lookalike issue clusters​

Load Issues alone with severity borders. Twin bugs filed weeks apart with different titles collapse into neighborhoods. Open the sidebar, mark distinct where needed, or file cleanup when it’s a true duplicate.

Close semantic coverage gaps​

Put Scenarios + Tests (or Events + Tests) on one canvas. Select an unlinked scenario or hot TrueCoverage event → Close by tests ranks peers by cosine similarity → Link the near-miss that should have been wired months ago.

Requirement overlap before you plan more​

Plot Stories + Scenarios. Dense blobs often mean one idea wearing three titles—or a story that needs a scenario split—before you (or an agent) author more surface area.

TrueCoverage hotspots vs automation islands​

Add Events with has test coverage borders. Uncovered events that sit next to well-covered automation clusters are high-ROI gaps: instrumentation, missing emits, or a test that almost covers the behaviour.

We already open-sourced the suite-side cousin—Semantic Graph for Playwright folders. Semantic Canvas brings the same instinct into the product graph: not just tests vs tests, but cross-type meaning across your QA entities.

Same underlying entities. Same link rules. Two complementary ways to see the graph.

The sidebar is where the work happens​

The canvas orients you. The detail sidebar decides.

Select a node and you get accordion lists ranked by cosine similarity—not by how close the dots look after UMAP squashed the space into 2D (those can disagree; trust the scores for linking).

Tune a cosine cutoff per source→target pair. Raise it when you’re hunting true duplicates; lower it when you’re fishing for coverage gaps.

Then act:

• Link — same model as InfiniTrace (including story↔scenario mappings and scenario annotation injection for scenario↔test)
• Mark as distinct — teach the system “similar ≠ duplicate”
• Create issue for duplicate — turn a lookalike pair into cleanup work
• View details — read-only modal, then jump to the full page when you need to edit

If a card that looks nearby on the map falls below your cutoff, that’s not a bug—that’s the difference between a 2D sketch and embedding space. The sidebar keeps you honest.

Borders that mean something​

Pretty clusters aren’t enough. You need health overlays.

Per type, paint node borders with operational signal:

• Latest execution status on tests / scenarios
• Priority / severity and due status on stories, scenarios, issues
• Has linked test on scenarios
• Has test coverage on TrueCoverage events

Drop Stories + Scenarios + Tests with execution status on the tests, and the map stops being art: requirement neighborhoods with live automation health.

Or load Issues alone with severity borders—and watch lookalike bugs collapse into clusters you can actually triage.

Combine that with folder filters and status chips for sharper triage: failing tests next to the stories they should cover, severity islands of duplicate bugs, uncovered hot events next to automation islands.

From links to neighborhoods​

Dashboards tell you what’s trending.

Atlas tells you where in the product structure something lives.

TrueCoverage tells you what real users do versus what tests cover.

InfiniTrace tells you how the pieces connect via explicit edges.

Semantic Canvas tells you what belongs together by meaning—even when the link was never created—and where quiet duplicates and near-misses still hide.

That’s the difference between knowing coverage is “82%” and seeing the blob of scenarios that are really one idea wearing three costumes.

Go open QA Brain → Semantic Canvas. Load Stories, Scenarios, and Tests with execution status—or Issues with severity—and click something you think is unique. If the sidebar shows a 90% twin you forgot existed… good. That’s the product working.

Full walkthrough: Semantic Canvas docs.
Pair it with: InfiniTrace.

The graph was always there​

Think about a single question a QA lead asks fifty times a week:

“What actually covers this story—and what didn’t we link yet?”

Or:

“This production event is hot in TrueCoverage. Which tests emit it?”

Or:

“This ExploreChimp journey found a bug. How does it connect back to plans and automation?”

Those aren’t dashboard questions. They’re traversal questions.

Traditionally, answering them meant hopping across Jira, a test runner, a coverage tool, an issue tracker—each holding a slice of the story, none holding the whole graph. TestChimp already consolidates those entities and their relationships in one platform: entity links, scenario annotations in SmartTests, SmartTest event emissions, exploration → journey structure, release and run context. The knowledge graph was real. What was missing was a way to walk it.

We’ve been cooking something for that.

Introducing InfiniTrace​

InfiniTrace is a horizontal link explorer for your project’s QA entities.

Open QA Brain in the sidebar → InfiniTrace tab.

You get boards—columns—like a kanban that never pretends to be a static board. Each board is an entity type. Board N shows what’s related to the card you selected on board N−1.

1. Choose where to start — Story, Scenario, Test, Issue, Event, Release, …
2. Select a card → it lifts; floating CTAs show related types you can fan out to
3. The next board fills with Linked entities (structural edges)
4. Open Non-Linked for top-N peers—often ranked by semantic proximity
5. Keep walking. Boards grow to the right. Change a mid-chain pick and the obsolete boards fall away

Deep links encode the whole chain in the URL (?locator=), so “look at this path” is a share, not a screenshot.

Docs: InfiniTrace.

Linked vs Non-Linked is the point​

Structural links are great when they exist.

The interesting work often sits in the gap: things that should be linked, or are nearly the same idea in embedding space, but aren’t wired yet.

That’s why every follow-on board has two panes:

From Non-Linked you can Link (when rules allow) or Create issue—without losing the chain you were walking.

Some edges stay derived (for example TEST ↔ EVENT from RUM / SmartTest emissions). InfiniTrace shows them; it doesn’t pretend every edge is a manual click.

Why this matters for agents too​

Humans aren’t the only consumers of this graph.

The same related-entity query model is what we want agents to use via CLI / MCP later: “give me type X related to entity Y, ordered by this strategy, filtered like this, top N.” InfiniTrace is the human-shaped surface of that API.

If you’ve been following our build-in-public arc—policy-traceable workflows, TrueCoverage, plans-as-code—the theme is the same: don’t hide the structure agents need in chat history. Put it in durable, queryable form.

InfiniTrace makes that structure visible and walkable for people first.

What you can actually do with it today​

A few chains that already earn their keep:

• Story → Scenario → Test → Execution — coverage and run health in one pass
• Event → Test — TrueCoverage hotspots meet automation identity
• Release / Test run → scenarios & executions — release triage without ten tabs
• Exploration → Journey → Issues / Tests — exploratory findings with lineage

Filter, search, sort per board. View details in a read-only modal. Jump to the full entity page when you need to edit.

From scavenging to tracing​

Dashboards tell you what’s trending.

Atlas tells you where in the product structure something lives.

TrueCoverage tells you what real users do versus what tests cover.

InfiniTrace tells you how the pieces connect—and where the graph is still incomplete.

That’s the difference between staring at metrics and actually following the thread.

Go open QA Brain → InfiniTrace, pick a story or a hot event, and walk it. If the Non-Linked pane surprises you with a near-miss that should have been linked months ago… good. That’s the product working.

Full walkthrough: InfiniTrace docs.

Loops win when you can close them​

Loops are all the rage in prompting today—and rightly so.

If you can trace the quality of an agent’s work back to the prompt, you get a feedback loop.

Better prompts → better outcomes.

We’ve been cooking something along those lines. But instead of treating every agent invocation as a raw blob of prompt text, we deconstructed agentic interactions a bit first.

Most agent work is three parts​

Look carefully at how agents are actually used in the SDLC. Most interactions are of the form:

<workflow> + <task description> + <behaviour guidance>

What’s interesting is that the first part has a surprisingly small vocabulary.

Author Story. Implement Task. Write Tests. Fix Issue. Fix Test Failure…

For QA, we found these distill into around 12 core workflows—plus a couple of composites for shorthand (run-qa, upkeep). The Workflows catalog is that vocabulary: modular playbooks the skill and the platform share.

From there, outcome quality mostly depends on two things:

Today, both still happen ad-hoc—with gut feel. Someone pastes a long prompt, tweaks a line that “felt” important last time, and hopes the next run is better. There is no durable artifact to improve. There is no evidence trail tying this outcome to that guidance.

We already hardened the task side​

We’ve been attacking the task-description lever for a while:

• DeFOSPAM requirement quality checks — score and fix ambiguous specs before agents spend tokens implementing or automating them
• Rich context for failures — fix-test-execution and related workflows pull execution detail instead of “tests are red, please fix”
• Test Planning as Code — stories and scenarios as markdown in Git, so the task itself is structured and agent-readable

That closes one half of the loop: better inputs into the agent.

The other half—how the agent should approach the work for your team—was still mostly vibes in a system prompt.

Introducing policy-traceable workflows​

Today we’re making that second lever first-class: policy-traceable workflows.

Each catalog workflow is backed by an optional policy—a Markdown file that defines how the agent should approach the task for your project. Policies are not secret sauce buried in a chat window. They live in your repository:

plans/knowledge/policies/*.policy.md

Version-controlled. Reviewable in PRs. Shared team-wide. Same files local /testchimp runs and Automations use.

---
workflow-id: implement
version: 1.2.0
---

Playbooks stay generic (the battle-tested skill steps). Policies hold the project choices: scoping rules, environment strategy, which composite subflows to run or skip, quality bars, exclusions, domain quirks.

Author or refresh them with /testchimp create policy <workflow-id>. Bump version whenever guidance changes—that version is what makes the loop measurable.

Full model: Workflows · Policies.

Trace every execution to policy · version · SHA​

A policy you can’t attribute is just another prompt.

For Plan → approve → Execute runs, the agent mints a stable workflow_execution_id, then reports mutative actions with:

• which policy file was used
• the policy version
• the git SHA it came from

…alongside workflow id, actor, branch, and entity identity. That lands on the Workflows execution timeline in TestChimp—not only in chat history that evaporates when the session ends.

The result is data you can use to improve policies. Data you can feed to an agent alongside the existing policy—to iterate on it.

Instead of tweaking prompts based on gut feel, you iterate on policies using evidence.

Ad-hoc prompting → modular, traceable workflows​

Put it together:

1. Small vocabulary of workflows — implement, create-tests, fix-issue, fix-test-execution, run-explorechimp, … plus composites
2. Context-rich tasks — governed requirements, execution detail, scoped branch diffs
3. Versioned behavioural guidance — *.policy.md in Git
4. Attribution on every run — policy file + version + git SHA on the execution timeline
5. Evidence → next policy version — close the loop

That is the same philosophy as skills as SaaS distribution (the playbook travels with the agent) and boiling the QA lake (agents in a continuous feedback loop)—applied to the behaviour contract itself.

From ad-hoc prompting → to traceable, structured, modular workflows.

Frequently asked questions​

What is a workflow policy?​

A policy is a project-owned Markdown file (plans/knowledge/policies/*.policy.md) that tells the agent how to run a catalog workflow for your team—scoping, env, quality bar, composite subflows, exclusions—without rewriting the skill playbook.

How is this different from a system prompt or CLAUDE.md?​

Those are useful, but usually opaque and hard to A/B. Policies are per-workflow, semver’d, synced to the platform, and recorded on every execution with file name, version, and git SHA—so you can compare runs and improve the guidance with evidence.

Do I need a policy for every workflow?​

Defaults are seeded for common composites (run-qa, upkeep) on /testchimp init. Some workflows (notably connect-to-test-env) need an explicit policy before dependent automation stays healthy. Atomic workflows can fall back to broader instructions—but named policies are what make optimization and auditability real.

How do I try an alternate policy without changing the default?​

Keep implement.policy.md as the team default and pass a variant:

/testchimp implement US-181 --policy implement-strict.policy.md

Same playbook, different behavioural contract. Compare outcomes on the execution timeline.

Where do I see policy traceability?​

Executions → Workflow Executions (and the Workflows UI timeline). After mutative actions, reported runs show the policy file and version used for that execution.

Try it​

1. Install the TestChimp skill and map plans/tests in Git
2. Run /testchimp init (or author policies with /testchimp create policy <id>)
3. Open plans/knowledge/policies/*.policy.md, encode one team convention, bump version
4. Run a workflow (/testchimp implement …, /testchimp run QA, …)
5. Check Workflow Executions for policy file · version · SHA—then improve the policy from what you see

Start here:

• Workflows overview
• Create policy
• Automations (same policies, event-driven)
• TestChimp skill

Better policies → better outcomes. Now you can prove which version got you there.

Further reading​

TestChimp

• Workflows
• Create policy
• Run QA
• Upkeep
• Implement Requirement
• Automations

Related posts

• Put QA on Auto-Pilot with Workflow Automations
• Requirement quality governance (DeFOSPAM)
• SKILLs as SaaS distribution
• Boiling the QA lake
• Test planning as code

Concepts

• Agent Skills
• TestChimp skills repository

The prompt-pasting tax​

If you've been living in the agentic QA loop, you already know the playbooks:

/testchimp fix issue BUG-1042
/testchimp implement US-181
/testchimp fix test execution …
/testchimp run QA

That works. It also creates a new bottleneck: someone has to notice the event and paste the prompt.

Agents compressed execution. They did not invent routing. Automations are that routing layer—on the same workflows, policies, and workflow-execution-id traceability you already use.

Full product docs: Automations.

What is a TestChimp automation?​

An automation is three parts:

Trigger + Action + Config

Policies stay *.policy.md in Git. Automations optionally attach them—the same files local /testchimp runs use. Workflows still define how the agent works. Automations define when that work starts and how the cloud agent is reached.

Project event  →  matching automation  →  aggregation window
                                              ↓
                                    workflow execution queued
                                              ↓
                         (optional) human approve invoke / plan
                                              ↓
                    ChimpHands on CI  or  labelled GitHub Issue  or  webhook
                                              ↓
                         /testchimp <workflow> … --workflow-execution-id …

Ways to reach a cloud agent​

Configure once under Project Settings → Automations (and ChimpHands setup for the native path), then pick a strategy per automation.

ChimpHands (recommended)​

TestChimp dispatches your repo's chimphands.yml GitHub Actions workflow with the rendered /testchimp … prompt. The agent runs on your CI with full checkout context, streams progress to the ChimpHands UI, and opens a PR when done.

One-click setup: GitHub App + workflow file + TESTCHIMP_API_KEY secret. See ChimpHands and Meet ChimpHands.

Via GitHub Issue (bring-your-own agent)​

TestChimp opens an issue on your mapped repo. Body = the full /testchimp … prompt (plus the workflow execution id). Labels always include testchimp, plus any extras you configure.

Point Copilot coding agent, Codex, Cursor Cloud, or your own runner at those labels. The agent checks out the repo, runs the prompt with the TestChimp skill + MCP, and reports back—so Executions → Workflow Executions stays current.

Deep dive: Cloud agents for automations.

Aggregation and human gates (because autonomy without brakes is chaos)​

Matching events should not always mean "spawn an agent immediately."

Default rate limit: 5 cloud agent invocations per hour per project (shared). Over-cap runs stay Queued—not dropped.

Two independent human gates (off by default):

1. Approve before invoke — review the rendered task before the cloud agent starts
2. Approve before plan execute — agent plans first; you approve before execute

Pending work surfaces in Executions → Workflow Executions. You can also copy the prompt and run it locally—the workflow-execution-id keeps the timeline accurate either way.

Mechanics: How automations work.

Recipes teams actually want​

Task templates use {{issue.id}}, {{story.title}}, {{batch.failed_count}}, {{project.repo_url}}, and friends—with autocomplete in the wizard.

More starters: Typical automation setups.

Why this matters in the agentic era​

We already argued that skills are SaaS distribution—the playbook travels with the agent. We argued that requirements must be governed before agents spend tokens. We argued that release governance is the ship decision, not a spreadsheet.

Automations close another gap: intent without attendance.

Without them, QA-on-autopilot still needs a human dispatcher. With them, the same /testchimp catalog becomes a control plane: events in → policy-backed workflow executions out → auditable status in the product. Local Claude / Cursor for interactive loops. ChimpHands and other cloud agents for the overnight and the "while you were in standup" cases.

That is how you boil the lake without standing next to the kettle (boiling the QA lake).

Frequently asked questions​

What are TestChimp Automations?​

Automations are event-driven rules that run TestChimp catalog workflows via a cloud agent when something changes in your project—issues, stories, scenarios, test executions, CI batches, or releases—optionally aggregated and gated by human approval.

How do Automations differ from Workflows?​

Workflows define how an agent should work (skill playbook + optional policy). Automations define when that work starts and which cloud invocation path to use. Both share the same execution timeline and workflow-execution-id reporting.

ChimpHands vs GitHub Issue—which should I pick?​

Use ChimpHands for the default path—native sessions on your GitHub Actions runner, platform chat, and QA-metered credits. Use GitHub Issue when you must keep an external label-watching agent (Copilot, Codex, custom bots) and already have the TestChimp GitHub App.

Will Automations spam my agents on every CI flake?​

Not if you configure them well. Prefer Buffered aggregation for batches, exclude ExploreChimp reporters on auto-fix rules, use conditions (status = BATCH_INVOCATION_FAILED, severity = HIGH), and keep the project hourly invocation cap. High-risk workflows should start with human gates on.

Can I still run workflows locally?​

Yes. Automations don't replace /testchimp in your IDE. From a pending execution you can also copy the prompt—including the workflow execution id—and run it with a local agent so status still updates in TestChimp.

Where do I create and monitor Automations?​

Create under Workflows → Automations (or contextual Automations CTAs on Issues, Plans, batches, releases, and workflow pages). Configure strategies and rate limits in Project Settings → Automations. Monitor and approve under Executions → Workflow Executions.

Try it​

1. Connect GitHub and set up ChimpHands (or configure GitHub Issue / webhook under Project Settings → Automations)
2. Open Workflows → Automations → Create Automation
3. Pick a trigger (start from a popular condition template)
4. Choose a workflow, task template, policy, and invocation strategy
5. Leave Buffered aggregation on; add invoke or plan-execute approval for risky jobs
6. Trigger a matching event—then watch Workflow Executions

Start here:

• Automations overview
• ChimpHands
• Create an automation
• Cloud agents
• Typical setups
• Workflows
• TestChimp skill

When something changes, the agent should already be working—not waiting for you to paste a prompt.

Further reading​

TestChimp

• Automations
• ChimpHands
• How automations work
• Cloud agents
• GitHub integration
• Fix issue
• Implement Requirement

Related posts

• Meet ChimpHands
• SKILLs as SaaS distribution
• Requirement quality governance
• Release governance
• Boiling the QA lake
• Test planning as code

Concepts

• GitHub Copilot coding agent
• Agent Skills