Smart Tests: Speed of Scripts + Adaptability of Agents

Introduction

Writing and maintaining test scripts is where developer dreams go to die. 💀

On one side, we have pure scripts. They’re fast, deterministic, and cheap to run — but notoriously brittle. Change a button’s label, and suddenly half your suite is red. Legacy “self-healing” platforms exist, but they come with vendor lock-in and enterprise price tags (tens of thousands a year).

On the other side, we have the Agent-Only pitch:

“Just write tests in plain English! No maintenance needed.”

Sounds magical, but the reality of pure agentic testing is harsh:

  • 🐌 Slow – every step requires inference, dragging down execution time
  • 💸 Costly – token usage explodes with test suite scale
  • 🎲 Non-deterministic – agents hallucinate, or worse, mask real bugs
  • 🕳️ Black box – debugging and triage become a guessing game

Neither extreme works. What’s needed is a hybrid model — the speed and determinism of scripts, with the adaptability of agents only when needed.

TestChimp Smart Tests

Smart Tests are our answer at TestChimp. They’re designed to give you the best of both worlds.

  • ✍️ Plain Playwright scripts generated from natural language scenarios
  • 📝 Intent comments before each step, describing the human-readable goal
  • ⚡ Script-first execution – tests run like normal Playwright tests, your existing playwright config files leveraged for consistent execution
  • 🤖 Agent fallback – if a step breaks, the agent uses the intent comment + live browser state to repair on the fly
  • 🔄 Self-healing with transparency – in CI, repairs are raised as a Pull Request; in IDE, the script is updated in place

 

So you always end up with clean, maintainable Playwright code — but without brittle breakage.

Test Authoring

With TestChimp, you can write test scenarios in three ways:

  • Tell it – Describe the scenario in plain English (in the TestChimp webapp or directly from your IDE via our VSCode extension).
  • Show it – Record a test journey with the TestChimp Chrome extension, then click Extract Test Scenario. This will extract the user actions from the session and create the scenario.
  • Ask for it – Generate scenarios from your user stories. Create a story in TestChimp, click Suggest Test Scenarios, and AI will brainstorm various test scenarios for the user story.

No matter how you start, TestChimp converts the scenario into a Smart Test script, using AI.

Test Execution

In IDE

With the VSCode extension (works in Cursor and other VS-based IDEs), you can:

  • Run exactly – run the test as a normal Playwright script.
  • Run with AI – enable hybrid execution: scripts run first, but when steps break, the agent repairs and updates the script in place.

This makes your IDE the center of test authoring and repair.

In CI

Smart Tests also run seamlessly in CI via the TestChimp GitHub Action.

  • Scripts run normally.
  • On failures, the agent repairs and continues.
  • At the end of the workflow, the agent raises a PR with all proposed changes.

You can configure success criteria:

  • ORIGINAL_SUCCESS – repairs are attempted, but if the original script failed, the test is marked failed.
  • REPAIR_WITH_CONFIDENCE – repairs can count as passes if the agent’s confidence is above your chosen threshold (1–5).

Either way, you review the PR before changes land.

Anatomy of a Smart Test

Here’s what a generated Smart Test looks like:

 

Doc Comments: Marks the test as a Smart Test. TestChimp Runner only runs Smart Tests in your test folder. This allows for easily introducing Smart Tests to your existing playwright test suites with minimal impact.

Repair Advice: This is a running understanding of fixes made to the test over time, which enables agent to get more accurate with proper generalisations for the test and better fixes over time.

Hashtags: This enables for semantic grouping of tests – which in turn helps the agent to organise and comprehend the smart test suite better, for tasks such as test suite augmentation.

After the Doc Comments section is the actual test. The test is simply a plain playwright script with intent comments.

Intent Comments: Explains the intent of the code unit after it. This helps agent to chime in when necessary and execute that step agentically.

Code Units: Those are the actual code units that are run by default. As long as they dont break, they will be executed as is – enabling fast, deterministic execution of tests – and agentic step in only when necessary.

Why Smart Tests Matter

With Smart Tests, you don’t have to choose between brittle scripts and slow, black-box agents. You get:

  • ✅ Fast, deterministic execution
  • ✅ Transparent repair workflow (PRs or in-place updates)
  • ✅ Maintainable Playwright scripts — no vendor lock-in
  • ✅ AI-powered adaptability when things break

This is what makes Smart Tests a smarter middle ground.

Try it out!

In your IDE:

  • Download TestChimp VSCode Extension (Search for TestChimp in the extension marketplace)
  • Write a test scenario in plain English, save the file as a .txt file
  • Right-click on the file, select TestChimp -> Generate Script
  • TestChimp will generate the script and write it to your tests folder.
  • Want Smart test execution in your CI? Checkout our GitHub Action guide.

From TestChimp Web:

  • Login to TestChimp.
  • In Homepage, click on “Create a Smart Test”
  • Write your test scenario in natural language
  • If you have a specific playwright.config.js file, you can attach it (to enable your env specific behaviours such as request header overrides, browser auth etc.)
  • The script will be generated for you, which can be downloaded and put in your codebase

 

Share the Post:

Related Posts