Published
Flaky Tests: Causes, Detection and How to Fix Them
Short answer
A flaky test passes and fails on the same code with no change to the code or the test. Most E2E flakes come from timing, shared data, test order and environment drift. Find them by re-running (Playwright marks a test flaky when it fails first and passes on retry), then fix the cause instead of adding more retries.
Last updated: 2026-11-10
What is a flaky test?
A flaky test is one whose result changes between runs even though the code under test has not changed. It fails on Tuesday, passes on Wednesday, and nobody can say what is different. Because the failure is not tied to a real bug, people learn to re-run the pipeline and ignore red builds, and that habit is the real cost.
Playwright gives the idea a precise definition. In its test runner, a test that passes on the first run is passed, a test that fails on the first run but passes on a retry is flaky, and a test that fails on the first run and every retry is failed. The runner lists flaky tests in their own section of the summary, so a green build can still tell you which tests needed a second attempt.
Why do E2E tests flake?
Most flakes trace back to one of five causes. Each has its own fix guide.
| Cause | What it looks like | Guide |
|---|---|---|
| Timing | Passes locally, times out in CI; a fixed waitForTimeout that is sometimes too short | Fix flaky waits and timeouts in Playwright |
| Shared data | Two tests, or two parallel workers, write the same record | Avoid test data collisions in parallel CI |
| Hardcoded data | A fixed email, ID or date that is already taken or has expired | Fix hardcoded test data that flakes in CI |
| Test order | A test passes only after another one has run | Fix E2E tests that depend on run order |
| Environment drift | Staging has different config, flags or seed data than local | Fix E2E tests that pass locally but fail on staging |
Two further causes are worth checking when the five above do not explain it: leaked state from missing teardown (stop E2E tests leaking test data) and selectors that change when the UI changes (fix E2E tests that break when selectors change).
How do I detect flaky tests?
Turn on retries and read the flaky section. Playwright does not retry failing tests by default. Enable retries with the --retries=3 flag or the retries option in the config. When a test fails, Playwright discards the worker process and browser and starts a new worker for the retry, so the retry begins from a clean browser. A test that fails and then passes is reported as flaky, and each attempt is printed.
import { defineConfig } from '@playwright/test';
export default defineConfig({
retries: process.env.CI ? 2 : 0,
use: { trace: 'on-first-retry' },
});
Repeat a suspect test. To check whether one test is unstable, run it many times in a row:
npx playwright test checkout.spec.ts --repeat-each=20
A test that is stable should pass all 20. One failure in 20 is a 5% flake rate, and that is enough to turn a pipeline red several times a week once your suite grows.
Record a trace on the retry. With trace: 'on-first-retry' you get a trace.zip for the failed attempt. Our guide to the Playwright Trace Viewer shows how to read it. Compare the failed trace with a passing run and look for a difference in network timing or in the data the page loaded.
Let the code know it is a retry. Tests, hooks and fixtures can read testInfo.retry. This is useful for clearing a server cache before a retry, and it is also a quick way to log when a retry happened.
How do I fix a flaky test?
Work in this order.
- Reproduce it. Use
--repeat-eachand, if the flake only appears in CI, run the same command with the same worker count there. A flake you cannot reproduce is a flake you cannot confirm you fixed. - Read the trace of the failed attempt. The Errors, Network and Console tabs usually name the cause: a request that never returned, an element that was covered, or an error in the app console.
- Replace fixed sleeps with waiting on a condition. Locator actions already wait for the element to be actionable. For anything else, assert on the state you need, not on elapsed time.
- Isolate the data. Give each test its own records and clean them up. If two workers can touch the same row, they will.
- Remove order dependence. Each test should set up what it needs. Playwright's serial mode (
test.describe.serial) skips later tests after a failure and retries the whole group together, so use it only when steps genuinely depend on each other. - Only then consider retries. A retry keeps the build green while the cause stays in the suite.
Should I just add retries?
Retries are a detection tool and a short-term bandage. They keep a pipeline green, which is also why a flaky test can sit in a suite for months. If you use retries, make the flaky count visible: Playwright's summary already lists flaky tests, and you can export the report and track the number over time. A rising count means the suite is getting less trustworthy even though the builds are green.
Flaky tests and agent-written tests
When a coding agent writes and repairs your tests, flakiness has a second source. An agent that sees a red test often "fixes" it by adding a longer wait, loosening an assertion or deleting the check. The test goes green, the flake stays, and the suite now asserts less than before.
This is where review matters. TestChimp links each Playwright test to a written scenario in your repo and flags agent-edited tests for human review, so a change that weakens a test is visible in review. For the review process, see How to trust AI-written tests and how to review tests your coding agent wrote.
Frequently asked questions
What is a flaky test?
A flaky test gives different results on the same code with no change to the code or the test. In Playwright, a test that fails on the first run and passes on a retry is reported as flaky.
What causes flaky tests?
The most common causes in E2E suites are timing and fixed waits, shared or hardcoded test data, tests that depend on run order, and differences between environments such as staging and local.
How do I find flaky tests in Playwright?
Enable retries with the retries config option or the --retries flag and read the flaky section of the summary. Run a suspect test many times with --repeat-each, and record a trace on the first retry to see what differed.
Does Playwright retry failed tests by default?
No. Failing tests are not retried unless you set retries in the config or pass --retries on the command line. When a test fails, Playwright starts a new worker for the retry.
Are retries a good fix for flaky tests?
Retries are good for detection and for keeping a pipeline moving while you investigate. They hide the flake from the build status but do not remove its cause, so track the flaky count and fix the root cause.
Can a coding agent make flaky tests worse?
It can hide them. An agent may add a longer wait or loosen an assertion to turn a test green, which makes the suite assert less. Reviewing agent-edited tests catches this.
Keep green builds meaning something
Fix flakes at the cause, and keep an eye on what your agents change in your tests. TestChimp links tests to scenarios, shows coverage and flags agent-edited tests for review. 14-day free trial, no credit card.