Add

Coding & Software

Debug a Flaky Test

Finds why a test passes sometimes and fails other times, ranks the likely causes and gives an experiment to prove each one.

Designed for general-purpose AI assistantsPublished November 2026

Customize

Fill in the fields and the prompt updates. What you type stays in this page and is never saved or sent.

Preview

A test in my project fails intermittently. Help me find the cause by experiment, not by guessing.

Test framework and language: {{framework and language}}
Where it fails: CI only
How often: roughly 1 in 10 runs
What changed recently: not sure

The test:
{{test code}}

Code under test or setup it depends on:
{{related code}}

Failure output from a failing run:
{{failure output}}

Output from a passing run, if I have one:
not available

Work through these categories and say which ones the evidence supports, which it argues against, and which I cannot judge yet: shared state between tests, test order, time or timezone, randomness, real network or disk, async work that is not awaited, resource limits in CI, and data that depends on other tests.

Then give me:
1. The three most likely causes, ranked, each tied to a specific line or fact I gave you.
2. For each cause, one experiment that would support or rule it out, such as running the test alone, in random order, in a loop, with a fixed seed or with the clock frozen. Give the command or code change.
3. What the fix usually looks like for each cause, without applying it until I confirm the cause.
4. What extra logging or output I should capture on the next failure if none of the three fits.

Do not suggest adding retries or sleeps as the fix. Treat them as a last resort and say what they would hide.

4 required fields are empty and will stay as a {{placeholder}} in the copied text.

How to use it

  1. Include a failing and a passing output if you can. The difference between them is often the whole diagnosis.
  2. Run the experiments in order and report each result back to the assistant before applying a fix.
  3. Say where it fails. A test that only fails in CI points to resource limits, parallelism or environment differences, not to the test logic.
  4. Paste the setup and teardown code too. Shared state usually lives there, not in the test body.

Example output

Illustrative only. Real answers depend on your input and the assistant you use.

Ranked causes:
1. Shared state between tests (most likely). test_cancel_order and test_create_order both write to the same orders table, and your fixture only truncates it at the start of the module.
2. Unawaited async work. The call to send_email on line 14 is not awaited, so the assertion on line 16 sometimes runs before the email record exists.
3. Time dependence. The assertion compares created_at to now() within one second, which fails when the test straddles a second boundary.

Experiments:
1. Run pytest -p no:randomly tests/test_orders.py::test_cancel_order 50 times in a loop. If it never fails alone, shared state or test order is one possible explanation, not proof. Next, run it together with test_create_order to see whether that pairing reproduces the failure.
2. Add await before send_email and run the loop again.
3. Freeze time with freezegun and rerun.

If none fits: log the row count of orders at the start and end of the failing test and capture it in CI.

Tips

A flaky test is rarely random. The ranked experiments turn "it fails sometimes" into a sequence of yes/no questions.

Keep a note of each experiment and its result in the same conversation so the assistant reasons from your evidence instead of restarting each time.

Be wary of fixes that add a sleep or a retry. They make the failure rarer, not gone, and they hide the next real bug.

Back to the Prompt Library

Related prompts

  • Coding & Software

    Debug an Error Message

    Paste an error and the code around it to get ranked causes, the smallest fix and a quick check to confirm the diagnosis.

    Customizable fields
    5 fields

  • Coding & Software

    Write Unit Tests for a Function

    Generates focused unit tests for one function, covering normal cases, boundaries and invalid input without mocking more than needed.

    Customizable fields
    3 fields

  • Coding & Software

    Document an API Endpoint

    Writes reference documentation for one API endpoint from its code, covering parameters, responses, errors and a working example.

    Customizable fields
    6 fields