# How do I deal with flaky tests in GitHub Actions?

GitHub Actions has no flaky-test tracking. Detect a flaky test by re-running one commit, quarantine it in a non-blocking job, then fix it. Working YAML.

By Grant Unwin, published 2026-11-17. Checked against GitHub's documentation on 2026-11-17. The web page: https://www.runners.io/blog/flaky-tests-github-actions

Treat a flaky test as three jobs, in this order: detect it, quarantine it, fix it. GitHub Actions has no flaky-test tracking that we could find in its documentation on the date this post was last checked, so you measure it yourself. Re-run the same commit: a test that passes and fails on one commit is flaky, and a workflow run can be re-run at most 50 times. Then move the test to a job that cannot block a merge.

**In short**

- A test is flaky when it passes and fails on the same code, so a re-run of the same commit is the first test.
- A re-run reuses the commit SHA and ref; a run can be re-run 50 times, up to 30 days after it started.
- gh run rerun RUN\_ID --failed re-runs only the failed jobs and the jobs that depend on them.
- continue-on-error: true on a job keeps a quarantined test from failing the run; leave that job out of your required checks.
- GitHub rounds each job up to a whole minute, so every re-run of a short job still bills a full minute.

## How do I tell a flaky test from a broken one?

It is flaky if it gives both answers on the same code. That is the definition GitHub’s own engineers use in [their 2020 post on flaky builds](https://github.blog/engineering/engineering-principles/reducing-flaky-builds-by-18x/), quoting Google’s: a test that shows both a passing and a failing result with the same code. A re-run of a workflow uses the same commit SHA and the same ref, so it tests exactly that. GitHub allows a re-run up to 30 days after the first attempt, and at most 50 re-runs of one run.

Our rule of thumb, which is ours and not GitHub’s: run the test N times on one commit. If it fails 0 times, it is stable. If it fails N times, it is broken, and no amount of retrying will help. Anything in between is flaky, and the fraction that failed is a number you can track and compare after a fix.

## How do I detect flaky tests in GitHub Actions?

There are four methods and they trade minutes for certainty. The table sets out what each one tells you and what it costs.

| Method | What it shows | What it costs |
| --- | --- | --- |
| Re-run the failed jobs on the same commit | A re-run reuses the commit SHA and ref, so a pass after a fail is the same code giving two answers | The failed job again, and every job that depends on it, each rounded up to a whole minute |
| Repeat the changed tests inside one job | A test that fails in some of N runs of one commit; 0 of N is stable and N of N is broken | N times the tests' own time, in one job |
| Let the test runner retry | Playwright reports a test that failed and then passed on retry as flaky; Jest reruns failed tests until they pass or the retries run out | Only the failing test runs again |
| Compare workflow runs of one commit | A commit with both a passed and a failed run, though not which test it was | None: it reads runs you already paid for |

**Re-run the failed jobs.** In the run summary choose *Re-run failed jobs*, or from a terminal run `gh run rerun RUN_ID --failed`. GitHub also re-runs the jobs that depend on the failed ones. Add `--debug` to turn on runner diagnostic logging for the attempt. This is the cheapest way to start, and its cost is a full job again: GitHub rounds each job up to a whole minute.

**Repeat the changed tests.** This catches a new flaky test before it merges. The job below finds the test files a pull request adds or changes, runs them five times on the same commit and fails only when the answers differ. The loop body was run locally against a stub test command: with a command that always passed, one that always failed and one that failed on alternate runs, it exited 0, 1 and 1, and wrote a count of failed runs to the summary each time. The workflow was linted with actionlint 1.7.12 and has not run on GitHub.

```yaml
name: flaky-check
on:
  pull_request:
    paths: ["tests/**"]
permissions:
  contents: read
jobs:
  repeat:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0
      - uses: actions/setup-node@v7
        with:
          node-version: 22
      - run: npm ci
      - name: Run the changed test files five times
        env:
          BASE_REF: ${{ github.base_ref }}
          RUNS: "5"
        run: |
          files=$(git diff --name-only --diff-filter=AM "origin/$BASE_REF"...HEAD -- 'tests/*.test.js')
          [ -z "$files" ] && { echo "No changed test files."; exit 0; }
          fails=0
          for _ in $(seq "$RUNS"); do
            # shellcheck disable=SC2086
            npx jest $files || fails=$((fails + 1))
          done
          {
            echo "### Repeat check"
            echo "Files: $files"
            echo "Failed $fails of $RUNS runs on the same commit."
          } >> "$GITHUB_STEP_SUMMARY"
          # 0 failures: stable. All failures: broken. Anything between: flaky.
          if [ "$fails" -gt 0 ] && [ "$fails" -lt "$RUNS" ]; then
            echo "::error::Flaky: $fails of $RUNS runs failed"
            exit 1
          fi
          [ "$fails" -eq 0 ] || exit 1
```

The base branch comes in through an environment variable, not straight into the script, so a branch name cannot become a command. The pattern in the `git diff` line is a git pathspec, where a `*` also matches a slash, so `tests/*.test.js` finds tests in nested folders.

**Let the test runner retry.** Playwright sorts every test into passed, flaky or failed, and calls it flaky when it failed on the first run and passed on a retry; retries are off until you set `retries: 3` or pass `--retries=3`. Jest has `jest.retryTimes(3, {logErrorsBeforeRetry: true})`, which works only with its default `jest-circus` runner and must sit at the top level of a test file or in a `describe` block. A retry can turn a red run green, so turn on `logErrorsBeforeRetry`, which logs the errors that caused the failure, or you will not see that it happened.

GitHub’s engineers found that where you retry matters. Their post says the first two methods caught about a quarter of flaky failures, and that rerunning the failed test three ways, in the same process, in the same process shifted into the future and on a different host, caught 90 per cent. Those are GitHub’s figures for its own codebase in 2020, not a promise for yours. The lesson for a workflow is that a retry on the same machine in the same process can pass for the same reason it failed.

**Compare runs of the same commit.** This costs nothing, because the runs already exist. The workflow below lists the commits on which your `ci.yml` both passed and failed. It tells you a workflow flaked, not which test; use it to decide where to look. The `jq` filter was run against a sample of six runs and printed only the one commit with a pass and a fail; the workflow was linted and has not run on GitHub.

```yaml
name: flaky-history
on:
  workflow_dispatch:
  schedule:
    - cron: "17 6 * * 1"
permissions:
  actions: read
  contents: read
jobs:
  same-commit:
    runs-on: ubuntu-24.04
    steps:
      - name: Commits with both a passed and a failed run
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
          gh run list --repo "$GITHUB_REPOSITORY" --workflow ci.yml --limit 200 \
            --json headSha,conclusion,displayTitle,url > runs.json
          jq -r '
            map(select(.conclusion == "success" or .conclusion == "failure"))
            | group_by(.headSha)
            | map(select(any(.[]; .conclusion == "success") and any(.[]; .conclusion == "failure")))
            | .[] | "\(.[0].headSha[0:10])  \(length) runs  \(.[0].displayTitle)"
          ' runs.json | tee -a "$GITHUB_STEP_SUMMARY"
```

## How do I quarantine a flaky test without hiding it?

Move it to a job whose failure does not fail the run, and keep it running so you can see it. Split the tests by folder: the job that gates merges ignores the quarantine folder, and a second job runs only that folder with `continue-on-error: true`, which GitHub documents as preventing a workflow run from failing when the job fails. Do not add the quarantine job to your required checks.

```yaml
name: ci
on:
  pull_request:
  push:
    branches: [main]
permissions:
  contents: read
jobs:
  test:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 22
      - run: npm ci
      - run: npx jest --testPathIgnorePatterns=tests/quarantine/
  quarantine:
    runs-on: ubuntu-24.04
    continue-on-error: true
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 22
      - run: npm ci
      - run: npx jest --testPathPatterns=tests/quarantine/ --passWithNoTests
```

A quarantine only works if it has an exit. Give each quarantined test an owner and a date, and delete the quarantine folder’s contents on that date or fix the test. Python projects can do the same with `@pytest.mark.xfail`: by default neither an expected failure nor an unexpected pass fails the suite, and `strict=True` makes an unexpected pass fail it, which tells you the test has recovered. Do not use `run=False` for a flaky test, because it stops the test running and you learn nothing.

The quarantine job is billed like any other job. If it is slow, it costs minutes without protecting anything, so keep the folder small.

## How do I fix the cause?

Look for what the test shares with the others or with the machine: the order it runs in, a port or a file, the clock, the network, and the memory and CPU of the runner. Four checks find most of them.

**Shuffle the order.** Jest’s `--randomize` shuffles the tests within a file and prints the seed, so a failure can be reproduced with `--randomize --seed 1234`. A test that fails only in some orders depends on one that ran before it. **Lower the parallelism.** If the failures stop with `--maxWorkers=1`, the tests share a port, a database or a file. And **check the machine.** A test that fails only when the job is busy may be starved: the post on [out-of-memory kills](https://www.runners.io/blog/github-actions-runner-out-of-memory) shows how to measure a job’s peak, and the one on [the shutdown signal](https://www.runners.io/blog/github-actions-runner-received-a-shutdown-signal) covers a runner that goes silent halfway through. The fourth check is time: pin the clock and the time zone in the test, since a test that fails near midnight or at the turn of a month is a test of the date.

## Should I retry failing tests automatically?

Use retries to unblock people, and track them so that they are not the fix. GitHub’s engineers kept builds green by rerunning a failed test, but recorded every failure, scored it by how many branches and developers it affected, and opened an issue assigned to the likely author once a test crossed a threshold. The retry kept the build green and the record found the tests that were worth fixing. They report that flaky builds fell from about 1 in 11 commits to under half a percent in six weeks, GitHub’s figure for its own tooling.

## What does a flaky test cost in minutes?

Take a ten-minute job on a standard 2-core Linux runner, which GitHub prices at $0.006 a minute: a run costs $0.06. If a flaky test fails one run in four and you re-run the failed job each time, the expected number of extra runs per pull request is one third, since 0.25 divided by 0.75 is a third, so about 3.3 extra minutes and $0.02. Across 40 pull requests a day that is about $0.80 a day, and the larger cost is a developer waiting for a re-run. The assumptions are made up for the sum; replace them with your own.

On our runners, if a build writes JUnit XML the job page gets a Tests tab that lists each failing test by name with its failure message, kept for 30 days, as the [test results page](https://www.runners.io/docs/observability/test-results) describes. It shows one job’s results. It does not compare runs, so the flaky history is still yours to keep. For the other ways to cut minutes, see the posts on [running less](https://www.runners.io/blog/speed-up-github-actions-by-running-less) and [making GitHub Actions faster](https://www.runners.io/blog/how-to-make-github-actions-faster). Build times measured on GitHub’s runner and on ours are on our [faster GitHub Actions runners, measured](https://www.runners.io/faster-github-actions) page.

## Sources

- [Re-running workflows and jobs](https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs)
- [Workflow syntax for GitHub Actions](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax)
- [Actions runner pricing](https://docs.github.com/en/billing/reference/actions-runner-pricing)
- [Reducing flaky builds by 18x (The GitHub Blog, 16 December 2020)](https://github.blog/engineering/engineering-principles/reducing-flaky-builds-by-18x/)
- [Retries (Playwright)](https://playwright.dev/docs/test-retries)
- [The Jest object: jest.retryTimes (Jest)](https://jestjs.io/docs/jest-object)
- [Jest CLI options](https://jestjs.io/docs/cli)
- [How to use skip and xfail (pytest)](https://docs.pytest.org/en/stable/how-to/skipping.html)
- [Test results (runners.io documentation)](https://www.runners.io/docs/observability/test-results)

## Frequently asked questions

### How do I deal with flaky tests in GitHub Actions?

Detect, quarantine, then fix. Re-run the same commit: GitHub reuses the SHA, so a pass after a fail is the same code giving two answers. Then move the test to a job with continue-on-error: true and fix the cause.

### Does GitHub Actions detect flaky tests?

We found no flaky-test tracking in GitHub's Actions documentation or changelog on the date this post was last checked. You detect them yourself, by re-running a commit, repeating tests in a job, using your test runner's retry reporting, or comparing runs of the same commit.

### How do I re-run only the failed jobs?

In the run summary choose Re-run failed jobs, or run gh run rerun RUN\_ID --failed. GitHub also re-runs the jobs that depend on the failed ones. A run can be re-run at most 50 times.

### How do I stop a flaky test blocking merges?

Run it in its own job with continue-on-error: true and exclude it from the job that gates merges. GitHub documents the setting as preventing a workflow run from failing when the job fails. Do not make that job a required check.
