Split by recorded timing rather than by file count, and add shards only while each one saves more waiting than it costs in machine time. We measured it on the 9,320 tests of networkx: one job took a median 132 seconds and billed 3 minutes. Four jobs split by timing with our script took 68 seconds and billed 5, and sixteen took 55 seconds and billed 16. Below are the runs, the splitter and a calculator for your suite. If you find GitHub Actions slow, our guide has more fixes.
- As one job, 9,320 tests took a median 132 seconds and billed 3 minutes on a GitHub-hosted runner.
- Split by recorded timing with our script, four jobs took 68 seconds and billed 5; sixteen took 55 and billed 16.
- Our timing split beat file count at four and eight shards and tied at sixteen. pytest-split did not beat it everywhere.
- Each shard repeats about 32 seconds of setup, and GitHub rounds every job up to a whole minute.
- The calculator finds the shard count past which another shard saves less wait than it adds to the bill.
Adding jobs shortens the wait, but every job repeats its setup and GitHub rounds each job up to a whole minute, so a job that finishes in 40 seconds bills 60. The only way to know where more shards stop paying is to count the minutes. We did, in a public repository on GitHub-hosted runners, where standard runners are free.
What we measured, and on what
The suite is networkx at release 3.7: 9,320 passing tests in 263 test files with recorded timings, run with plain pytest in one process. We chose it because it is a real suite that anyone can read, with a few very uneven files. Every job ran on ubuntu-24.04: check out, install, run its share. We ran the whole suite as one job ten times, and then each of twelve splits (three ways, at 2, 4, 8 and 16 shards) five times: 450 split jobs and 10 baseline jobs. Every split ran all 9,320 tests with no failures.
We report absolute seconds and the minutes GitHub would bill, never a ratio. One job took a median 132 seconds, from 117 to 177 over the ten runs. The pytest step took a median 94 seconds (pytest itself printed 91) and about 36 seconds were everything else: starting, two checkouts, the install. That 36 seconds is the number to hold on to. Every shard pays it again. The method, the workflow and every row are in the repository.
The three ways to split
By file count cuts the list of test files into equal chunks in path order. It needs no data and it is what most pipelines do. By timing needs a record of how long each test took. We tried two: a 30-line script of ours, and pytest-split with its least_duration algorithm, which assigns each test to the group with the smallest total so far (its default is duration_based_chunks). Both read the same timings, recorded once with --store-durations.
Our script is the whole idea in one place: sort the files by recorded time, longest first, and give each to the shard with the least work so far.
#!/usr/bin/env python3
"""Print the test files (or pytest-split arguments) for one shard.
usage: split.py STRATEGY SHARDS INDEX TESTS_DIR DURATIONS_JSON
STRATEGY
count consecutive chunks of equal file count, files in path order
greedy largest recorded file time first, each file onto the least-loaded shard
"""
import json, math, pathlib, sys
strategy, shards, index = sys.argv[1], int(sys.argv[2]), int(sys.argv[3])
root, durations_path = pathlib.Path(sys.argv[4]), sys.argv[5]
files = sorted(p.relative_to(root).as_posix() for p in root.rglob("test_*.py"))
per_test = json.load(open(durations_path)) # pytest-split format: "path::test" -> seconds
seen = {}
for test_id, seconds in per_test.items():
f = test_id.split("::")[0]
seen[f] = seen.get(f, 0.0) + seconds
mean = sum(seen.values()) / max(len(seen), 1)
weight = {f: seen.get(f, mean) for f in files} # a file with no record counts as the mean
if strategy == "count":
size = math.ceil(len(files) / shards)
mine = files[index * size : (index + 1) * size]
elif strategy == "greedy":
load = [0.0] * shards
buckets = [[] for _ in range(shards)]
for f in sorted(files, key=lambda f: (-weight[f], f)):
k = min(range(shards), key=lambda k: (load[k], k))
buckets[k].append(f)
load[k] += weight[f]
mine = sorted(buckets[index])
else:
sys.exit(f"unknown strategy {strategy}")
print("\n".join(mine))
And this is the workflow that runs four shards with pytest-split. It is the recipe file in the repository, with the timings committed beside it. The measured runs used a larger workflow, F2-test-splitting.yml, that passes the same pytest-split flags.
jobs:
test:
runs-on: ubuntu-24.04
timeout-minutes: 15
strategy:
fail-fast: false
matrix:
group: [1, 2, 3, 4]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
repository: networkx/networkx
ref: 49ca2290b80c1389eb4d8d8e49dcba73f8fec016
path: nx
- uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
with:
python-version: "3.13"
- run: python -m pip install --quiet -e 'nx[default,test]' pytest-split
- name: run this shard
working-directory: nx
run: >
pytest networkx -p no:cacheprovider -q
--splits 4 --group ${{ matrix.group }}
--splitting-algorithm least_duration
--durations-path ../studies/F2-test-splitting/durations.json
Two keys matter. fail-fast: false lets every shard report, so one failure does not hide the others. The group list must match --splits. The matrix post covers the rest of the strategy block.
How long it took, and what it billed
Medians over five runs of each split, with the lowest and highest in brackets. Wall time is how long the slowest job took, in seconds. Billed minutes are every job rounded up and added. The last column is the pytest time of the longest and the shortest shard, in seconds.
| Split | Shards | Wall time, s | Billed minutes | Longest / shortest shard, s |
|---|---|---|---|---|
| One job | 1 | 132 (117 to 177) | 3 (2 to 3) | 91 / 91 |
| By file count | 2 | 97 (90 to 120) | 4 (3 to 4) | 56.8 / 29.6 |
| By file count | 4 | 78 (60 to 86) | 5 (4 to 6) | 42.0 / 11.2 |
| By file count | 8 | 69 (66 to 83) | 9 (9 to 10) | 35.8 / 2.3 |
| By file count | 16 | 55 (51 to 63) | 16 (16 to 17) | 20.9 / 0.6 |
| By timing, our script | 2 | 85 (81 to 108) | 4 (4 to 4) | 45.7 / 41.4 |
| By timing, our script | 4 | 68 (61 to 85) | 5 (5 to 6) | 29.7 / 17.1 |
| By timing, our script | 8 | 57 (51 to 69) | 8 (8 to 9) | 22.9 / 6.2 |
| By timing, our script | 16 | 55 (50 to 57) | 16 (16 to 16) | 15.3 / 1.4 |
| By timing, pytest-split | 2 | 112 (86 to 113) | 4 (4 to 4) | 75.5 / 49.4 |
| By timing, pytest-split | 4 | 76 (72 to 79) | 7 (7 to 8) | 39.3 / 27.9 |
| By timing, pytest-split | 8 | 62 (60 to 85) | 9 (8 to 10) | 26.8 / 17.5 |
| By timing, pytest-split | 16 | 58 (55 to 62) | 16 (16 to 18) | 19.8 / 12.2 |
At $0.006 a minute on a standard Linux runner, the one-job run is $0.018 and sixteen shards are $0.096. In a public repository like ours both are free. In a private one, the bill is the point. Build times measured on GitHub’s runner and on ours are on our faster GitHub Actions runners, measured page.
Three things stand out. Sixteen shards bought a wall time of 55 seconds against 132 for one job, and the bill went from 3 minutes to 16. Four shards split by timing took 68 seconds for 5 minutes, and doubling to eight saved 11 more seconds for 3 more minutes. And at sixteen, all three ways of splitting finished within 55 to 58 seconds, because each job’s fixed 32 to 36 seconds had become most of its time.
Is splitting tests by timing better than by file count?
Yes, with our script, though the gain is modest. At four shards it finished in 68 seconds against 78 for file count, and at eight in 57 against 69. By file count, the longest shard at sixteen ran 20.9 seconds and the shortest 0.6, and 30 of the 80 shards ran their tests in under two seconds. Each still paid its fixed cost and a whole billed minute.
pytest-split did not beat file count everywhere: 112 seconds at two shards against 97, 76 at four against 78, and 62 at eight against 69. A shard cannot know which tests are its own until the whole suite is collected, and we think that is why even its shortest shard at sixteen ran 12.2 seconds. We timed collection on its own: all of networkx takes about 9.9 seconds, 9.3 to 10.1 over five runs. Our script names files, so each shard collects only its own. For a splitter that works this way, collection is part of every shard’s fixed cost.
Why recorded timings are not the whole story
One file broke our prediction. The timings recorded for test_common.py, which holds 740 tests marked as expected failures, put it at 4.4 seconds. The shard that held it ran 22.9 seconds at eight shards when we split by timing, and 35.8 by file count. The other seven ran a median 9.8 seconds by timing and 4.5 by file count. With our script it was the longest shard in all 15 of our runs at four, eight and sixteen shards. It is not collection: that file collects alone in 0.65 seconds. We do not know what it is.
The suite as a whole shows the same gap. The recorded durations add up to 67 seconds, and pytest reported 88 for the run they were recorded in. So the lesson is not that timings are wrong. It is that a split is a prediction, and the check is cheap: after the first run, compare each shard’s time to the others’. If one is far above, look at what is in it, and record the timings again. pytest-split’s README advises refreshing them after major changes to the suite.
How many shards before it costs more than it saves?
Going from n to n + 1 shards shortens the wait by the suite’s test time divided by n times n + 1, and it adds one more job’s fixed seconds of machine time. The extra shard stops paying when n times n + 1 passes the test time divided by the fixed time. That puts the knee near the square root of the test time over the fixed time, with the slowest shard’s overrun counted in. For our suite, with about 91 seconds of tests, 32 of fixed cost and a slowest shard running 30 per cent over an even share (as at four shards), the knee is one shard: the setup costs about a third of the suite, so splitting never lowers the bill. It does lower the wait, and that is the trade.
The calculator below is that model, with GitHub’s rounding. Each job is billed on its own length, rounded up, so a shard under a minute bills one. We checked it against our 12 measured splits, feeding it each split’s own measured imbalance: it matched the billed minutes exactly in nine and was within two minutes in all twelve, and its wall times ran a few seconds under the measured ones. The imbalance is an input because it is the thing you cannot know until you run a split.
- That bills 5 minutes, $0.03 at GitHub's Linux 2-core rate, against 3 minutes as one job.
- Past 1 shard, each extra shard saves less wall time than the 32 seconds it adds to the bill.
- The fewest billed minutes is one job, at 3 minutes.
| Shards | Wall time | Billed minutes | Over one job | At GitHub's rate |
|---|---|---|---|---|
| 1 | 2m 03s | 3 | none | $0.02 |
| 2 | 1m 31s | 4 | +1 | $0.02 |
| 3 | 1m 11s | 4 | +1 | $0.02 |
| 4 | 1m 02s | 5 | +2 | $0.03 |
| 6 | 0m 52s | 6 | +3 | $0.04 |
| 8 | 0m 47s | 8 | +5 | $0.05 |
| 12 | 0m 42s | 12 | +9 | $0.07 |
| 16 | 0m 39s | 16 | +13 | $0.10 |
| 24 | 0m 37s | 24 | +21 | $0.14 |
| 32 | 0m 36s | 32 | +29 | $0.19 |
Each job is rounded up to a whole minute on its own, as GitHub bills it. A public repository’s standard runners are free.
Try the other end. A suite of 20 minutes of tests with 60 seconds of fixed cost and a 10 per cent overrun has its knee at four shards: the calculator gives 390 seconds and 25 billed minutes at four, against 1,260 seconds and 21 minutes as one job. Four extra minutes bought a wait of 6.5 minutes instead of 21. Past it, eight shards save another 165 seconds and cost 7 more minutes, and sixteen shards cost 48 minutes in all for a 143 second wait. Splitting pays when the tests are long next to the setup.
What to do
Measure your fixed cost first: the time from a job starting to its first test. If it is a third of your test time or more, cache the install before you split, because every shard pays it (the cache post and the run-less post cover it). Then record timings with a complete run, split by them, and compare the shards after the first run. Add shards while the next one still saves more waiting than it costs, using the table or the calculator, and stop there. A test with no recorded time is given the average, so new tests do not break the split. If your shards write JUnit XML, our test results page shows each job’s tests, and the pricing post has GitHub’s rates for the minutes you add.
One suite, one language and one day. A suite with a heavier install or longer tests moves the knee. JavaScript runners that split by file, such as Jest and Playwright, we did not run, so none of the figures above is about them.
Sources
- Actions runner pricing (docs.github.com)
- GitHub Actions billing (docs.github.com)
- pytest-split README (github.com)
- networkx (github.com)
- The runs, workflows and data behind this post (ci-measurements, study F2) (github.com)
- ci-measurements: public measurements of GitHub Actions (github.com)
Frequently asked questions
How many shards should I split my GitHub Actions tests into?
Enough that each extra shard saves more waiting than the seconds it adds to the bill: roughly the square root of test time over fixed time. For 91 seconds of tests with 32 fixed it is one. For 20 minutes of tests with 60 fixed it is four.
Is splitting tests by timing better than by file count?
Yes with our script, though modestly. At four shards our runs took 68 seconds against 78 by file count, and at eight 57 against 69. At sixteen both took 55, because setup was most of each job. pytest-split did not win everywhere.
Does splitting tests across jobs cost more minutes?
Usually. Every shard repeats its setup and GitHub rounds each job up to a whole minute. Our suite billed 3 minutes as one job, 5 as four shards split by our script, and 16 as sixteen. Standard runners in public repositories are free.
How do I record test timings for pytest-split?
Run a complete suite with --store-durations, keep the file it writes, and pass it with --durations-path. A test with no recorded time gets the average. Refresh the file after major changes, as its README advises.