Our runners against GitHub-hosted, measured.

Reading the table

Every percentage is against our own row, and on all of these columns less is better — so a positive number is slower or dearer than us, and a negative one beats us. Read the two rows here as two measurements built the same way, not as a map of the market. Shorter bar is better: each is drawn against the slowest row in its own column, which is why a bar cannot be compared across columns — a cargo build takes minutes and an eslint run takes seconds. Colour grades the gap on a fixed scale: yellow is level with us, orange half behind, red twice the time or the price. Our own row is the mark everything is set against and is shown green; on any other row green would mean that provider beat us, and nothing earns it for merely keeping up.

The machine on each row

Machine is the shape each provider advertises, at the label the benchmark asked for. Both rows are the same shape. GitHub’s is the private-repository runner; a public repo gets 4 vCPU / 16 GB and that row would move.

What arrives is not always what is advertised. Probed inside every timed job by saturating every visible core, we deliver 1.9 of our 2 advertised cores and GitHub delivers 1.8 of its 2. Both are medians over the 44 approved rounds from 2026-09-07 to 2026-10-08 in which our guest was shown the 2 vCPU we sell. Rounds before 2026-09-07 18:45 UTC ran our guest with 4 vCPU rather than the 2 we sell, and were withdrawn from this comparison because that machine is no longer sold.

What is timed, and who times it

Times are end-to-end wall clock — from the workflow being created to the job finishing, including queue, provisioning, checkout, toolchain, dependencies and build. They are read from GitHub’s own API rather than reported by the runners, so no provider times itself, ourselves included. GitHub-hosted is not the reference row, but it is measured exactly the same way: if that is what you run today, read that row. The cost column is not end to end; it is worked out from billed time only, below.

Queue

Queue is how long a job waits before a machine picks it up — created to started. It is already inside the workload times to its left, not added to them, so do not count it twice. Nobody charges for it, ourselves included: our meter starts when your code starts, so queue never reaches the cost column. It is here because your pull request is blocked for it either way, and a runner that is quick once it starts but slow to find you a machine is not a quick runner. The figure is a typical wait rather than a worst case: a job that queued three or more times its own provider’s median queue is dropped, and the longest wait it removed on this data was 95 seconds.

This is the column we lose. A job typically waited 36s for one of our machines, against 27s on GitHub-hosted. GitHub’s scheduler hands a job straight to its own runners. Ours has to hear about the job from GitHub, give it a fresh machine and register that machine as a runner before the job can be assigned to it — time the arrangement costs before any of your code runs, rather than a slow machine. It is already inside every end-to-end figure above, so the speed comparison pays for it in full.

How cost per build is worked out

Cost per build is worked out only from the time each provider bills: the job itself, from the moment a machine picks it up to the moment it finishes, as GitHub records it. Queue is in no row’s cost, ours included. Each provider’s minutes follow that provider’s own billing rule: we round each job up to the whole minute, with a one-minute minimum; GitHub rounds each job up to the whole minute. Each workload’s median job is billed under that rule at the provider’s published rate for the exact machine it was benchmarked on, and the workloads are averaged. Both rows use the pay-as-you-go rate — what a minute costs someone who signed up and ran a build, with no commitment and no prepaid discount.

Cold builds, and each row's configuration

Every provider’s configuration, in full. The benchmark is built to measure a cold build on every row: it never calls actions/cache, dependencies are fetched fresh each run, and toolchains are installed by each project’s own workflow. Anything that would carry state from one run into the next is turned off, so that a row measures a provider rather than that provider’s history. That is also the reason some of their features do nothing here, which we would rather say than let a cold run be mistaken for a full product comparison.

Rounds, the window, and the one exclusion

Each figure is the median of the rounds that provider completed, taken from 44 rounds between 2026-09-07 and 2026-10-08 — the window reaches about 55 days back from the newest round, so older rounds stop counting — with the observed range beneath it and the worst swing in its own column. Providers miss rounds: a runner is never scheduled, a build falls over. So a comparison uses only the rounds both providers ran, recomputed rather than read off the two medians above — otherwise a provider that happened to miss the busy hours would be judged on its good ones. Rounds are never discarded for being incomplete; the round nobody finished is the one worth keeping. Not run means no provider completed that workload in the rounds a row rests on: the benchmark did not run it then, so it counts against nobody. The thinnest comparison here rests on 43 shared rounds. Rounds are taken 4 times a month, each at a different hour (04:17, 09:17, 14:17 and 20:17 UTC), so working hours are in the sample as well as the quiet hour: a benchmark run only at 04:00 measures the one time nobody’s CI is busy. Rounds before October 2026 were taken four times a day. Rounds that flatter us least still count — publishing only a favourable hour would be true and indefensible at once. One exclusion, applied to everybody. A job whose queue was at least three times that provider’s own median queue is dropped — that is GitHub’s scheduler, not the machine we are measuring. It is keyed to each provider’s own behaviour rather than to a round, so it cannot remove our bad day and keep a competitor’s. In this sample it removed 1 job: GitHub-hosted (1).