Playtest bots in CI: catch performance regressions before players do

Have a bot play the same seeded run of your game on the old code and on the new code, on the same machine, and time the frames both times. If the new build is slower by more than a margin you set, fail the pull request. That's an automated playtest, and it catches the stutter unit tests can't see.

The method works with any way of measuring frames. This guide shows it with plain code first, then with glitchless-playtest for web games and the measure flag in the Unity, Godot, Unreal and GameMaker SDKs.

Why unit tests miss stutter

A unit test asks whether the code gives the right answer. Stutter is about how long a frame took, and the code behind it is usually correct: a shader that compiles the first time an effect appears, a burst of objects spawned in one frame, garbage the engine stops to collect, a level that loads on the main thread, a leak that only hurts after five minutes. Each one passes every test you have.

They show up only when the game runs: real rendering, real frames, minutes of play in a row. So the test has to play the game and time it.

A fast machine hides some of it, too. A 60 Hz screen shows a new frame every 16.7 ms, so a build that needs 9 ms per frame and one that needs 14 ms both look like a smooth 60 fps on your desk. On a player's slower phone, the second one drops frames. That's why a good check also reads CPU time per frame and draw calls, which grow even when the frame rate doesn't move.

What a playtest bot does

A playtest bot is the game playing itself. It starts a level, plays it through the same input code a player uses, and records numbers as it goes: frame times, CPU time per frame, draw calls, hitches, errors, deaths, and whether it finished.

There are two kinds. A scripted bot follows a fixed path: load level 3, run right, trigger the boss. It's good for one known heavy moment. An autoplay bot plays with simple rules and a skill setting, so it covers more of the game. Either way, every random choice comes from a seed, so the same seed plays the same run on both builds and the only thing that changes is the code.

An autoplay mode needs five things:

• A switch on the command line or in the URL: --autoplay --seed=7, or ?autoplay=1&seed=7&level=1.
• A seeded random for every random choice in the game, not only the bot's: spawns, loot, enemy decisions.
• The player's code path. The bot sets the same input a controller would instead of moving the character directly, so it runs the code players run.
• A clear end: a flag set when the level is done or the run is over, and a time cap in the runner in case it never is.
• A bot label on its data, so its runs never count as players in your analytics.

// autoplay.js: ?autoplay=1&seed=7&skill=0.9&level=1 makes the game play itself, the same way every time.
const params = new URLSearchParams(location.search);
export const autoplay = params.get('autoplay') === '1';
export const startLevel = Number(params.get('level') ?? 1);
const skill = Number(params.get('skill') ?? 0.9);

// A seeded random (mulberry32). Use it for every random choice the game makes, not only the bot's,
// so spawns, loot and enemy decisions repeat exactly on both builds.
let state = Number(params.get('seed') ?? 1) >>> 0;
export function random() {
  state = (state + 0x6d2b79f5) >>> 0;
  let t = state;
  t = Math.imul(t ^ (t >>> 15), t | 1);
  t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
  return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
}

// Called each frame in place of reading the keyboard, so the bot runs the same code a player does.
export function botInput(player, level) {
  const gap = level.nextGap(player.x);
  return { right: true, jump: !!gap && gap.x - player.x < 2 && random() < skill };
}

// Tell the runner the run is over: the level is done, the bot lost, or the game hit an error.
export function runOver() {
  window.__GLITCHLESS_PLAYTEST_DONE__ = true;
}

window.__GLITCHLESS_PLAYTEST_DONE__ is the flag glitchless-playtest waits for; any runner can poll the same flag. In Unity, Godot or Unreal, read the same switches from the command line.

Save a baseline and compare against it

Play the bot on the code before your change and keep its numbers. That's the baseline. Play it again on the change and compare metric by metric. In CI the baseline is the target branch: build it and play it in the same job as the pull request, so both runs share a machine.

Keep percentiles, not averages. One 200 ms freeze in a minute of smooth 60 fps play still averages 59.8 fps, but the player felt every bit of it. Record the 50th, 95th and 99th percentile frame times, the worst frame, and the share of frames that missed the screen's frame time.

If you measure with your own code, the script below does the comparison. Each run writes its numbers to a JSON file; the script takes the median of each metric over several runs, compares the two sides, and exits with code 1 on a regression, which fails the CI step. It also reads the line the Glitchless engine SDKs print, shown further down.

// compare.mjs: fails the build when the change plays slower than the baseline.
//   node compare.mjs base-1.json base-2.json base-3.json -- pr-1.json pr-2.json pr-3.json
// Each file is one run's numbers: plain JSON, or the {"glitchless_measure": {...}} line an engine build prints.
// A metric fails only when the change's median is worse than the baseline's by BOTH margins.
import fs from 'node:fs';

const RULES = {
  pct_frames_over_budget: { pct: 50, abs: 2, fail: true }, // % of frames slower than the screen's frame time
  'frame_ms.p95': { pct: 15, abs: 2, fail: false },        // warns only: vsync makes it jump between steps
  render_cpu_ms: { pct: 30, abs: 1, fail: true },
  draw_calls: { pct: 25, abs: 50, fail: true },
};

function load(file) {
  const text = fs.readFileSync(file, 'utf8');
  const line = text.split('\n').find((l) => l.startsWith('{"glitchless_measure"'));
  const json = JSON.parse(line ?? text);
  return json.glitchless_measure ?? json;
}
const value = (run, key) => key.split('.').reduce((v, k) => v?.[k], run);
const median = (xs) => xs.sort((a, b) => a - b)[Math.floor(xs.length / 2)];

const args = process.argv.slice(2);
const cut = args.indexOf('--');
const base = args.slice(0, cut).map(load);
const pr = args.slice(cut + 1).map(load);

let failed = false;
for (const [key, rule] of Object.entries(RULES)) {
  const a = median(base.map((r) => value(r, key)).filter((v) => v != null));
  const b = median(pr.map((r) => value(r, key)).filter((v) => v != null));
  if (a == null || b == null) continue;
  const worse = b - a >= rule.abs && ((b - a) / (a || 1)) * 100 >= rule.pct;
  if (worse && rule.fail) failed = true;
  console.log(`${worse ? (rule.fail ? 'FAIL' : 'warn') : 'ok  '}  ${key}: ${a} -> ${b}`);
}
process.exit(failed ? 1 : 0);

With glitchless-playtest on a web game, the same steps are two commands: npx glitchless-playtest --save baseline before you change the code, and npx glitchless-playtest after, which compares with baseline and exits with code 1 on a regression. Saved runs go in .glitchless/playtest/, so add that folder to .gitignore.

Set pass and fail margins

Give each metric two margins and call it worse only when it clears both. The relative margin keeps normal noise on big numbers from failing a build. The absolute margin keeps big percentages on tiny numbers from failing it: render time going from 0.3 to 0.4 ms is 33% worse and means nothing. Then decide which metrics block a merge and which only warn.

These are the margins glitchless-playtest starts with, and a fair starting point for any game:

The 95th percentile only warns because of vsync. On a 60 Hz screen, frame times bunch up at 16.7 and 33.3 ms, so the 95th percentile jumps from one to the other when about 5% of frames are slow. The share of slow frames moves smoothly, so that's the one that fails a build.

To tune margins for your game, play the same build five times and see how far each number moves between identical runs. Set each margin above that spread.

Run it on every pull request in GitHub Actions

For a web game, the job checks out the pull request and its target branch, builds both, serves both, has the bot play each in a headless browser on the same runner, posts the result on the pull request and fails the check on a regression.

The workflow below does that with glitchless-playtest, which drives a browser through Playwright and reads the numbers back from a Glitchless server running in Docker beside it. Before you use it, install both in your game (npm install -D glitchless-playtest playwright), make your autoplay mode read autoplay, seed, env and intake from the URL (the runner sets intake to the server's address), and list your scenarios in glitchless.playtest.json. It assumes a Vite game whose npm run build writes the build; with another bundler, serve its output folder with any static server.

The server accepts data sent with the client token gl_pub_dev. If your build sends another, start the container with -e GLITCHLESS_CLIENT_TOKENS=<your token>.

# .github/workflows/playtest.yml
# Builds the game from the target branch and from the pull request, has the autoplay bot play both
# on the same runner, posts the comparison on the pull request, and fails the check on a regression.
name: Playtest

on:
  pull_request:

permissions:
  contents: read
  pull-requests: write

jobs:
  playtest:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - name: Check out the pull request
        uses: actions/checkout@v4
        with:
          path: pr

      - name: Check out the target branch
        uses: actions/checkout@v4
        with:
          ref: ${{ github.base_ref }}
          path: base

      - uses: actions/setup-node@v4
        with:
          node-version: 22

      - name: Install and build both versions
        run: |
          (cd pr && npm ci && npm run build)
          (cd base && npm ci && npm run build)
          (cd pr && npx playwright install --with-deps chromium)

      - name: Start a Glitchless server and serve both builds
        run: |
          docker run -d -p 7777:7777 ghcr.io/glitchless-gg/glitchless
          (cd pr && npx vite preview --port 5173 --strictPort > /dev/null 2>&1 &)
          (cd base && npx vite preview --port 5174 --strictPort > /dev/null 2>&1 &)
          npx -y wait-on http://localhost:7777/v1/health http://localhost:5173 http://localhost:5174

      - name: Play the target branch, then the pull request
        id: playtest
        working-directory: pr
        run: |
          set +e
          npx glitchless-playtest --channel chromium --baseline-url http://localhost:5174/ --game http://localhost:5173/ > ${{ runner.temp }}/playtest.md
          echo "exit=$?" >> "$GITHUB_OUTPUT"
          cat ${{ runner.temp }}/playtest.md

      - name: Post the report on the pull request
        if: always()
        env:
          GH_TOKEN: ${{ github.token }}
        run: gh pr comment ${{ github.event.pull_request.number }} --repo ${{ github.repository }} --body-file ${{ runner.temp }}/playtest.md

      - name: Fail on a regression
        if: steps.playtest.outputs.exit != '0'
        run: exit 1

Hosted runners have no GPU, so the browser draws in software and frame times come out slow. Both builds still run on the same machine, so draw calls, render CPU, errors and crashes compare reliably. To fail on frame times as well, run the job on a self-hosted runner with a GPU (runs-on: [self-hosted, linux, x64, gpu], where gpu is a label you give it) or on one of GitHub's GPU larger runners, which come with the Team and Enterprise Cloud plans. GitHub advises against self-hosted runners on public repositories, since anyone can open a pull request that runs on them.

On a pull request from a fork, the workflow's token is read-only, so the comment step can't post. The check itself still passes or fails.

The workflow is adapted from the example in the Glitchless repo, which playtests the Glitchless demo game on every pull request.

Unity, Godot, Unreal and GameMaker builds

An engine game works the same way with a build in place of a browser. Build the target branch and the pull request, run each build with its autoplay switch and a measure flag, write the numbers to a file, and compare them with the script above.

The Glitchless SDKs for these engines have that flag built in. It waits through a warm-up, measures for the seconds you give it, prints one line of JSON ({"glitchless_measure": {...}}) with frame-time percentiles, the share of slow frames, render CPU, draw calls, hitches with their causes and new errors, writes it to a file, and quits. --autoplay is your game's own switch; the SDKs read it only to mark the session as a bot's.

# ci/measure.sh: play two Unity builds three times each, taking turns, then compare them.
# Run it on a machine with a GPU and a display, after building the target branch into build/base
# and the pull request into build/pr. --autoplay is the game's own switch.
set -e
for i in 1 2 3; do
  for side in base pr; do
    "./build/$side/MyGame.x86_64" --autoplay --glitchless-warmup=5 --glitchless-measure=30 \
      --glitchless-out="$PWD/$side-$i.json"
  done
done
node ci/compare.mjs base-1.json base-2.json base-3.json -- pr-1.json pr-2.json pr-3.json

In GitHub Actions, run the script in a job on your GPU runner after your usual build steps. Frame times from a build that draws nothing mean little: Godot's --headless, for one, renders no frames and runs them uncapped, so measure on a machine with a GPU and a display.

Keep the numbers stable

A check that fails at random gets ignored. These keep the noise down:

• Same machine, same job. Build and play the baseline and the change on the same runner, one after the other. Never compare with numbers from your laptop or from last week's runner: a different GPU plays a different game.
• Warm up first. The first seconds compile shaders, load assets and fill caches. Skip them (--glitchless-warmup=5 in the engine SDKs) unless loading is what you're testing, and then test it on purpose.
• Several runs, taking turns. Two runs of the same build never match exactly. Play each side three times or more and compare medians, and alternate between base and change, so a machine that heats up or gets busy slows both.
• Percentiles, not averages. The 95th and 99th percentiles, the worst frame and the share of slow frames show hitches. Average FPS hides them.
• Numbers vsync can't hide. On a fast machine both builds may hit the screen's frame rate. Render CPU, update CPU and draw calls per frame still grow when the code gets slower.
• One bot at a time. Browsers or builds running side by side fight over the GPU. glitchless-playtest plays its scenarios one after another for this reason.
• Same settings every run. Fix the window size, the quality preset and the seed, and keep other jobs off the runner while it plays.

When the check fails

1. Read what got worse, and where. A regression on one level points at what changed there. A new error comes with its message and location, and a new crash is always real.
2. Play it again to rule out noise. If the same metric fails again, it's the code. If it flips between runs, add runs or widen that margin; don't rerun until it happens to pass.
3. Replay it locally with the same seed and scenario. With glitchless-playtest, --headed shows the browser windows and --quick plays only the first scenario.
4. Find the cause with a profiler on that moment: Chrome's Performance panel, the Unity Profiler, Godot's profiler or Unreal Insights. Read the diff for what that level runs: new effects, more spawns, allocations every frame, assets loaded mid-play.
5. Fix it and push again. If the cost is on purpose (a new effect worth a millisecond, for example), explain it in the pull request and merge. The target branch is the baseline, so the next pull request is measured against the new numbers.

Guides to the usual causes: Three.js, Unity, Godot and Unreal Engine 5 shader stutter.

Playtest bots with Glitchless

Glitchless is open source. Its runner, glitchless-playtest, plays your web game's autoplay scenarios in a real browser and compares slow frames, render and update CPU, draw calls, hitches, errors, crashes, deaths and level completion with the baseline, for the whole run and level by level, then writes the report in Markdown for the pull request. Sessions the game marks as a bot's (sessionType: 'synthetic') stay out of your players' numbers.

Your coding agent can run the same check while it works. The MCP server has a glitchless_playtest tool, so the agent saves a baseline before it changes code and treats a FAIL as unfinished work. After you ship, the same SDK measures real players, and the agent can compare the new version with the old one: debugging with real player data covers that part.

Before you change anything, run glitchless_playtest with save_as "baseline". Then make the change and run glitchless_playtest again. If it says FAIL, fix the regressions and run it again until it passes, then show me the comparison table.

Frequently asked questions

Can I run playtests on GitHub-hosted runners without a GPU?

Yes, for some numbers. The browser draws in software there, so frame times are slow and noisy, but both builds run on the same machine, so draw calls, render CPU, errors and crashes still compare reliably. To fail a build on frame times, use a runner with a GPU.

Do I need Glitchless to do this?

No. You need an autoplay mode with a seed, a way to record frame times, and a comparison like the script above. glitchless-playtest saves you writing the runner, the level-by-level comparison and the pull-request report, and the engine SDKs give you the measure flag.

Why not fail the build when FPS drops under 60?

Because that number depends on the machine more than on the code. A fixed line passes on a fast runner and fails on a slow one with the same build. Comparing with a baseline from the same machine measures only what your change did.

How long does a playtest take?

As long as the scenarios you choose. In glitchless-playtest each run stops at its time cap (max_s, 90 seconds unless you set it) or as soon as the game sets its done flag, and in CI it plays every scenario on both builds. --quick plays only the first scenario, for a faster check.

Can it catch memory leaks?

Yes, if the run loads several levels. Watch counts that should come back down after each level: geometries and textures in a web game, objects and nodes in Godot. glitchless-playtest warns when geometry grows with each level load, and the Godot and Unreal measure output includes memory and how much it changed over the run.

Sources

• GitHub Docs: Contexts reference (github.base_ref)
• GitHub Docs: Events that trigger workflows (pull requests from forks)
• GitHub Docs: GitHub-hosted runners
• GitHub Docs: Using self-hosted runners in a workflow
• GitHub Docs: Secure use reference (self-hosted runners)
• GitHub CLI manual: gh pr comment
• Playwright: Browsers
• glitchless-playtest README
• Glitchless example workflow: playtest-github-action.yml

Something wrong or out of date? Email support@glitchless.gg and we'll fix it. Glitchless isn't affiliated with GitHub, Microsoft, Google, Unity Technologies, the Godot Foundation, Epic Games or YoYo Games; product names are their owners' trademarks.