Why your screenshot tests pass locally but fail on CI

Ask a team why they gave up on screenshot tests, and you'll hear the same answer again and again: they were flaky. A test passes on a developer's laptop, fails on CI, and when you open the two images side by side, they look identical. After a few weeks of that, nobody trusts the tests, and they get deleted.

But in our experience, screenshot tests are some of the most stable tests you can write. Run the same screenshot test a hundred times on the same machine, and you'll get the same image, pixel for pixel, a hundred times. The tests aren't flaky. The setup is.

(Everything here applies whatever you call them: iOS developers call them snapshot tests, Flutter calls them golden tests, and on the web they're visual regression tests.)

Record on one machine, verify on another

Almost every screenshot testing library works the same way. You run the tests in record mode to save reference images, commit them, and then run the tests in verify mode to compare new screenshots against those references.

In practice, those two steps happen on different machines:

• Record happens on a developer's MacBook, when they make a UI change.
• Verify happens on CI, which is usually a Linux machine.

Rendering isn't identical across platforms. Font rasterization, anti-aliasing, gradients and shadows can all come out a few pixels differently on macOS and Linux. It's not even just Mac versus Linux: we've seen rendering differences between M1 and M4 Macs, and between macOS versions. iOS snapshots additionally depend on the simulator and Xcode version.

So the same code produces slightly different images on the two machines, and the test fails. It's not random. It will fail the same way every time. But to the developer looking at two images that look the same, it feels flaky, and that's what costs you the team's confidence in the tests.

Why thresholds don't fix it

The usual response is to add a threshold: treat two images as the same if, say, 99.9% of the pixels match. Most libraries support this, and it does make the failures go away. But it brings its own problems.

What's a good threshold?

Is it 99%? 99.9%? 99.99%? There's no right answer, so teams pick a number, see a failure, and loosen it. Then they loosen it again.

Thresholds hide real changes

Imagine a full-screen screenshot where a card's rounded corners accidentally became square. That's a real regression that a designer would notice immediately, but it only changes a few hundred pixels out of millions. A threshold loose enough to absorb rendering noise is often loose enough to let that through.

They don't stop the support questions

Even with a threshold, you'll eventually hit an image that crosses it, and someone asks: why did my test fail, the screenshots look the same to me? Look at the issues for any popular screenshot testing library, and you'll find plenty of them.

The fix: record on CI

If the problem is that record and verify run on different machines, the fix is to run them on the same one. Record your screenshots on CI, for every commit, on your main branch and on your pull requests. Then compare the screenshots from a pull request against the ones from its base commit on the main branch.

Both sets of images come from the same CI machine image, so they match pixel for pixel unless the UI actually changed. You can compare images exactly, with no threshold at all. (Screenshotbot goes further: by default, two screenshots only count as the same if their files are identical.)

Recording on CI fixes more than flakiness:

• Nobody needs to learn how to record. Without it, everyone who touches UI needs to know how to re-record screenshots, including design system engineers whose change breaks screenshots on a product team's screen.
• Less friction. UI developers change UI all day. Not having to re-record and commit screenshots every time makes them much more willing to add screenshot tests early, instead of waiting for a screen to become stable.
• No screenshots in Git. The reference images live with your CI results, not in your repository, so you don't need Git LFS. We've written about doing this for Paparazzi, Roborazzi, swift-snapshot-testing and Flutter golden tests.

The catch is that you need somewhere to store the screenshots from every CI build, and a way to find the right ones to compare against and review the changes. You can build that yourself, or use a tool like Screenshotbot that's built for it.

Some tests really are flaky

None of this means every screenshot test is deterministic. Some really are flaky: a screen with an animation captured mid-way, a timestamp or a relative date, content that's still loading, or random test data. Recording on CI won't fix those.

The best fix is usually in the test: mock the clock, disable animations, wait for loading to finish, and use fixed test data. When that isn't practical, you can mask the part of the screenshot that changes, so it isn't compared.

And if a flaky test slips through, disable it. One noisy test can train the whole team to ignore screenshot failures, which is worse than missing the one regression it would have caught. You don't have to catch every regression. Prioritize developer productivity, and the team will keep writing screenshot tests.

When you can't see the difference

Sometimes a screenshot does change and you can't tell why by looking at it. Don't squint at two images side by side: use a tool that highlights the pixels that changed. Screenshotbot's comparison view does this, and if you want something local, Pixel-diff is our open source desktop app for comparing images.

Watch the talk

We talked about this, and the other reasons teams give up on screenshot tests, at next app devCon 2026:

The slides and links to everything mentioned in the talk are on the resources page.

Wrapping up

When a screenshot test fails on CI and the images look the same, the test usually isn't flaky. It was recorded on a different machine. Thresholds paper over that and hide real changes along the way. Record on CI instead, compare exactly, and save your attention for the few tests that really are flaky.

Screenshotbot records and compares your screenshots on CI for Android, iOS, Flutter and web. Try it for free, or get in touch if your team has given up on screenshot tests and wants to try again.