Roborazzi screenshot tests without Git LFS

Roborazzi has quickly become one of the most popular screenshot testing libraries for Android. It captures screenshots from Robolectric tests on the JVM, so there's no emulator or device to manage, and because it runs inside Robolectric you can screenshot Composables, Views, or a whole Activity after interacting with it.

Each screenshot is compared against a reference image recorded earlier, and those reference images have to live somewhere. A common answer, and the one Google's Now in Android sample uses, is to commit them to your repository, and once they start to add up, to move them into Git LFS. For a handful of screenshots that works fine. As your team writes more screenshot tests, it starts to hurt.

In this post we'll look at why Git LFS gets painful as Roborazzi test suites grow, and how to run the same Roborazzi tests with Screenshotbot without storing a single PNG in your repository.

How Roborazzi works with Git LFS

A Roborazzi test looks like this:

@RunWith(AndroidJUnit4::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
class HomeScreenTest {
  @get:Rule
  val composeTestRule = createComposeRule()

  @Test
  fun homeScreen() {
    composeTestRule.setContent { HomeScreen() }
    composeTestRule.onRoot().captureRoboImage()
  }
}

By default, Roborazzi writes screenshots to build/outputs/roborazzi, which isn't checked in. To keep reference images in the repository, teams point outputDir somewhere under src/, and the workflow looks like this:

1. A developer changes some UI, and runs ./gradlew recordRoborazziDebug on their machine.
2. Roborazzi writes the new reference images to the output directory.
3. The developer commits the images along with the code. Git LFS stores the image contents on the LFS server, and leaves a small pointer file in the repository.
4. CI runs ./gradlew verifyRoborazziDebug, which fails if any screenshot doesn't match its reference image.

To use LFS, you add a line like this to .gitattributes, matching wherever you've put your reference images:

**/src/screenshots/**/*.png filter=lfs diff=lfs merge=lfs -text

This is a reasonable default. It keeps screenshots next to the code that produced them, and it doesn't need any extra infrastructure. But every one of those four steps gets worse as the number of screenshots grows.

Where it starts to hurt

Your repository only ever grows

Every time a screenshot changes, a new copy of the image is stored forever. Change a color in your design system and you may re-record hundreds or thousands of screenshots in one commit. And because Roborazzi makes it easy to capture every Compose Preview, or the same screen across devices, themes and font scales, screenshot counts grow quickly. LFS keeps the large files out of the Git packfiles, but they still count against your storage and bandwidth quota on GitHub, GitLab or Bitbucket, and they still have to be downloaded.

We wrote about this in more detail in Can Git LFS scale?

Every CI job pays for it

A CI job that runs verifyRoborazziDebug needs the reference images, so it has to fetch them from LFS. That's extra time and bandwidth on every build, including the builds that didn't touch any UI. CI caches help, but they're one more thing to tune and keep warm.

Roborazzi's README suggests a way to avoid committing images altogether: upload the screenshots from each main branch build as GitHub Actions artifacts, and download the latest ones in pull request builds to verify against. That keeps the repository small, but now you're maintaining a homegrown screenshot store: artifacts expire after their retention period, the latest main branch build isn't necessarily the one your branch is based on, and reviewing the results means downloading the diff images, or setting up a bot to post them as pull request comments.

Recording on a laptop isn't the same as recording on CI

Robolectric Native Graphics renders on the JVM, but the results still depend on the operating system: the Roborazzi FAQ notes there are no guarantees of identical rendering across macOS, Linux and Windows. Reference images recorded on a developer's Mac can differ by a few pixels from the ones CI renders on Linux, which turns into failures that nobody can reproduce locally. The FAQ's advice is to both record and verify on CI, and teams end up adding a comparison threshold, or a CI job that re-records the screenshots and pushes a commit back to the branch, as Now in Android does.

Merge conflicts on binary files

Two branches that change the same screen will both change the same reference images. Git can't merge PNGs, so whoever merges second has to re-record and push again. On a busy repository with a shared design system, this happens a lot.

Reviewing changes is awkward

When a pull request changes fifty screenshots, the reviewer has to click through fifty image diffs in the code review tool, one file at a time, mixed in with the code changes. It's easy to miss the one screenshot that changed for the wrong reason.

A different approach: don't store screenshots in Git

Reference images are build artifacts. They're generated from your code, so they don't need to be stored next to it in version control: they only need to be stored somewhere you can compare against, and tied to the commit that produced them.

That's what Screenshotbot does. Instead of recording on a developer's machine and committing the result:

1. CI runs Roborazzi in record mode on every commit, for both your main branch and your pull requests.
2. The screenshots are uploaded to Screenshotbot, which stores them against the commit.
3. For a pull request, Screenshotbot compares its screenshots against the ones from the merge base on your main branch, and reports the changes as a build status on the pull request.
4. A reviewer looks through the changes side by side, with the differences highlighted, and accepts or rejects them.

Nothing is committed, so the repository stops growing, there's nothing to fetch from LFS on CI, and there are no merge conflicts on images. And because every screenshot is recorded on CI, they're all rendered in the same environment.

This scales to tens of thousands of screenshots, which is where the LFS approach tends to fall over.

Setting up Roborazzi with Screenshotbot

You don't need to change your Roborazzi tests at all. The Screenshotbot Gradle plugin detects Roborazzi, reads its output directory, and adds the tasks you need.

First, add the plugin to your root build.gradle.kts:

plugins {
  id("io.screenshotbot.plugin") version "1.29.13"
}

Then apply it in each module that has Roborazzi tests:

apply(plugin = "io.screenshotbot.plugin")

Create an API key from your Screenshotbot account, and add it to your CI as the SCREENSHOTBOT_API_KEY and SCREENSHOTBOT_API_SECRET environment variables. Then replace verifyRoborazziDebug in your CI with:

./gradlew recordRoborazziDebugScreenshotbotCI

That's it. You'll see the screenshot changes on your next pull request. You may need to adjust the task name for your build flavor. The Android documentation has the full details.

Moving your existing screenshots out of LFS

Once Screenshotbot is running on CI, you can delete the reference images from your repository. Remove the LFS rule from .gitattributes, and remove the outputDir setting from your roborazzi block, so screenshots go back to build/outputs/roborazzi where they won't be committed. Then:

git rm -r --cached '*/src/screenshots/*'
git commit -m "Move Roborazzi screenshots to Screenshotbot"

Older commits still reference the old images in LFS, so keep them around until you no longer need to build those commits.

Running screenshot tests locally

Developers can still check their changes before pushing. After a one-time ./gradlew :installScreenshotbot to set up an API key, record a baseline and then verify your changes against it:

./gradlew :app:recordRoborazziDebugScreenshotbot
# make your changes
./gradlew :app:verifyRoborazziDebugScreenshotbot

See local runs for how this works.

Wrapping up

Git LFS is a fine way to start with Roborazzi. But reference images behave differently from source code: they change in bulk, they can't be merged, and they need a visual review. Once you have more than a few hundred of them, it's worth moving them out of Git and into a tool built for them.

Screenshotbot works with Roborazzi, Paparazzi, Compose Preview Screenshot Tests and the other popular Android screenshot testing libraries, as well as iOS, Flutter and web. Try it for free, or get in touch if you'd like help migrating a large Roborazzi suite.