Concepts

This document summarizes the basic concepts and terminology used in Screenshotbot.

A screenshot is just an image, usually a PNG, with associated metadata such as its name and debugging information.

A run is a collection of screenshots that are sent to Screenshotbot. Typically associated with a single CI build, but you may create multiple runs per CI build. For example, you might create one run for each build target in your repository.

A channel is an arbitrary tracking name associated with runs. For example, you might call your channel name example.com-ios-app.

The promoted run is the current run that’s going to be associated with the channel. You can consider it the current “golden” screenshots. The promoted run links to the previous promoted run and so on. We use this to get the history of a given screenshot. Certain runs may not get promoted, for example, if you re-run a CI build on the main branch, or if you have a run from a pull request.

A comparison is just comparing two runs. The runs could be from different channels. For example, you might compare screenshots from your staging server and your production server.

A report is a pre-generated comparison, that’s usually associated with some kind of notification. Each time any screenshots change on your main branch, we generate a report, and depending on your notification settings, we’ll notify you that the promoted screenshots have changed. We’ll also generate reports for Pull Requests/Merge Requests/Diffs, and in this case we’ll link to the report from your code review tool.

For Pull Requests (or for any non-main branch), we compare against the first known run on the merge-base to generate the report. In this case we don’t look at the promotion state of the run.

A Web Project is a project that is run by us, and not on your CI server. We generate screenshots against your publicly available website periodically, and create appropriate runs and reports. You can still test websites without our Web Project infrastructure, as long as you generate the screenshots yourself.

You might see us using the term screenshot and image. For now, you can disregard this distinction. But more precisely, an image is just the PNG file without the associated metadata such as name or commit information.

Baseline runs

For each commit in your repository, Screenshotbot stores a set of screenshots. We track screenshots across multiple branches, and we also maintain these screenshots for older commits.

In particular, you aren't expected to approve a set of "golden" images, instead we figure out the baseline commit and compare against the screenshots from that commit.

There are primarily two places where we compare commits: on Pull Requests and on the main branch. These modes behave differently, and they can be used independently of each other.

Comparison on Pull Requests

For Pull Requests (PRs) we compare against the merge-base for the PR. For example, if the PR is branched off an older version of the main branch, then we'll compare against the older screenshots. For most developers, this is all you need to know.

However, things do get a bit more complicated: e.g. how does Screenshotbot calculate the merge-base? Some CI systems have environment variables that tell us what the merge base is, but this is not reliable. For example, your build might have started as soon as you pushed your branch, i.e. before you create the PR. For this reason, we don't rely on these environment variables even when available.

Instead, Screenshotbot stores a commit graph (but we don't have access to your full Git repository). We use this graph to compute the merge base of your PR branch and the main branch on the server side, and this is the commit we use as the baseline.

If you have release branches or long running feature branches, then we instead try to find the "best" merge base against all the release branches and the main branch.

Our merge-base algorithm doesn't fully match Git's merge-base because we need to handle some edge cases. For completeness, here's the algorithm: Given a commit C, and a set of commits S corresponding the current commits of all the release and main branches, we calculate the subgraph G₁ that's reachable from S without going through C. We then calculate the subgraph G₂ that's reachable from C. The source nodes of the subgraph G₁ ∩ G₂ are our merge-bases. There could be multiple merge-bases, so we pick the newest by timestamp, but this is rare.

Handling build failures and build ordering

In some cases, the screenshots for a merge-base might not available. This might happen because the build failed completely, or it might just be that the build for the merge-base is still running.

In this case, we wait five minutes for the build to complete. If the build isn't available in that time, we'll walk back the graph and find the first commit for which a build is available and use that for comparison. Screenshotbot will also show a warning message when we do this.

You can also notify Screenshotbot of build failures to reduce this waiting period.

Main branch comparisons

On the main branch, once your PR is merged we send Slack or email notifications notifying subscribers about changes that were merged in.

We are careful about the order in which we send notifications. For instance if C₁ is an ancestor commit of C₂, then we never send a notification for C₁ after C₂.

In order to support this, the changes from multiple commits might be bundled into the same notification.

We also try our best to keep the number of commits per notification as low as possible by waiting for builds from previous commits to be available.

Release branches work similarly to main branches, with minor changes.