all notes

· JUnit · GitLab CI · Reporting

Failure screenshots in GitLab test reports

GitLab can show a failed test's screenshot right in the pipeline report, if the JUnit XML points at it the right way.

GitLab's test report shows a failed test's message, but the screenshot usually sits in the job artifacts where nobody opens it. GitLab will display it inline if the testcase's system-out holds an attachment line with a path relative to the project directory.

<testcase name="checkout [web]" classname="tests/checkout.e2e.ts">
  <failure message="cart is empty" type="ASSERTION_FAILED">...</failure>
  <system-out>[[ATTACHMENT|.e2e/results/checkout/attempt-1/screenshots/001-failure.png]]</system-out>
</testcase>

I added this to the e2e framework's JUnit reporter. The interesting part was the path, not the XML:

  • The base is $CI_PROJECT_DIR when set, else the project root. A monorepo whose tests live in apps/web needs the apps/web/ prefix, or GitLab finds nothing.
  • Paths go through realpath on both sides, so a symlinked checkout or artifacts folder still resolves.
  • A screenshot outside the base gets no link: GitLab could not serve it anyway.
  • Failed, timed-out, flaky, and test.fail cases all link their screenshot, through one helper that writes every system-out.

Mutation testing showed where the unit tests were thin: dropping the realpath on one side still passed until I added a test where the artifacts folder itself is a symlink.