A passing accessibility scan is not WCAG compliance

Adding an automated accessibility scan to a Playwright suite takes a few lines with @axe-core/playwright, and it catches real problems. It’s also easy to read a green result as more than it is.

Here’s the scan from my lab repo. It loads a page, waits until it’s ready, scans for rules tagged WCAG 2.0 and 2.1 A and AA, attaches the full results to the report, and fails on any violation:

import AxeBuilder from '@axe-core/playwright';

test('home page has no detectable WCAG A/AA violations', async ({ page }, testInfo) => {
  await page.goto('https://playwright.dev/');
  await expect(page.getByRole('link', { name: 'Get started' })).toBeVisible();

  const results = await new AxeBuilder({ page })
    .withTags(['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa'])
    .analyze();

  await testInfo.attach('axe-results', {
    body: JSON.stringify(results, null, 2),
    contentType: 'application/json',
  });

  expect(results.violations).toEqual([]);
});

Make the result reviewable

Attaching the JSON matters. When the scan fails, the report shows each finding’s rule, its impact, the affected elements and a link to the fix. Without it, all you know is that an array wasn’t empty.

The results also contain an incomplete list: checks axe couldn’t decide on its own. Those aren’t passes. They need a person to look at them.

What a green scan covers

Only the rules axe can automate, on this page, in this state. Menus that weren’t opened, dialogs that weren’t triggered and pages that weren’t visited weren’t scanned. Many WCAG criteria can’t be judged automatically at all, like whether alt text actually describes the image or whether the focus order makes sense. Keyboard and screen reader testing still has to happen.

When it goes red

If the page develops an issue later, the right move is to investigate the finding, not to remove the assertion or disable the rule to get the build green. A rule turned off to make the test pass is the accessibility version of a check that can’t fail.

The full spec is in my playwright-lab repo.