The benchmark
We planted 16 defects. Off-the-shelf axe-core finds 10. We find 16.
Automated accessibility testing detects a portion of what is wrong on a page. Everybody in this category knows it and almost nobody publishes a number, because the number sounds bad. Here are both of ours, how they are measured, what still gets past them, and the fixture so you can run your own tool against the same page and disagree with us.
63%
axe-core on its own. 10 of 16 planted defects. This is the number worth knowing before buying any scanner, including this one — the engine underneath is open source and free.
100%
With Covrane’s own rules. 16 of 16, of which 3 are reported for a person to confirm rather than asserted as failures. Across 2 fixtures built from different markup: 32 of 32. Measured 2026-09-21 against axe-core 4.13.0, re-measured on every build.
Method
How the number is produced.
- Plant the defects. One HTML page carries 16 accessibility defects, each recorded in a manifest with an id, a description, and whether we expect any automated tool to be able to see it at all.
- Scan it with the real scanner. The same code path the product runs against a client site: axe-core 4.13.0, requesting the tags wcag2a, wcag2aa, wcag21a, wcag21aa, wcag22aa, best-practice.
- Compare against the manifest, not against a hope. Each planted defect is matched to the rule that should have fired. A defect we expected to catch and did not is a failure, and it is recorded by name.
- Write the number to a file nobody may edit by hand. The result is written to CATCH_RATE.md on every run. This page is built from that file and the manifest, and the build fails if the two disagree.
The fixture is never tuned to improve the number. On 2026-09-19 this scanner scored 16 of 16 here — a perfect run. Two of the rules that produced it were then measured against 39 real public sites, were wrong almost every time they fired, and were deleted. That took the number back down to 16 of 16, where it stands. A scanner advertising 100% on its own fixture is the thing this page exists to argue against, and we are not going to make an exception for ourselves.
The ceiling
Nothing on this page is missed, and that is a fact about this page.
Every defect the fixture plants, at the levels we scan, is now reported by something. That sentence is worth distrusting, so here is what is behind it. The fixture has 17 defects in it because we put 17 defects in it. A real site has whole classes of problem nobody here thought to plant, and a fixture cannot tell you about those. A scanner that reaches the top of a test it wrote for itself has learned the test.
The number that means something is on pages nobody built to be measured. The two rules that report the things axe-core cannot see were each pointed at dozens of real sites before they shipped, and the results, including every false positive, are in the repository rather than summarised here. One earlier rule fired 116 times across 39 sites and was right at most twice; it was deleted rather than tuned. Another was wrong 23 times out of 24 and was deleted too, then rebuilt to conditions written down in advance and measured over 648 layout containers on 55 sites before it was allowed back.
This is why every Covrane report ends on a limitations page saying so, and why the sample report carries it too. It is also why the honest version of this product ends with a person, not a scan.
Not scored
The 1 planted on purpose and left out of the number.
This product scans WCAG Level A and AA. The fixture plants 17 defects and 1 of them fails only a Level AAA criterion, so scoring it would measure the scanner against a bar it does not claim to clear. It stays in the fixture you can download, and it stays named here, because a denominator you cannot audit is not a measurement.
- P14<a>Click here</a> as the only link text to a PDF. Fails 2.4.9 Link Purpose (Link Only), Level AAA.
It was counted as a miss until 20 September 2026. The rule that used to report it (“Click here”, “Read more”) fired 116 times across 39 real sites and was right at most twice, because WCAG 2.4.4 is Link Purpose In Context and is satisfied when the surrounding sentence supplies the purpose. The rule could see the link text but not the sentence around it, so it was asserting the half of the criterion it could not evaluate. The fixture plants that link inside a paragraph that does supply the context, which means the page passes 2.4.4 and fails 2.4.9 instead. Level AAA. Out of scope. The rule was not brought back; only the label was corrected.
What we add
The 6 axe-core misses and our own rules catch.
These are the reason the second number on this page is higher than the first. Each one needs something a static analyser does not do: pressing keys and seeing whether focus escapes, reading the alt text rather than checking it exists, looking for a captions track. 3 of the 6 are reported for a person to confirm rather than asserted as a failure, and the report labels them that way — a rule that cannot be certain should say so rather than round up.
- P11<img src="ceo.jpg" alt="image"> — alt attribute present but meaningless covrane-alt-text-uninformative
- P12A modal that traps keyboard focus with no Escape handler and no way out covrane-dialog-keyboard-trap
- P13Visual order contradicting DOM order via CSS order covrane-visual-order-mismatch
- P15<video> with no <track kind="captions"> covrane-video-caption-missing
- P16A form error indicated only by a red border — no text, no aria-describedby covrane-error-colour-only
- P17A decorative image with a long, noisy alt describing the file name covrane-alt-text-uninformative
Every one of them is also scored against a second fixture that plants the same defect classes in different markup, so a rule fitted to the first page shows up as a rule that only works on the first page. Both fixtures score 16 of 16, with zero false positives against the clean control pages.
The floor
The 10 axe-core detects on its own.
Every one of these is caught on every run, with zero false positives against a clean control page. 2 of them are tagged best practice by axe rather than as a WCAG failure, and the report says which is which rather than blending them into one count.
- P01<img src="chart.png"> with no alt attribute image-alt
- P02<input type="text" id="email"> with no associated label label
- P03A link containing only an unlabelled image link-name
- P04<button> containing only an icon <span>, no accessible name button-name
- P05<html> with no lang attribute html-has-lang
- P06Body text #949494 on #ffffff at 14px (about 2.8:1) color-contrast
- P07<iframe> with no title attribute frame-title
- P08<ul> with a <div> as a direct child list
- P09tabindex="5" on a link. Tagged best-practice by axe, not a WCAG failure. tabindex · best practice
- P10Heading jump from <h1> straight to <h4>. Tagged best-practice by axe, not a WCAG failure. heading-order · best practice
Check it
Run your own tool against the same page.
The fixture and the manifest are above. Point whatever you use at the page, compare what it reports against the 16 entries in the manifest, and see what number you get. If your tool catches something ours does not, we want to know: write to contact@covrane.com and we will re-measure and change this page.
The engine underneath is axe-core, which is open source and free, and it does most of the work: 10 of the 16 we find here are its. Our own rules add 6 more by driving the browser rather than reading the DOM. What the product adds beyond detection is the crawl, the grouping, the dated record, and a report you can hand to somebody. The detection is the commodity half, and pretending otherwise invites exactly the question you should be asking.
Measured 2026-09-21 · axe-core 4.13.0 · page built 21 September 2026