Accessible on Paper, Broken in Reality: Why Real-User Testing Reveals What Compliance Audits Miss

Your product passes every WCAG audit.  

Alt text? Present.  

Colour contrast ratio? Compliant.  

Keyboard navigable? Checked.  

ARIA roles? Assigned.  

So why are disabled users still unable to use it? 

This is the accessibility gap that no automated tool will ever flag, and it's more common than most teams realise. 

The Reality Gap 

A product can tick every box on a compliance checklist and still fail real users in ways that are invisible to auditors. Consider a contact form that passes every automated check: every input has a programmatic label, keyboard navigation works end to end, and colour contrast meets minimum ratios. Yet a screen reader user still hears nothing more meaningful than “edit text” for each field because the accessible names are not descriptive or are incorrectly mapped. The form is technically valid, but functionally unclear, leaving users guessing what to enter and where. 

Or take a navigation menu that is technically keyboard accessible and structurally correct for assistive technology. Every item can be reached in order, focus states are present, and dropdowns open and close without errors. But the experience still breaks down because categories like “Services,” “Resources,” and “Support” are vague and overlapping, offering no real way to predict where content lives. When a submenu closes or a modal is dismissed, focus is technically managed, but users can still be dropped somewhere unexpected, forcing them to reorient and start again.  

Technically compliant; practically unusable. 

Why Automated Tools Only Get You So Far 

Automated testing tools like axe, WAVE, and Lighthouse are valuable, but they catch only around 30 - 40% of real accessibility issues. Everything else requires a human. 

What automation cannot detect includes: 

• Cognitive clarity: complex instructions, jargon, and confusing error messages
• Context and meaning: alt text that is technically present but meaningless (think img_final_v3.png)
• Interaction patterns: confusing form flows and illogical tab order in real-world use
• Emotional barriers: anxiety from session timeouts, sensory overload, or overly complex language 

A tool can confirm that a button has a label. It cannot tell you that pressing the button appears to do nothing, or that an error message is written in language users don't understand, or that a form field outline is too faint to see for someone with low vision even if it’s meeting minimum contrast standards. 

Making Findings Actionable 

Real-user testing is only useful if findings drive change. A good findings tracker includes the issue, its severity, a suggested fix, and the specific user impact. For example, “screen reader users cannot identify form fields because labels are not programmatically descriptive,” rated high severity with a recommendation to update accessible names and label associations. Or “keyboard users lose focus context when closing a modal, causing them to restart navigation from the top of the page,” with a fix to return focus to the triggering element. 

The goal is to translate observations into a backlog that developers can act on immediately, turning “users struggled to complete checkout due to unclear error messages” into a concrete task to improve inline validation and error announcements. Not to create a repository ghost town where insights are stored but never implemented. 

Integrating Accessibility Testing into Product Development 

Accessibility user testing should not be treated as a one-off compliance activity. It works best when it sits alongside regular auditing practices as part of ongoing product development, with each approach doing a different job. Audits confirm whether technical standards are met, while user testing confirms whether people can actually complete tasks in real conditions. A practical approach starts early, with accessibility acceptance criteria defined during planning so both compliance requirements and user needs are clear before design and build begin. 

In design, co-design should be used to embed accessibility from the outset, involving users with lived experience to shape interaction patterns, navigation, and content structure before implementation. During build, teams rely on semantic HTML and established ARIA patterns, supported by periodic automated and manual audits to identify structural issues. Alongside this, lightweight and continuous user testing validates real-world experience on high-risk journeys. Findings from audits, co-design, and user testing are combined into a single backlog and prioritised by user impact rather than source. A clear accessibility definition of done ensures work is not considered complete if it passes audit but still prevents users from completing core tasks, keeping compliance and usability aligned throughout development. 

Four Takeaways 

1. Compliance is not the same as usability. 
Passing WCAG checks does not mean people with disability and access requirements can actually use your product.  

2. Real users reveal hidden barriers. 
Only testing with people with disabilities surfaces the issues that automated tools and expert audits miss. There is no substitute. 

3. Integrate Accessibility testing into your workflow. 
One accessibility audit per year, or per release, is not enough. Make it a recurring part of how your team works. 

4. Inclusive design benefits everyone. 
The improvements that make products work for disabled users, such as clearer language, better focus management, and logical interaction flows, make products better for all users. 

--

This article is based on a presentation “Accessible on Paper, Broken in Reality” by Charlii Parker, Senior Digital Usability and Accessibility Consultant at TTC Global's Digital Accessibility Practice.