EU Runs World’s Largest Accessibility Test

While many of us were getting ready for the 2021 holidays, governments throughout Europe were racing to put the final touches on the first reports for the EU’s Web Accessibility Directive. Under the EU Web Accessibility Directive issued in October 2016, all government agencies throughout the EU were required to ensure their websites and apps are accessible to people with disabilities by June 2021.

Every country in the EU has to monitor the accessibility of its digital assets annually and report its findings to the EU every three years. December 23rd, 2021 was the end of the first reporting period. My colleagues and I in the Deque’s European office have been reading the reports as they become available, and some interesting trends are starting to emerge.

European Government Websites Not Fully Accessible

All 27 EU members conducted research. The United Kingdom produced a report as well, though there was no formal obligation to the EU. The UK ceased to be a member state of the European Union on the 31st of January 2020.

Out of the more than 800 in-depth website audits that were done during the monitor, only four (4) were reported to be fully accessible (i.e. fully complying with EN 301 549 and WCAG 2.1 Level A + AA). Mobile applications turned out slightly better, with eight (8) out of 286 tested apps being fully accessible.

That’s not to say web accessibility is in a terrible state across Europe. Having a fully accessible website is sort of like having a dust-free house. If you work hard at it, it is something you might achieve, but keeping it that way is almost impossible. The real question is how common and how substantial the accessibility problems are. This is difficult to say, as there is no standard way to measure this. The EU does not prescribe any particular metric, and so quite a few countries didn’t try to quantify their findings. Unfortunately, this means that in the next report, it will be difficult to judge if web accessibility in those countries improved over time.

Kudos to the countries that did attempt to quantify their findings. While we can’t make direct comparisons between countries because they don’t use the same metrics, three years from now when they report again, we will be able to see to what extent accessibility has improved across the board. One example that is public is www.toegankelijkheidsverklaring.nl, a register of Dutch Government websites, ranked A to E.

Accessibility Statements Are On The Rise

Perhaps a little more telling are the accessibility statements that the Web Accessibility Directive requires. An accessibility statement is a web page that includes information on the current state of accessibility for that website. Every EU government website and app is required to publish a public and easy to locate accessibility statement. This statement must also include information on how someone can report accessibility issues.

Of the 25 monitoring reports, we found that 13 countries reported how many accessibility statements they found when sampling government agency websites within a given country. The differences here are stark, ranging from as high as 96% to as low as 12%, with an average of around 50%. This suggests a significant difference in awareness of accessibility requirements in different countries. The countries reporting the highest levels of accessibility statements are the following:

1. Denmark (96%)
2. United Kingdom (90%)
3. Netherlands (78%)
4. Slovakia (54%)
5. Austria (54%)

It should be mentioned that the way this percentage is measured varies. Mostly, these percentages come from checking how many monitored websites had an accessibility statement. However, which websites get monitored is not truly random. Instead, the monitored websites have to strike a balance between organizations of different sizes and regions and prioritize websites that people with disabilities deem most important. In some cases, the list of websites used for monitoring is in part based on an index of websites with an accessibility statement.

Consistent Test Results

One thing EU countries seem to love to do is to report how common certain accessibility problems are. This is something Deque reported on when studying how many accessibility issues can be tested automatically (the answer is 57%). This gives some interesting insights into how varied accessibility testing can be, even using the same accessibility standard. Some key findings:

1. The European standard EN 301 549 (v2.1.2) includes a good number of accessibility requirements that are not part of the Web Content Accessibility Guidelines (WCAG). For example, section 11.7 requires apps to support user preferences configurable through the operating system. It is therefore notable that more than half of the countries did not consider the additional EN requirements in their monitoring efforts.
2. While several countries list WCAG Success Criterion 4.1.1: Parsing as the most common accessibility problem, others don’t even have it in their top 10. This is not because some countries do this significantly better than others, but because they test in different ways. While it’s a good thing that countries like Denmark, France, and Sweden have a test methodology, inconsistencies between them creates problems for organizations that provide services in multiple EU countries.
3. There is a surprising diversity of accessibility test tools in use in this EU monitor, with almost 50 different tools. This speaks to the broad diversity and availability of accessibility testing tools. It is encouraging to see quite a few of those tools support Accessibility Conformance Test Rules, an effort largely funded by the EU to encourage consistent and high-quality accessibility testing. Of the simplified monitoring done, 60% of the tools used were based on WCAG Test Rules.

Other Highlights

United Kingdom: Many things look great in this report, and if you plan to read any at all, we would suggest you read this report published by the UK government. What we loved was how the UK communicated their results and retested the sites 3 months later to see how much they had progressed. This showed that sharing these results greatly improved these sites. Many EU countries did not appear to have shared their findings with the monitored organizations. We hope the UK convinces them to do differently. We can only applaud the UK for setting a strong example as a former EU member. 

Ireland: The Irish report included qualitative summaries for each in-depth audit they did, which they shared with the organizations whose websites and apps were monitored. These summaries give insight that is hard to gauge from the perspective of other countries. Several countries noted in their report that just sharing raw results with organizations led to a lot of questions. An approach similar to the Irish could help address this.

Netherlands: The Dutch did something unique: require accessibility reports and plans for improvement to be included in the accessibility statements. This gives insight into what efforts organizations have taken. The Dutch government created a compliance level A through E– A being fully conforming to EN 301 549, B is non-conforming but with clear knowledge of what is left to address and plans to remediate it, all the way through E meaning no information at all. However, the monitor found that almost all sites with compliance level A still had some accessibility issues.

Cyprus, France, and Portugal: These countries are somewhat late in publishing their reports, so their results were not used for this post. This post may be republished once these reports are available.

Final Thoughts

The intent of monitoring the web accessibility directive isn’t to compare accessibility between different countries. However, the lack of consistency does also make it difficult to figure out what approaches are successful and which ones are not. For example, Finland held a large-scale media campaign to promote awareness of inclusion and accessibility. It would be great to know if their campaign was effective, or if that kind of effort is better spent in, for example, creating free training courses as Croatia did.

Setting up EU-wide monitoring of government web accessibility is a great step forward for establishing digital accessibility as a standard across Europe. There is much that can be learned from this first reporting period. As always, there is room for improvement. The need for standard metrics in accessibility, for example, is a common limitation of the current standards, which the upcoming WCAG 3.0 may address. The next reporting period for the EU ends in December 2024, although we hope monitoring agencies will consider publishing preliminary results annually.

If you’d like to learn more, the EU has collected all reports, including an automated English translation. Our own research was done with automated translations, which does have its limitations.