For site owners
A11yDeskBot, the crawler in your logs.
A11yDeskBot checks websites for accessibility problems on behalf of digital agencies. It visits a site because an agency added it to an A11yDesk account and confirmed that it was authorised to have it scanned. This page says how it behaves on your site, and how to stop it.
What it is, and why it visited
A11yDesk is software that digital agencies use to monitor their clients' websites against the WCAG 2.2 accessibility guidelines, levels A and AA. A11yDeskBot is the part that visits the sites. It loads each page in a web browser, as a visitor would, and runs automated accessibility checks on what the page shows.
It visits only sites an agency has added to its account. Before a site can be added, the agency confirms that it is authorised to run automated accessibility scans of it, and the confirmation is recorded against the person who gave it, with the date. Our terms make that confirmation a warranty the agency gives us.
How to recognise it
Every request it makes carries this user agent, for your pages and for the images, styles and scripts they load:
Mozilla/5.0 (compatible; A11yDeskBot/1.0; +https://a11ydesk.com/crawler)
The name to match, in robots.txt or in a firewall rule, is A11yDeskBot.
How it reads robots.txt
- It requests
/robots.txtbefore anything else, and checks every page against it before loading it, the pages listed in your sitemap included. It reads your sitemap only where robots.txt allows that too. - The rules in a
User-agent: *group and in any group whose User-agent line names A11yDeskBot all apply to it. The name is matched whatever its capitals. - Every Disallow rule in those groups is honoured, including
*wildcards and a$at the end of a rule. Rules are matched against the path of each page, without its query string. - Allow lines are ignored, so an Allow cannot reopen a path that a Disallow covers. That is deliberately strict: where a file could be read two ways, the crawler takes the reading that fetches less.
- Crawl-delay is honoured, up to 30 seconds between pages.
- No robots.txt, answered with a 404 or a 410, means every page may be checked. A robots.txt that cannot be read, because of a server error, a refusal, a timeout, too many redirects or a redirect off the site, stops the scan before any page is loaded.
How fast it goes, and how far
- One page at a time. After each page it waits at least 0.6 seconds before starting the next, or longer if your Crawl-delay asks for it.
- Each page loads its images, styles and scripts, as it would for a visitor. Files shared between pages are cached for the rest of the scan.
- At most 500 pages a scan, or fewer if the agency set a lower limit, and no further than six links from your home page or your sitemap.
- A scan stops after 40 minutes, wherever it has got to.
- At most one scan of a site a day: on the schedule the agency chose, weekly, fortnightly or monthly, or when they start one by hand.
What it does not do
- It does not fill in or submit forms, type into fields or press buttons. It reaches pages by following links and reading your sitemap, and in no other way.
- It does not log in. It has no account on your site, and every scan starts in a fresh browser with no cookies.
- It stays on the site the agency added. That address with and without www counts as one site. Links anywhere else, other subdomains included, are not followed, and a page that redirects off the site is discarded, with nothing from it kept.
- It does not follow links to files, such as PDFs, office documents, images or archives.
- It takes no screenshots and keeps no copy of your pages. For each page it records the address and the HTTP status, and for each failed check the rule, where the element sits on the page and a short extract of its HTML, no more than 1 KB.
How to block it
Add this to your robots.txt:
User-agent: A11yDeskBot
Disallow: /From its next visit it loads none of your pages, and the agency that added your site sees in its dashboard that the scan was blocked. It still requests robots.txt at each scan, and nothing else, because that is where it reads the rule. To keep it out of part of a site only, disallow that path instead of /.
You can also refuse requests whose user agent contains A11yDeskBot at your firewall, your server or your content delivery network. The agency sees the scan as blocked in the same way. Block it by user agent rather than by address: it has no fixed IP address.
Report a scan you did not authorise
If A11yDeskBot is scanning a site you own or run, and nobody entitled to give permission gave it, email helloa11ydesk.com with the domain. We suspend scanning of that site while its ownership is verified, and the agency that added it sees why in its dashboard. That is what clause 2 of our terms commits us to.
Blocking it in robots.txt, as above, works from its next visit, whether or not you write to us.
An agency whose own scan was blocked? The help page says how to let the crawler in.