RotproofBot

RotproofBot is the crawler behind Rotproof, a service that checks websites for broken links and tells the people who run them what to fix. Rotproof is operated by TRD Ventures LLC. If you found RotproofBot in your logs, this page explains what it does and how to stop it.

How to recognize it

Every request sends this user agent:

Mozilla/5.0 (compatible; RotproofBot/1.0; +https://rotproof.dev/bot)

We don't imitate a browser or device, and we don't rotate user agents to get past blocks.

What we fetch

• Pages on our customers' sites. We crawl a site on a schedule only after its owner proves they control it, with a DNS record or a meta tag. We start from the homepage and sitemap and read the links on each page. A one-off free scan covers at most 25 pages.
• The links on those pages. If a customer's page links to your site, we check that one URL. We don't crawl the rest of your site.

We send a HEAD request first, and a GET if that doesn't give a clear answer. If a link fails, we check it once more about 60 seconds later. We follow up to 10 redirects and give up on a request after 15 seconds.

We only request http and https URLs on default ports. We don't run JavaScript, log in or submit forms.

How fast we go

We send at most one request at a time to each host, at least 1 second apart. If you answer 429 or 503, we honor Retry-After and back off.

robots.txt

We follow robots.txt for every page we crawl on a customer's site. Checking a single link isn't a crawl, so those checks don't read it.

What we keep

For each link we keep the URL, its status and redirect chain, the headers we looked at, a content hash and a short excerpt as evidence. On customers' own pages we also keep the page title and the text around each link. We never store full pages.

How to opt out

To stop us crawling your site, add this to your robots.txt:

User-agent: RotproofBot
Disallow: /

To stop all requests to your site, including single link checks, email bot@rotproof.dev with your hostname. We'll add it to our opt-out list.

If you block us, links to your site show as Blocked in our customers' reports, never Broken.

Contact

For questions or problems with the crawler, email bot@rotproof.dev. To report a vulnerability, see our security policy.