Free tool

Test your robots.txt: who may fetch which URL.

Enter your site and a path. We read your robots.txt and show, for seven crawlers at once, whether the URL is allowed and which rule decided.

Leave the path empty to test the homepage. We test seven crawlers at once and show the rule that decides.

A robots.txt file tells crawlers which parts of a site they may fetch. It is read before anything else, which is why one wrong line can hide a whole site, and why the file deserves a test every time you change it.

How a decision is made

The tester follows the same logic Google documents for its crawlers, which is also the standard published as RFC 9309:

Pick the group. A crawler uses the group whose User-agent line matches its name most specifically, and falls back to * only when nothing more specific exists. A crawler with its own group ignores the * group completely.
Match the rules. Each Allow or Disallow value is a path prefix. * matches any run of characters and $ anchors the end of the URL.
Let the most specific rule win. The longest matching path decides. When an Allow and a Disallow tie on length, the less restrictive one, Allow, wins.
No match means allowed. If no rule applies to the URL, it may be fetched.

For each of the seven crawlers the result shows the verdict and the exact rule that produced it, so you can see why /blog/ is open while /blog/drafts/ is not.

The crawlers we test

Crawler	Why it matters
Googlebot	Indexing in Google Search
Googlebot-Image	Image search
Mediapartners-Google	The crawler AdSense uses to review pages and match ads. Blocking it can stop a review and ad matching
Bingbot	Bing, and the engines that reuse its index
GPTBot, ClaudeBot, PerplexityBot	AI crawlers that publish user agents you can address in robots.txt. Whether to allow them is a business decision, not a technical one

What else the tester flags

A Disallow: / under User-agent: * that blocks the whole site.
The AdSense crawler blocked on the tested path, as an error on the homepage and a warning elsewhere.
Rules that block what looks like CSS, JavaScript or asset folders, which Google needs to render your pages the way visitors see them.
A Noindex: line, which Google has not supported since 2019.
A Crawl-delay line, which Google ignores and Bing respects.
No Sitemap: line, and any declared sitemap that does not answer with a success status.
An HTML page served in place of the file, which is a soft 404 that crawlers will ignore.
Server errors and the file size limit.

Common mistakes

Mistake	What happens	Fix
`Disallow: /` left over from staging	The whole site is blocked	Remove it, then test the homepage here
Blocking `/wp-content/` or `/assets/`	Pages cannot be rendered as visitors see them	Allow the folders that hold CSS, JavaScript and images
Using robots.txt to hide a page	The URL can still appear in results, without a snippet	Use `noindex` on a crawlable page
One file for several subdomains	Rules apply only to the host, protocol and port that serve the file	Give each subdomain its own file
Case mismatch in a path	`/Private/` and `/private/` are different	Match the exact case your URLs use

The robots.txt file must sit at the root of the host: https://your-domain/robots.txt. For the full list of traps and what each costs, read robots.txt mistakes that block Google and AdSense, and for how this fits an application, the AdSense requirements checklist.

Frequently asked questions

Does blocking a page in robots.txt remove it from Google?

No. robots.txt blocks crawling, not indexing. Google documents that it cannot index the content of a blocked page but may still index the URL and show it without a snippet. To keep a page out of results, let Google crawl it and add a noindex directive.

What happens if my robots.txt returns an error?

A 404 is read as no restrictions. A 5xx error is treated as a serious problem: Google documents that it pauses crawling for about 12 hours while retrying, then relies on a cached copy for up to 30 days before it may stop crawling the site.

Is there a size limit?

Google enforces 500 KiB and ignores anything after it. Real files are rarely close, but generated files that list every URL can reach it.

Related guides

Documents, colorful charts and a notebook spread across a wooden desk

AdSense approval

AdSense Requirements for a Website: The Approval Checklist

What Google states about AdSense requirements for a website, what it leaves unsaid, and the checklist of content, pages and technical points that decides a review.

October 7, 2026 · 6 min read