robots.txt Mistakes That Block Google and AdSense

Googlebot made 57 requests to this site between 23 September and 7 October 2026. Of those, 37 asked for robots.txt, and only 3 distinct pages were read. Ours is a young site, so treat the ratio as an illustration and not a norm, but it shows what the file is: the first thing a crawler reads and the thing it comes back to most. A wrong line there is applied quickly, and to everything.

The AdSense help page for sites that are not ready to show ads names this file directly: robots.txt must not block the AdSense crawler. A blocked crawler cannot review the pages you want approved, and Google does not detail the cause of an individual refusal, so this is a mistake you have to catch yourself.

The rules that cause most of the damage

Google reads one file per host, protocol and port. https://example.com/robots.txt says nothing about https://blog.example.com/ or http://example.com/, and anything beyond the first 500 KiB of the file is ignored.

Inside the file, a crawler obeys only the most specific group that names it, and falls back to the * group only when no group does. Groups that name the same crawler are merged. Each Allow or Disallow value is a path prefix, * matches any run of characters, $ anchors the end of the URL, and the longest matching rule wins. When an Allow and a Disallow tie on length, Google applies the less restrictive one, which is Allow.

What your server answers for the file itself matters just as much:

The prefix rule produces the slips we see most often. Each of these rules does something different from what its author had in mind:

4 mistakes, before and after

1. The staging leftover. A staging site is told to stay out of search with Disallow: /, the site launches, and the file goes live with it. It is the shortest line in the file and the most expensive, because a site Google cannot crawl cannot be approved. Our AdSense checker gives its “crawlers are not blocked” check the highest weight it has, 5 out of 5.

# Before: copied over from the staging site
User-agent: *
Disallow: /

# After
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

2. A specific group that silently replaces the * group. Rules under User-agent: * apply to every crawler that has no group of its own. Add a group for Googlebot, even for a single line, and Googlebot stops reading the * group altogether. Here it may now fetch /admin/:

# Before
User-agent: *
Disallow: /admin/
Disallow: /cart/

User-agent: Googlebot
Disallow: /cart/

# After: repeat every rule the crawler must still follow
User-agent: *
Disallow: /admin/
Disallow: /cart/

User-agent: Googlebot
Disallow: /admin/
Disallow: /cart/

3. Blocking the files that draw the page. Google renders pages, so a stylesheet or script it cannot fetch means it sees a page that looks and may read differently from the one your visitors get. Blanket rules on asset folders are the usual cause. The WordPress default shows the better pattern: block only what has no search value and let the longest rule open the one exception.

# Before
User-agent: *
Disallow: /wp-includes/
Disallow: /wp-content/
Disallow: /assets/

# After
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

4. Using the file to remove a page from results. Disallow stops crawling, not indexing: Google documents that a blocked URL can still be indexed and shown without a snippet. A Noindex: line in robots.txt does nothing either, since Google stopped supporting it in 2019. Worse, a blocked page is a page whose own noindex tag Google can never read.

# Before
User-agent: *
Disallow: /thank-you/
Noindex: /thank-you/

# After: no rule at all. The page stays crawlable and says so itself:
# <meta name="robots" content="noindex">   (or the HTTP header X-Robots-Tag: noindex)
User-agent: *
Allow: /

The crawler AdSense uses

The AdSense crawler identifies itself as Mediapartners-Google. Mistake 5 is blocking it, and it happens in two ways. The first is a group pasted in from a blocklist that names it with Disallow: /. The second is quieter: any rule you set for * is also a rule for this crawler unless you give it a group of its own, so a broad block meant for scrapers catches it too.

# Before
User-agent: *
Allow: /

User-agent: Mediapartners-Google
Disallow: /

# After
User-agent: *
Allow: /

If * blocks folders on purpose and you want the AdSense crawler held to the same limits, name it anyway and repeat those rules in its group, for the reason shown in mistake 2. Our recommendation is to keep this group explicit: a visible line is easier to audit than an assumption about fallbacks. The AdSense requirements checklist lists where this check sits in a review.

AI crawlers are an editorial decision

Whether to let AI crawlers read your site is a business choice, not a technical fault, and robots.txt is where you record it. The syntax is the same as for any crawler, and several User-agent lines can share one group:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: PerplexityBot
Disallow: /

Two cautions. First, the file is a request that well-behaved crawlers honor. RFC 9309 itself says it is not a security measure, so it will not stop a crawler that ignores it. Second, vendors publish separate tokens for separate purposes, and the names change. Take them from each vendor’s own documentation, not from a list passed around on forums. Google, for one, documents Google-Extended as the token that controls whether content it crawls is used for its AI models, and says blocking it does not affect inclusion in Search.

Ours allows everything, which is a decision, not an oversight. In the same 14.8 days, OpenAI’s crawlers read 7 pages and Anthropic’s crawlers read our robots.txt 28 times and our sitemap 29 times but no page at all. The full breakdown is in who crawls a small site.

Test the file before and after every change

Fetch it the way a crawler does, and look at the status and the type before the content:

curl -sI https://example.com/robots.txt | head -n 5
curl -s https://example.com/robots.txt | grep -nEi '^disallow:[[:space:]]*/[[:space:]]*$'

The first command should show a 200 and a plain text content type. A friendly HTML error page served with a 200 at this address is a separate trap: the file looks present and says nothing. The second command prints every line that blocks the whole site. It will also catch a legitimate Disallow: / under a bot you chose to block, so read which group each line sits in.

Then run the robots.txt tester. It shows, for Googlebot, the AdSense crawler, Bing and the AI crawlers at once, whether a URL is allowed and which rule decided. Search Console’s robots.txt report shows the copy Google fetched last.

Do this on every deploy, not once. A file generated by your build or your CMS can change without anyone touching it, and Google keeps a cached copy, so a fix is not instant. When you are done, declare the sitemap with an absolute Sitemap: line, which works from anywhere in the file: XML sitemaps for small sites covers what to put in it.