Seostupidity

The noindex you forgot to remove

By Matt Hopkins ·

You build the new site on staging with a noindex on every page so Google stays out. Then you launch, and the noindex launches with it. Nothing breaks and nothing looks wrong, but your pages drop out of search. There are a few versions of this mistake, and two of them come from robots.txt doing something other than what people think.

The staging noindex that went live

A noindex can live in two places: a <meta name="robots" content="noindex"> tag in the page's head, or an X-Robots-Tag: noindex HTTP response header. Google's noindex documentation supports both, and once Googlebot reads either one, it drops the page from results even if other sites link to it.

That's what you want on staging. The trouble is that both carry over easily. The meta tag comes along when you copy a template or migrate a database. The header lives in server or CDN config that nobody reviews at launch, and viewing the page source won't show it.

WordPress has its own version. When the "Discourage search engines from indexing this site" box under Settings, Reading is ticked, WordPress has added a noindex,nofollow robots meta tag to the site's pages since version 5.3. Tick it on staging, copy the database to production, and it's still ticked.

Recovery isn't instant either. Google has to recrawl a page to notice the tag is gone, and the same documentation says that for less important pages, a revisit can take months.

Noindex in robots.txt does nothing

Some old guides tell you to put a Noindex: line in robots.txt. Google never published that as a supported rule, and in July 2019 it announced it would retire the code that handled unsupported rules like it on September 1, 2019. The current noindex documentation says plainly that a noindex rule in robots.txt is not supported by Google. If that line is your only protection, you have no protection.

A Disallow hides your noindex

This one catches careful people. You add noindex to a page, and to be extra safe you also block it in robots.txt with Disallow. Now Google can't crawl the page, so it never reads the noindex. Google's robots meta tag specification says these rules are only discovered when a URL is crawled, and that if robots.txt blocks the crawl, they won't be found and will be ignored. The noindex page says the same: the crawler never sees the rule, and the page can still appear in search results.

The two rules do different jobs. Disallow controls crawling. Noindex controls indexing. Google can only read the indexing rule on a page it's allowed to crawl.

Disallow doesn't remove anything

The flip side: blocking a page in robots.txt doesn't take it out of Google. Google's introduction to robots.txt says it is not a mechanism for keeping a web page out of Google. If other pages link to a blocked URL with descriptive text, Google can index the URL without visiting it, and the result shows up with no description because Google never read the page.

So a Disallow added to "clean up" old URLs can leave them in the index with a blank snippet, and Google has no way to read anything on those pages that says otherwise. Robots.txt has other surprises too, like blocking AI crawlers you meant to allow.

The fix

  1. Check production, not staging. View the source of your live pages for a robots meta tag, and check the response headers for X-Robots-Tag (curl -sI https://example.com/ shows them). Do it on several page types, not only the homepage. In WordPress, make sure the Discourage search engines box is unticked.
  2. Keep staging out with a password instead. Google's robots.txt guide names password protection, alongside noindex, as the way to keep a page out of Google. If a login wall reaches production by mistake, you'll notice in seconds. A forgotten noindex tag you won't.
  3. Delete any Noindex line from robots.txt. It does nothing for Google. Use the meta tag or the header.
  4. To remove a page, let Google crawl it. Lift the Disallow, then add noindex, or return 404 or 410 if the page is gone for good. The 2019 post lists both as ways to drop URLs from the index. Leave the page crawlable so the noindex keeps working.
  5. For something urgent, Search Console's removal tool hides a URL quickly, but the 2019 post describes it as temporary. Fix the page underneath it as well.
  6. After a launch or a fix, request a recrawl of your most important URLs with the URL Inspection tool, which the noindex documentation points to, instead of waiting for Google to come back on its own.