Spider Crawling
See a URL the way a crawler does: status, headers, canonical, robots rules, and the text that comes back. It also fetches the page as a normal browser and compares the two. If the title, status, or word count changes, you have a cloaking question to answer.
What to look for
- Status. A 200 for a browser and a 403 for Googlebot is a crawl problem. A 200 for both is fine.
- Title and H1. If the bot gets a different title, the page is being rewritten for crawlers. Sometimes that is an SEO plugin, sometimes it is cloaking. Read it.
- Canonical. The canonical the bot sees is the one that counts. If it points at a different URL, that is the page Google will try to index.
- Meta robots and X-Robots-Tag. Either one can noindex the page. The header version is easy to miss.
- robots.txt. A blocked path stops the crawl. It does not remove the URL from the index.
- Text. If the bot gets an empty shell and the browser gets content, the page depends on JavaScript. Google can render JS, but it is slower and less reliable. Server-render the text if it matters.
Cloaking, A/B tests, and the difference
Not every difference is cloaking. An A/B test that excludes bots is a normal experiment. A geo-redirect that sends Googlebot to the US version is normal. Cloaking is showing a crawler something different on purpose to manipulate the index. The test is intent, and the evidence is the diff.
If you run an A/B test, exclude known bots in the test tool. If you use a CDN or WAF, make sure verified Googlebot IPs are allowed. If you use a page builder that serves a blank shell to bots, fix the rendering. Those three cover most of the false alarms.
Why the old page was useless
The old spider tool showed a text dump and a list of links. It did not show the status, the headers, the canonical, or the robots rules. A text dump without the HTTP response is half a diagnosis. This version fetches the page twice and puts the differences first.
Questions
Does this use the real Googlebot?
No. It sends a Googlebot user agent from our server. That is enough to catch a lot of cloaking and broken bot handling, but it is not Googlebot and it cannot see Google's index. Google's own URL Inspection tool is the only way to see the real crawl.
What is cloaking?
Showing a different page to a search engine than to a human. Some of it is innocent (A/B tests that exclude bots). Some of it is a violation. If the title, status, or word count differs, read the difference before you panic.
Why does Googlebot get a 403?
A firewall or CDN rule is blocking the user agent. That can stop Google from crawling the page. Check your WAF logs and allow verified Googlebot IPs. A fake user agent from our server may still be blocked even when real Googlebot is allowed.
What does robots.txt have to do with this?
A Disallow rule does not remove a page from the index. It stops the crawl. The page can still be indexed from links. If you need a page out of the index, use noindex and let Google crawl it.
Related tools
If the bot view is broken, the ranking cannot be fixed with a title rewrite.
The Fix Sprint starts with access and inventory, then checks what Google actually sees: status codes, robots, canonicals, sitemaps, and the pages with impressions and no clicks. $1,500 for one site up to 50 pages.
See the fix service Ask a question