How to Fix Crawl Errors in Google Search Console

Google splits crawl problems into two categories. Site errors are conditions that block Google from reaching your server at all - DNS failures, server timeouts, robots.txt fetch failures. URL errors are conditions where Google reached the server but a specific URL failed - 404s, soft 404s, redirect loops, 5xx responses on specific pages. The two require completely different investigation paths.

Site errors are urgent. When Google cannot reach your server for even a few hours, crawl rate drops sharply and takes days to recover. The Search Console Crawl Stats report shows daily fetches and average response time - any sudden drop in fetches paired with a spike in average response time indicates a site-level problem.

URL errors are patient. Individual 404s do not hurt the site as a whole and are a normal part of the web. What matters is the pattern: 404s on important pages, mass 404s from a bad deploy, or 404s on pages that used to have traffic. Fix in that priority order.

When Googlebot sees a 5xx response, it interprets that as your server being overloaded and slows the crawl rate almost immediately as a courtesy. If 5xx persists for hours, Google starts dropping pages from the index within days on the assumption they are no longer available. This is the single fastest way to lose organic traffic.

The most common cause is your server timing out under a spike in crawl volume. This can happen after a large sitemap submission, after a competitor's link earns you sudden fame, or after an accidental infinite crawl loop from a bad URL parameter setup. Rate-limit your origin behind a CDN, cache aggressively at the edge, and use a WAF to allow Googlebot without letting scrapers hit the origin.

Check your access logs for user agent 'Googlebot' specifically. Any 5xx served to that user agent is far more costly than the same 5xx served to a random visitor. If you rate-limit, whitelist Googlebot's IP ranges (verified via reverse DNS lookup - do not trust the user agent alone since scrapers spoof it).

A soft 404 is a page that returns HTTP 200 but tells the user 'no results' or 'coming soon' or has content so thin it functionally is not a page. Google detects this via content evaluation and drops the URL from the index while it still shows as 200 to your server logs. You will not notice unless you check the Coverage report specifically.

The fix is to make the honest choice explicit. If the page should not exist anymore, return a real 404 or 410. If the page exists but should be a redirect (a moved product, a renamed category), return a 301. If the page exists and should serve content, add real content - at least 200 words of unique, useful body copy and appropriate structural elements.

Empty search results pages are the most common soft 404 source on ecommerce sites. Instead of showing 'No results found' on a URL that returns 200, either redirect to a parent category (with 302 for a temporary state) or return 404 and log the search term so you can add content for high-volume empty queries.

Every hop in a redirect chain costs a crawl budget request and delays the final page load. A→B→C is twice as expensive as A→C and results in slower crawl and slower user experience. Google will follow up to about five hops before giving up and treating the URL as unreachable. Most chains you find in real audits are three to seven hops long, accumulated over years of migrations.

The audit is mechanical: crawl your site with any SEO crawler (Screaming Frog, Sitebulb) with the 'follow redirects' option enabled and export the report showing all redirect paths. Sort by chain length descending. Rewrite the first-hop redirect on every chain to point directly to the final URL. Update internal links and sitemaps to point to the final URL, not the first-hop URL, so the redirect eventually falls out of use entirely.

Redirect loops (A→B→A) are catastrophic. Google abandons the URL entirely and drops any linked-from equity. Test every non-trivial redirect chain in a browser incognito window before deploying. A single loop can quietly de-index a critical section of your site.

Google fetches robots.txt before every crawl session. If the fetch fails (5xx or timeout), Google's behavior depends on the failure type. Persistent 5xx causes Google to stop crawling the site entirely as a safety measure. Transient failures cause it to fall back to a cached copy for up to 24 hours. Either way, robots.txt failures are always a P0 incident.

The most common cause of robots.txt failures is the file being served by the wrong subsystem. If your CMS serves robots.txt from a database and the database is down, robots.txt returns 500 - even though your static pages still work fine. Serve robots.txt from a static file or edge worker so it stays available independently of your application stack.

The second most common cause is a well-meaning developer editing robots.txt during a deploy and shipping a syntax error. Robots.txt is unforgiving - a single stray character can turn a targeted disallow into a site-wide block. Test every change with Search Console's robots.txt tester before deploying, or add a pre-deploy check that validates the file against the robotstxt spec.

Search Console's Coverage report categorizes every URL Google knows about into Valid (indexed), Excluded (deliberately not indexed), and Errored (should be indexed but is not). Reading it correctly is a skill. Excluded categories like 'Alternate page with proper canonical' and 'Page with redirect' are usually fine. Excluded categories like 'Crawled - currently not indexed' and 'Discovered - currently not indexed' are the ones to investigate.

Sort each excluded category by count descending. The largest categories tell you where you are wasting the most crawl budget or losing the most potential indexation. Fix the biggest category first - the impact is proportional to the count.

The Coverage report updates on a delay of several days to a week. Do not judge fixes on same-day data. Make a change, wait a full week, then re-check the count for that category. A dropping count is the confirmation your fix worked.

Every deploy is an opportunity to break something. Sites that treat SEO seriously add a pre-deploy crawl check to the CI/CD pipeline. The check crawls the staging environment for a fixed set of URLs, verifies each returns a 200, has a title, has a canonical, and does not have an accidental noindex. Any failure blocks the deploy.

This catches an enormous category of bugs before they hit production: template changes that removed a title tag, layout changes that added a rogue noindex, routing changes that broke a canonical, migration changes that introduced a redirect loop. The one-time engineering cost of building the check pays for itself the first time it prevents a bad deploy.

Screaming Frog can be run in headless mode from a CI job. So can Sitebulb. Custom lightweight crawlers in Node or Python take a few hours to build. Pick whichever fits your stack and get it in place before the next major refactor, not after.