Default UA vs. Googlebot UA: what your audit hides

Default UA vs. Googlebot UA: what your audit hides

Ran paired crawls — default crawler UA vs. spoofed Googlebot Smartphone — on 102 sites.

— Identical responses: 79.4% ▓▓▓▓▓▓▓▓░░
— Different status code under Googlebot: 8.7% ▓░░░░░░░░░
— Different content / blocked: 11.9% ▓▓░░░░░░░░

Most divergence: CDN bot-management serving 403 to non-verified Googlebot, or edge rules stripping content.

So what: auditing with the default UA gives you a site that may not exist for Google. Always run a second pass as Googlebot UA and verify whether your edge does reverse-DNS validation — spoofed UA alone gets blocked on 11.9% of sites. The audit that matters is the one Google actually receives, not the one your tool's default fetches.