Conditional requests save real crawl budget, and almost nobody configures them

Conditional requests save real crawl budget, and almost nobody configures them

Myth: crawl budget is about which URLs Google fetches.

Partly. It's also about how much it costs to fetch them. Two URLs cost the same in your head; they don't on the wire.

Here's the thing. Googlebot sends If-Modified-Since and If-None-Match headers. If your server answers 304 Not Modified, Google gets confirmation for a few bytes instead of re-downloading the whole page. That stretches the same budget across far more URLs.

Most servers ignore the headers and return a fresh 200 every time. You're paying full price to tell Google nothing changed.

The tradeoff: ETags are precise but need consistent generation; Last-Modified is simpler but easy to get wrong. Either beats full re-downloads.

The cheapest crawl is the one that returns 304.