Crawl budget is a big-site problem you probably do not have
Unpopular position: on a site of a few thousand pages, crawl budget is not your bottleneck. Your pages are not indexed because they are not worth indexing, not because the crawler ran out of patience.
Budget starts to matter when you hold hundreds of thousands of addresses, when URLs are generated from parameters, or when the server is slow enough that the crawler throttles itself.
What to do instead on a small site:
— Delete the pages nobody would miss. Thin, duplicated, near-identical template output.
— Consolidate. Three weak pages on one topic beat each other up. One good page does not.
— Make sure internal links actually reach everything you care about. Orphans go uncrawled because nothing points at them.
If you are small and deep in log files, you are doing the fun work instead of the useful work. I have been guilty of exactly this.
Fix worth before you optimise access.
The Access Log
@TheAccessLog
Crawl budget is a big-site problem you probably do not have
Этот пост опубликован в Telegram-канале The Access Log. Подписаться можно по ссылке: @TheAccessLog.