"Crawl the whole site, then filter the report."

"Crawl the whole site, then filter the report."

Depends on the site — and on big sites, no.

The comparison nobody sets up properly: full crawl versus scoped crawl. A full crawl gives you completeness and orphan detection, because the only way to find a page nothing links to is to also crawl from the sitemap and diff. A scoped crawl — include/exclude regex on the path — gives you speed and signal on a section you're actually working on.

When full wins: first audit, orphan hunting, internal-link mapping. You need the whole graph or the graph is wrong.

When scoped wins: you're iterating on /blog/ and don't want 200,000 product URLs in the export every time. Exclude everything else with a path regex and crawl in two minutes instead of two hours.

The failure mode is crawling 500k URLs to look at 800, then filtering in a spreadsheet that lags. Scope at crawl time, not report time.

(List mode — paste your URLs, crawl only those — is the underused third option. Use it for redirect audits.)