The site that hid its own content from itself

The site that hid its own content from itself

A recipe network rendered everything server-side, did everything 'right,' and still bled 33% of organic traffic after a routine ops change. No code shipped to the pages themselves.

The symptom puzzled everyone: rendered HTML in the testing tool came back blank, even though real browsers showed full recipes.

The investigation moved to robots.txt. Someone had added Disallow: /_next/ to 'keep crawlers out of build internals.' But that path held the JavaScript bundles. Google fetched the page, tried to render it, couldn't load the blocked scripts, and rendered an empty shell. The content existed; the engine that assembled it was walled off.

Turns out blocking your JS directory is functionally the same as blocking your content when the page depends on hydration. The HTML is just a husk without the scripts.

The fix: remove the disallow on the script path. Crawlers need to fetch JS, CSS, and JSON endpoints to render — those aren't 'internals,' they're the page.

Result: -33% reversed to +4% in seven weeks, and the blank-render reports cleared within two crawl cycles. One line in a text file had blindfolded Google.