The CDN rule that starved the renderer

The CDN rule that starved the renderer

A news aggregator rendered headlines client-side from a JSON API. Overnight, indexed articles dropped 47%. No deploy, no content change, no robots edit.

The symptom: rendered HTML came back as a skeleton — headline placeholders, no actual headlines. Browsers were fine.

The investigation traced the page's network calls during render. The headlines came from api.site.com/feed, and that subdomain sat behind a CDN security rule someone had tightened the night before to block 'non-browser' traffic by user agent. Googlebot's renderer fetched the API, got a 403, and rendered the page with empty data. The HTML page was crawlable; its data source was not.

Turns out your content is only as crawlable as the API it depends on. Block the JSON endpoint and you've blocked the page, even if the page itself is wide open.

The fix: allow Googlebot's verified ranges through to the API endpoint, and add a smoke check that renders a sample page through Google's tester after any CDN rule change.

Result: -47% recovered to baseline in six weeks, and the team added 'render a live page after every edge-config change' to their deploy checklist permanently.