Myth-Busting SOP: 'robots.txt Disallow Hides Sensitive Paths'

Myth-Busting SOP: 'robots.txt Disallow Hides Sensitive Paths'

Disallow: /admin doesn't block access — it publishes a map of what you want hidden, and the URL stays reachable. Robots is a crawl request, not a control. Fix the confusion:

— Step 1: Remove sensitive paths from robots.txt; you were advertising them.
— Step 2: Protect those paths with authentication or IP rules at the server, not a text file.
— Step 3: To keep a page out of search, use X-Robots-Tag: noindex on an access-controlled URL.
— Step 4: Confirm the path returns 401/403 to anonymous requests, not 200.
— Step 5: Verify by fetching the URL directly — robots.txt won't stop a curl, only auth will.

robots.txt guides crawlers; it guards nothing. Run this every time.