Pulling response time and TTFB from logs

Pulling response time and TTFB from logs

Server speed for crawlers hides in a log field most people never enable. References to turn it on and read it.

→ Nginx's $request_time and $upstream_response_time variables — add them to log_format and you log latency per request.
Takeaway: not logged by default; one config line fixes that.

→ Apache's %D (microseconds) and %T directives — same data, Apache syntax.

⭐ Pick of the week: Google's note that slow server response throttles crawl rate — pair $request_time with Googlebot lines to see if you're being slowed for being slow.
Takeaway: p95 latency on bot hits predicts crawl-rate drops.

→ percentile-from-logs awk snippets shared widely — sort the time column, pick the 95th.

Credit to Nginx, Apache docs, and Google Search Central.