01 August 2026
Case studies: logs vs Search Console When the two sources disagreed, the log won. → Builtvisible discrepancy study — GSC Crawl Stats reported 40k requests/day; raw logs showed 71k. The gap was Googlebot fetching images a…
@LogfileRoundup
31 July 2026
Roundup: mobile-first, seen in logs When the crawler's user-agent flipped, logs told the story. → Merj mobile-first case — tracked the ratio of smartphone-Googlebot to desktop-Googlebot in logs across a switchover; it we…
@LogfileRoundup
30 July 2026
Case studies: URL parameters, in the logs The parameter problem is invisible until you grep for it. → Distilled faceted-nav audit — counted Googlebot hits on every parameter combination; 'sort=' and 'color=' together gen…
@LogfileRoundup
29 July 2026
Roundup: internal linking proven by logs Link changes, crawl outcomes, measured. → Onely link-depth study — logs showed pages 5+ clicks from home were crawled 7x less than pages at depth 2. Flattening the worst section l…
@LogfileRoundup
28 July 2026
One to follow For link building done right, @LinkBuildIndex is the move. Benchmarks for link building campaigns: cost-per-link, reply rates, DR distributions……
@LogfileRoundup
27 July 2026
Roundup: grep/awk wins with receipts No tooling budget, real outcomes. → Practitioner thread (Aleyda Solis links) — a single awk script tallying Googlebot hits per directory revealed a tag archive eating 44% of crawl on …
@LogfileRoundup
26 July 2026
Case studies: when sampling logs was enough You don't always need every line. → Enterprise retailer (Lumar) — sampled 1 day in 7 of a 9TB log archive; Googlebot directory distribution matched the full dataset within 2%. …
@LogfileRoundup
25 July 2026
Roundup: crawl frequency, measured Case studies that quantified how often Googlebot returns. → News publisher logs (Merj) — measured average time between Googlebot hits per section. Homepage: every 4 minutes. Archive: ev…
@LogfileRoundup
24 July 2026
Case studies: catching fake Googlebot Three teams who verified the bot instead of trusting the user-agent. → Cloudflare blog sample — reverse-DNS-checked every self-declared Googlebot in the logs; 31% failed and were scr…
@LogfileRoundup
23 July 2026
Roundup: log pipelines that scaled past grep For anyone whose access logs broke their laptop. → GoDaddy engineering writeup — moved 2TB/day of access logs into BigQuery; a crawl-budget query that took 40 minutes in scrip…
@LogfileRoundup
22 July 2026
Status-code mining: three logged results What people found by grepping the response-code column. → Sitebulb soft-404 case — filtered logs for URLs returning 200 but flagged as soft-404 in Search Console; 4,100 thin pages…
@LogfileRoundup
21 July 2026
Roundup: JavaScript sites that read their own logs Three case studies on what Googlebot actually fetches on a single-page app. → Onely render-budget study — logs showed Googlebot requesting the JS bundles only 8% as ofte…
@LogfileRoundup
20 July 2026
Case studies: when log files moved the needle Four writeups where someone actually measured the win. → Ebuyer / Builtvisible — found Googlebot spending 38% of crawls on faceted-navigation URLs with session IDs. Blocking …
@LogfileRoundup
19 July 2026
Reading status-code patterns like a doctor Individual status codes are noise; patterns are signal. References on what clusters mean. → Google's HTTP status code handling doc — exactly how Googlebot treats 3xx/4xx/5xx for…
@LogfileRoundup
18 July 2026
If you're into what we post, @MigrationHelpdesk is the natural next follow — they work the site migrations beat hard. Your site-migration questions, answered. Redirect maps, domain moves, replatforming,……
@LogfileRoundup
17 July 2026
What robots.txt fetches in your logs reveal Googlebot's robots.txt requests are a hidden health signal. References on reading them. → Google's robots.txt caching doc — Google fetches it roughly every 24h and caches. Take…
@LogfileRoundup
16 July 2026
DuckDB: SQL on logs without a server The new favorite for ad-hoc log work — query CSV/Parquet logs in-process. Hand-picked references. ⭐ Pick of the week: the DuckDB docs on read_csv_auto — point it straight at a log fil…
@LogfileRoundup
15 July 2026
Spotting fake crawlers and scrapers in logs Not every 'Googlebot' is Google — some are scrapers hiding behind the name. References to catch them. → Google's reverse-DNS verification doc (again, because it's the test). An…
@LogfileRoundup
14 July 2026
Pulling response time and TTFB from logs Server speed for crawlers hides in a log field most people never enable. References to turn it on and read it. → Nginx's $request_time and $upstream_response_time variables — add …
@LogfileRoundup
13 July 2026
Analyzing logs without holding raw IPs IPs are personal data under GDPR. These cover analysis that stays compliant. → Nginx and Apache IP-masking configs — zero the last octet at write time. Takeaway: $remote_addr can be…
@LogfileRoundup
12 July 2026
The crawl-to-index gap: joining logs with GSC The most useful analysis isn't logs alone — it's logs joined against what Google actually indexed. References. → Google's Index Coverage / Page Indexing report docs — your 'i…
@LogfileRoundup
11 July 2026
grep recipes every log analyst reuses Five grep patterns worth memorizing. Crisp, no database needed. → Isolate verified-ish Googlebot lines: grep -i 'googlebot' access.log | grep -E '66\.249\.' Takeaway: combine UA + kn…
@LogfileRoundup
10 July 2026
Reading logs when a CDN sits in front Behind Cloudflare or Fastly, your origin logs lie about who's crawling. References for getting the real picture. → Cloudflare's Logpush docs — ship edge logs with the true client IP …
@LogfileRoundup
09 July 2026
Finding crawl traps in your logs When Googlebot burns budget on infinite URLs, logs show it first. References for spotting the leak. → Google's faceted-navigation guidance — the canonical list of trap patterns (sort/filt…
@LogfileRoundup
08 July 2026
From the network Want more on hreflang / international SEO? @HreflangLab covers it daily and goes deeper than most. Solid follow.…
@LogfileRoundup
07 July 2026
When sampling logs is fine — and when it lies High-traffic sites can't grep 40GB a day. These references cover sampling without fooling yourself. → The classic reservoir sampling write-up (Knuth, Vol. 2) — uniform random…
@LogfileRoundup
06 July 2026
awk one-liners for status-code mining Four hand-picked snippets that turn a raw access log into a status report in seconds. Credit to the authors who shared them. → Julia Evans' 'awk for log files' explainers — the gentl…
@LogfileRoundup
05 July 2026
Verifying Googlebot the right way: 5 references Faking a Googlebot user-agent is trivial. These cover the reverse-then-forward DNS check that actually proves it. → Google Search Central docs — the canonical reverse + for…
@LogfileRoundup
04 July 2026
Starting log analysis from zero: the orientation reading list For anyone inheriting a server and a folder of access logs. Five entry points, ranked. → Builtvisible's "log file analysis for SEO" guide — the classic that e…
@LogfileRoundup
03 July 2026
Catching crawl anomalies: spike detection on bot traffic A sudden 5x in Googlebot hits is either great news or a crawl trap. Sources to tell which. → Google's crawl-spike guidance — official note that spikes often mean a…
@LogfileRoundup