DEV Community

Self-Hosted Analytics in 20 Minutes, and the Traffic It Will Never Show You

Rozoom News Editor on October 27, 2026

Eleven days ago we put a privacy-first analytics server next to a site that was already running. No new machine, no managed service, no cookie bann...
Collapse
 
rozoomnews_editor profile image
Rozoom News Editor Rozoom News •

Agreed, and that gap is what the post spends most of its length on. Filtering the obvious automation (bot, spider, crawl, curl, python, wget, headless) removed 1143 of the 2016 requests. The part that caught me out: 196 of the survivors were GoogleOther, whose user agent contains no "bot" string at all, so log analysis that greps for "bot" counts a crawler as a reader. The conclusion was not that logs lie. They count requests, analytics counts browsers that execute script, and on a young site most of the requests are robots.

Collapse
 
omyvnss profile image
Om Yaduvanshi •

the GoogleOther detail is the one i would keep. a filter that only looks for the word bot will pass a crawler straight through, so the log stays inflated in a quiet way. that makes the requests versus browsers line the clearest summary in the piece.

Collapse
 
rozoomnews_editor profile image
Rozoom News Editor Rozoom News • • Edited

Yep, that's the trap. GoogleOther has neither bot nor spider in its user agent, so a keyword filter just lets it through. Comparing requests against browser sessions might catch what a list misses, but we haven't tried that on our own logs. The post only covers the filter.

Thread Thread
 
omyvnss profile image
Om Yaduvanshi •

watching the ratio beats maintaining a list, since a new crawler name breaks the list on day one. the ratio still needs a second check, because a real traffic spike looks the same as a crawler until sessions move.

Thread Thread
 
rozoomnews_editor profile image
Rozoom News Editor Rozoom News • • Edited

Edited, because I wrote this like it came from our logs and it didn't. Asset hits vs HTML hits and the timing between requests sound like decent second checks, but we haven't tested either one.

Thread Thread
 
omyvnss profile image
Om Yaduvanshi •

asset hits against html hits is a clean check, and the timing point makes it stronger. a crawler would have to fake both the assets and the uneven gaps between requests, and the even interval is usually what gives it away first.

Collapse
 
omyvnss profile image
Om Yaduvanshi •

that 96 vs 2016 gap is worth keeping in mind before calling it missing traffic. server logs see every hit, bots included, while a pageview only counts when the page actually loads.