When I first looked at the raw numbers coming from my Cloudflare Worker, the disparity was startling: over three hundred requests that carried a browser-like User-Agent were being reclassified as something else once I applied simple network and header checks. This 74.5% shift did not magically reveal the true count of human readers, but it did expose a critical flaw in my earlier assumption that any browser UA equated to a person engaging with my content. The realization forced me to step back and ask what each counter was actually measuring, and why the two sources—my custom edge counter and Cloudflare Web Analytics—were telling such different stories. In practice, this kind of gap is a goldmine for anyone operating a site at scale, because it highlights where instrumentation blind spots hide automated scripts, misconfigured crawlers, or even legitimate traffic that looks synthetic due to missing headers or unusual network origins.
The initial shock came from comparing a daily client identifier of 113 with a mere 52 page loads recorded by the analytics beacon. At first glance, the eleven‑fold difference suggested an explosion of readership that defied intuition, especially after publishing a new article that never appeared in the “top pages” list. My late‑night notes reflected a mix of fascination and unease: who were these supposed readers, where were they coming from, and why did the numbers feel so disconnected from the activity I could see on the surface? The answer lay not in a conspiracy of hidden fans but in the fact that the client identifier counted unique identifiers over a window that did not align with the analytics snapshot, and that the beacon itself omitted certain requests—such as those to /stats pages or prefetches—by design. Recognizing that the two metrics were tracking distinct events turned the alarm into a starting point for a methodical investigation rather than a cause for panic.
Before jumping to any ratio that would claim to separate humans from bots, I needed to lock down four foundational decisions. First, I had to account for the ways Cloudflare’s beacon can miss page loads: script blockers, network loss, browser caching, and differing eligibility criteria all create blind spots. Second, I acknowledged that a script can run inside an automated browser, meaning that the presence of JavaScript execution does not guarantee a human user. Third, I accepted that there is no universal constant that cleanly divides these phenomena; each site must tune its thresholds based on observed behavior. Fourth, I consulted the Cloudflare Web Analytics FAQ (checked September 6, 2026) to confirm that the beacon’s limitations are documented and that any ratio derived from raw counts must be treated as a hypothesis, not a measurement.
With those premises in place, I built a deterministic classifier inside the Worker that works purely from request characteristics—no client‑side instrumentation required. The first rule I applied was to examine the originating network before even looking at the shape of the request. After processing any signature‑based or named‑client matches, I checked whether the request’s IP belonged to a curated list of known hosting providers. If it did, the request was labeled “cloud‑browser,” regardless of whether it passed later navigation‑header tests. In a 72‑hour sample ending September 3, this single rule moved 430 out of 844 successful GETs out of the raw browser bucket, with the largest cluster being 374 requests from a single Google Cloud instance claiming Chrome Mobile 114. The header‑based checks alone could not separate that traffic, proving that network context is often a stronger signal than the UA string.
I deliberately kept the cloud‑browser label visible because such traffic can represent genuine human activity—think of a developer testing a site from a cloud IDE or a researcher using a hosted browser instance. By tagging the request with its network provenance, I retain the observation while preventing it from inflating the “human browser” count. Curating the hosting list, however, involves trade‑offs: a overly broad list would sweep in legitimate residential or corporate traffic, while a too‑narrow list lets some automation slip through via lesser‑known clouds or VPNs. The goal is not to achieve perfect separation but to make the classification transparent so that I can later examine which requests fell into each bucket and why.
The second rule focused on the presence—or absence—of Fetch Metadata headers, which modern browsers are expected to send when navigating a page. For any request that claimed to be Chrome 76+, Firefox 90+, or Safari/iOS 16.4+ but lacked a Sec‑Fetch‑Mode header, I marked it as an http‑client with the reason “no‑fetch‑metadata.” Older or unparsable UA strings fell into a legacy‑browser bucket, while requests that did send Fetch Metadata but failed to satisfy a valid navigation combination (such as missing Sec‑Fetch‑Dest or having an illogical mode‑dest pair) became “not‑navigation‑shaped.” These thresholds were derived from published browser support matrices as of September 6, 2026, and they serve as capability expectations rather than authentication proofs. Notably, I did not require Sec‑Fetch‑User, because complementary research showed that Safari and its WebViews often send the other Fetch Metadata headers without it, making a blanket rule against WebViews untenable.
During a follow‑up audit I uncovered a subtle bug in my own Accept‑header handling. The original helper function returned true for Accept: text/html;q=0 (which explicitly rejects HTML) and false for Accept: text/* (which, under RFC 9110 semantics, actually accepts HTML). The implementation had been using a simple substring match that ignored quality values and media‑range rules, leading to misclassifications in edge cases. After correcting the function to parse quality, specificity, and representation parameters according to the RFC, a local regression test showed seven of twelve previously flawed cases now returning the correct boolean. While this exercise does not quantify the historical impact of the bug, it demonstrates the importance of grounding header parsing in the official specifications rather than relying on heuristic shortcuts.
The design of my classifier borrows from several existing projects, but I deliberately avoided claiming novelty for the combination. Techniques such as declared‑bot detection, ASN‑based network classification, and attaching a human‑readable reason to each decision have appeared in tools like GoatCounter, isbot, and Anubis. What sets my approach apart is the explicit retention of those reasons alongside the classification, enabling a post‑mortem audit trail. I also integrated the site’s existing Web Bot Auth signature verification, but I resisted the temptation to treat a verified signature as proof that a person had delegated the request to an assistant. A signature merely binds the request to a cryptographic identity; it says nothing about whether that identity is acting on behalf of a human, a scheduled job, or a fully autonomous agent.
Looking at the signed‑request log since the verifier went live, I found nine entries with a stored verification result through September 5 UTC. These are not fresh, independent verifications of every historical request but rather cached outcomes from the verification pipeline. The earlier claim that no signed request had ever arrived was therefore incorrect; a signed client can indeed be a crawler, as demonstrated by Exa’s documentation of ExaSearchBot fetching and indexing pages. The versioned Web Bot Auth draft clarifies that the signature authenticates the request’s provenance, not the delegator’s intent or purpose—a distinction that is crucial when we later try to infer human engagement from automated traffic.
My current stats view still groups verified signatures and declared AI‑agent User‑Agents under a single “AI agents” label, a conflation that obscures whether the traffic is authenticated automation or merely a bot pretending to be one. I plan to repair this grouping so that analysts can see, for example, that a request carries a valid ExaSearchBot signature but is still categorized as an automated fetch rather than a human‑directed AI assistant. The older thirty‑day snapshot revealed dozens of matches for ChatGPT‑User, PerplexityBot, Amazonbot, and GPTBot across numerous paths, but those numbers are merely access patterns. They do not, by themselves, confirm that thirty‑three distinct humans were reading via ChatGPT, nor do they explain why those crawls happened—whether they were triggered by a user query, a site‑map refresh, or a third‑party service.
The underlying motivation for keeping all this traffic visible is a publishing‑centric question: I want to understand how readers are interacting with my work, whether through a traditional browser, a headless script on a home connection, or an AI agent running in the cloud. Imagine a scenario where a researcher uses a tool like Playwright to scrape my article for a literature review, or a personal assistant powered by a large language model fetches the piece to summarize it for a user. I do not wish to block or miscount such interactions; instead, I aim to categorize them transparently, respecting privacy while gaining insight into the pathways by which content spreads. In an ideal world, I would even like to see citation‑style metrics akin to those on arXiv or Google Scholar, showing how often my pieces are referenced or recommended by automated agents that subsequently influence human readers.
To answer that question, I envision a table that lists, for each request, the claimed User‑Agent, any verified signer, the documented role of the signer (if known), and any directly observable trigger such as a referrer or query parameter. Most production requests will leave the trigger column empty, which is fine—it signals that further investigation is needed, perhaps via a separate identity study that does not rely on the edge counter as a proxy for reading actions. By keeping the trigger field optional, I avoid over‑loading the classifier with responsibilities it cannot fulfill while still providing a useful starting point for deeper analysis.
Even after applying the two core rules, the “Browser” category still contains a mix of genuine human browsers and sophisticated automation that manages to mimic navigation headers. My fetcher‑header study confirmed that tools like Grok can drive ordinary browsers to emit the exact header combinations my classifier uses to flag human‑like traffic. Consequently, the Browser bucket is not a pure lower or upper bound on human readership; its error can swing in either direction depending on whether it mistakenly excludes a legitimate cloud browser or includes a cleverly disguised script. Recognizing this bidirectional uncertainty prevents me from over‑interpreting the raw count and encourages me to treat the Browser category as a hypothesis that must be continually tested against external signals.
A recent production window added a tangible case for scrutiny: twenty‑two of the ninety‑five remaining Browser observations shared the uniuit.com referrer and repeatedly requested the homepage or a specific article pattern. A Worker log inspection identified a Redroid Android WebView as the likely client for nineteen of those rows, later corroborated by zone‑data correlation. While this cluster suggests a coordinated automated script—perhaps a scraper or a testing suite—it does not, on its own, explain the entire gap between the edge counter and the analytics beacon. Setting those twenty‑two rows aside as a sensitivity test still leaves a ratio of roughly 5.2×, indicating that other sources of discrepancy remain at play, such as differences in session‑ID counting, script‑blocker prevalence, or variations in how prefetches are handled.
I addressed a related symptom—referrer spam—by adopting a versioned reporting policy that hides matching referrer names in public dashboards while preserving the underlying data for internal analysis. This approach, described in my companion piece “How I Defend My Analytics Against Referrer Spam,” ensures that historical comparisons stay consistent: the original September 3 comparison retains its raw scope, but public views no longer amplify noisy referrer domains. The episode also reinforced a broader lesson: any change to classification logic must be accompanied by a careful migration strategy that retains the ability to reconstruct past views, even if those views were based on flawed assumptions. When I first deployed the header and network checks together, the default stats view temporarily showed only three page views for the week, prompting a frantic “why did we lose all the views?” moment. Restoring the historical view by attaching the evidence available at the time of each observation proved essential for maintaining trust in the data.
Reflecting on the misstep of assuming WebViews uniformly lack Fetch Metadata, I reminded myself to consult the existing body of knowledge before forming blanket rules. A 2019 Android WebView issue demonstrated that, post‑Chromium 76, those views do send the expected headers, and a September 2025 Safari compatibility report confirmed that Safari and its WebViews regularly transmit Sec‑Fetch‑Mode while omitting Sec‑Fetch‑User. Armed with that evidence, I rejected the over‑generalized claim and instead focused on investigating the actual behavior of the traffic hitting my site. The raw Cloudflare logs, often overlooked in favor of higher‑level dashboards, became a vital source of ground truth for validating or refuting hypotheses about header presence, network origins, and timing patterns.
Using those logs, I reconstructed a partial picture of network attribution for older rows. By grouping requests by hour, path, country, and device and checking whether all members of the group agreed on an ASN, I could assign a network source with a confidence level derived from sample agreement. Of the nearly two thousand pre‑evidence observations, about 1,429 received an ASN under this sampled method, while the remainder were marked as ambiguous or unmatched. The migration script recorded these attributions as asn_source = ‘zone‑sample’, explicitly acknowledging that the method is a statistical approximation rather than a per‑request join. Later, I discovered that a bug in the INSERT statement had left the asn_source field null for a large batch of subsequent observations, invalidating my earlier claim that every row carried network provenance. A follow‑up repair now writes asn_source = ‘request’ whenever an ASN is available, leaving the field empty only when the information is truly missing.
The net effect of the new rules is striking: they reclassified 277 of the 372 original browser‑UA observations, or 74.5%, yet the remaining count is still uncalibrated and cannot be read as a precise measure of human readership. The comparison window I used—September 4 00:00 to September 6 00:00 UTC, with the end excluded—shows that after the rules, 95 Browser observations remain, compared to 14 page loads recorded by Cloudflare Web Analytics (or 16 when bot‑tagged RUM loads are included). Ratios such as 95/14 ≈ 6.79× or 95/16 ≈ 5.94× illuminate the scale of the discrepancy but do not, by themselves, reveal the underlying mix of humans, bots, and ambiguous agents. What they do provide is a repeatable, evidence‑based starting point for further experimentation, such as invoking controlled clients, examining trace spans, or isolating specific header combinations.
The transferable lesson from this work is straightforward: retain as much raw request evidence as practical, make the reasoning behind each classification visible and queryable, and compare data at multiple collection stages before treating any aggregate as a proxy for human behavior. By following this method, I have turned an opaque counter into a diagnostic tool that lets me ask targeted questions—Why did this request land in the cloud‑browser bucket? Which header caused it to fall back to http‑client? How does the network context shift over time?—and then design the next experiment based on the answers. Readership remains an open question, one that will require complementary techniques such as surveys, first‑party cookies (where privacy‑compliant), or panel‑based measurement, but the edge‑level classification now provides a solid foundation upon which to layer those efforts.
As a final, actionable takeaway, consider implementing a lightweight, rule‑based classifier on your own edge platform—whether Cloudflare Workers, AWS Lambda@Edge, or a similar service. Start by separating traffic based on network provenance (known hosting providers, residential ISPs, corporate ASNs), then apply header‑based sanity checks (Fetch Metadata, Accept, User‑Agent plausibility) that are version‑specific and grounded in RFCs. Store not only the final label but also the rule(s) that triggered it, preserved in a queryable database. Use this enriched dataset to compare against your existing analytics, identify systematic divergences, and iteratively refine your rules. Treat each ratio as a hypothesis, validate it with external evidence (logs, traceroutes, controlled tests), and remember that the goal is not to achieve perfect bot‑human separation but to achieve observable, explainable, and improvable visibility into who—or what—is actually reaching your content.