I wanted to track status changes on a public-sector case-lookup portal – the kind of site where you punch in a reference number and it tells you where things stand. No official app for it, just a form on a government website. Classic weekend homelab project: poll the page, diff the result, push a notification when it changes.

It took about twenty minutes to prove the plan wouldn’t work as designed, and most of a day to find out why – and what to do instead.

The First Wall

curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 \
  (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36" \
  -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" \
  -H "Accept-Language: en-US,en;q=0.9" \
  -H "Origin: https://example-case-portal.gov" \
  -H "Referer: https://example-case-portal.gov/lookup" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  --data "receiptNum=EXAMPLE0000000000" \
  https://example-case-portal.gov/lookup

HTTP/1.1 403 Forbidden

<h1 data-translate="block_headline">Sorry, you have been blocked</h1>
<h2 class="cf-subheadline">You are unable to access example-case-portal.gov</h2>
<p data-translate="blocked_why_detail">This website is using a security service to
protect itself from online attacks...</p>

A full, realistic header set – correct User-Agent, Accept, Accept-Language, Origin, Referer – and an outright Cloudflare block. My first assumption was that the POST body looked suspicious somehow. So I tried the “polite” version: GET the page first to pick up a session, then POST with those cookies, like a real browser would.

curl -c cookies.txt <headers> https://example-case-portal.gov/lookup   # GET
curl -b cookies.txt <headers> -X POST ... https://example-case-portal.gov/lookup  # POST

Both 403. The block fired on the bare GET, before any form data was involved. Whatever was flagging me, it wasn’t the request content.

Headers Are Not Identity

The instinct here is “just spoof more headers.” I’d already spoofed all of them. The problem is that headers are a text layer sitting on top of a connection whose shape gives away the client long before any header is read:

  • TLS fingerprint (JA3/JA4). The TLS ClientHello – cipher suite list and order, extensions, supported curves – is characteristic of the TLS library doing the handshake. Chrome’s BoringSSL produces a different fingerprint than curl’s OpenSSL or Python’s ssl module, regardless of what the User-Agent header claims two layers up. There is no header for “make my TLS handshake look like Chrome’s.”
  • HTTP/2 fingerprint. Frame ordering and HPACK header-compression behavior differ by client library the same way.
  • JS-executed proof-of-work. Some challenges run JavaScript client-side and only clear you once it passes. A header-only client has no JS engine and can never earn that pass on its own.

The response I got wasn’t even the interactive “checking your browser” interstitial – it was Cloudflare’s hard WAF verdict page, a firewall rule rejecting the connection outright. That’s a stronger signal than “add better headers and retry.” It meant the fix, if there was one, had to happen below the HTTP layer entirely.

Escalating to a Real Browser

Headless Playwright was the obvious next step – a real browser has a real TLS stack and a real JS engine.

resp = page.goto("https://example-case-portal.gov/lookup",
                  wait_until="domcontentloaded")
print(resp.status, page.title())
# 403, "Just a moment..."

Progress, technically: “Just a moment…” is Cloudflare’s interactive challenge page, a step up from the hard block. But polling the title for fifteen seconds, it never moved. Inspecting the DOM explained why:

<input type="hidden" name="cf-turnstile-response"
       id="cf-chl-widget-j58ov_response" value="">

Cloudflare Turnstile. It runs invisible checks and auto-passes real browsers, but it’s specifically built to detect automation frameworks and either fail them or fall back to an interactive solve. I tried the first, cheapest tier of evasion – disabling AutomationControlled, patching navigator.webdriver to undefined, setting a realistic viewport and locale:

browser = p.chromium.launch(
    headless=True,
    args=["--disable-blink-features=AutomationControlled"],
)
page.add_init_script(
    "Object.defineProperty(navigator, 'webdriver', {get: () => undefined});"
)

Same result. Stuck on “Just a moment…” indefinitely. Turnstile is built to catch exactly this tier of patch – it wasn’t going to be that easy, and treating it like a puzzle to solve with more evasion flags was already the wrong framing.

The Real Fork in the Road

At this point the honest options were, in order of how far they go from “run a browser” toward “actively defeat a security control”:

  1. A non-headless browser via a virtual display – cheap to try, still just “a real browser,” not guaranteed to help.
  2. Deeper stealth-patch bundles – more effort, and Turnstile is explicitly built to catch most known patches.
  3. Solve the challenge by hand once, capture the session cookie, reuse it until it expires, repeat manually.
  4. A paid CAPTCHA-solving service. Reliable, but now you’re paying money specifically to defeat a security control the site operator chose to deploy.
  5. Stop scraping. Find another way to get the same data.

Option 4 is where a lot of scraping guides quietly land, and it’s worth naming what that actually is: a deliberate, ongoing decision to pay a third party to break an anti-automation control on someone else’s infrastructure. That’s a different category of engineering decision than “add a realistic header,” and it deserves to be made on purpose, not slid into because the first four attempts didn’t work.

So I looked for option 5 instead – and it turned out to already exist.

The API I Should Have Looked for First

The portal in question has a public-facing consumer site (the one throwing Turnstile at me) and, separately, an official developer API program for exactly this use case – OAuth2 client-credentials auth, a documented request/response schema, generous rate limits, sandbox and production tiers, the works. It’s not hidden, exactly. It’s just not what shows up when you search for “how to check my case status,” because it’s aimed at integrators, not end users, and most people never need to look past the consumer page.

None of the scraping problems above apply to it. No TLS fingerprinting, no Turnstile, no arms race – it’s a REST endpoint with a token.

The lesson isn’t “always find the API, scrapers are bad.” It’s narrower and more useful than that: organizations that harden their consumer-facing site against bots often also run a completely separate, sanctioned integration path for the same data, because they’d rather issue you a key with a rate limit than have you disguised as a browser hammering the page real users use. If a site is fighting you this hard, that’s a decent signal an official path exists – and it’s worth ten minutes of searching before the first curl command, not after the fourth escalation.

One More Thing the API Didn’t Fix

Even with clean, sanctioned API access, polling on any interval is still polling. If the source system already offers a push notification – email, webhook, SMS – for the same event, no amount of polling frequency beats it. Push fires the instant the backend records the change. Poll finds out, on average, half a polling interval late, no matter how good the underlying access is.

I’d been assuming “build an automated checker” was inherently more real-time than “subscribe to an alert.” It’s the opposite, structurally. The only things polling can add over a push notification you already trust are things push doesn’t give you at all – a persisted history, a dashboard across multiple items, routing into your own notification stack. If none of those are actually needed, the scraper (or the API client) isn’t solving a problem; it’s re-solving one that was already solved, more slowly.

Lessons

1. A header is not an identity. TLS and HTTP/2 fingerprints, and JS-executed proof-of-work, all sit below the layer headers operate at. If a “realistic User-Agent” isn’t getting you past a 403, the fix isn’t a better header – it’s a different client entirely.

2. Diagnose which wall you’re hitting before trying to climb it. A hard WAF block and an interactive JS challenge look similar (both 403, both “blocked”) but call for different responses. One means “this client will never pass.” The other means “this client might pass with the right browser behavior.”

3. Expect the cheap evasions to be the first ones caught. navigator.webdriver patches and automation-flag toggles are the most common headless “fixes” on the internet, which also makes them the most heavily fingerprinted. If a protection service is worth deploying, it’s worth building for the well-known bypasses.

4. Look for the sanctioned integration path before building the unsanctioned one. A site that fights bots hard on its consumer page is often more likely, not less, to have an official API elsewhere – they’d rather hand out a rate-limited key than fight an arms race with every hobbyist. Ten minutes of searching beats a day of fingerprint evasion.

5. Push always beats poll, structurally. No polling interval, however fast, beats an event-driven notification for the same event. Before automating a poll loop, check whether the thing you’re polling for is already something you’d get told about directly – and if so, build for the gap that’s actually left, not the one you assumed existed.