When The WAF Bit Back

For a stretch this week, some of you tried to visit icecreamboets.com and got served a 134-byte page that said, in its entirety, 403 Forbidden. No header, no apology, no contact form, no map of the trucks, no information about anything at all. Just the polite-but-firm digital equivalent of being asked to leave a club you have not entered yet.

We are sorry. It was our fault. Here is the whole story, including a tour of the part of the internet that did the rejecting, because we think the explanation is more interesting than the apology.

The accident

Someone close to the team noticed it first. The home page would sometimes render, sometimes not. Sub-pages 403’d. Clearing cookies did nothing. Switching Chrome profiles changed the failure rate. It looked random, and random is the worst kind of bug, because random means it is almost certainly your own infrastructure and not the internet at large.

A capture of one of the failing requests was what gave it away. The bad response had no server header and no x-cloud-trace-context. The next request on the same connection — a request for /favicon.ico — sailed through with server: Google Frontend and a trace id. Two requests, same TCP/QUIC connection, same client, two completely different layers answering.

The favicon was reaching our actual application. The home page was not.

A small tour of the front of our house

Before we explain what was going wrong, it is worth saying what is even there. The site you are reading right now is hosted on a stack that, when written out, looks like this:

   you                                                                you
    |                                                                  ^
    v                                                                  |
  +----------------------+      QUIC / HTTP3      +-------------------+
  |  Google Cloud HTTPS  |  ------------------>   |  Google Frontend  |
  |   Load Balancer      |                        |   (Cloud Run)     |
  +----------+-----------+                        +---------+---------+
             |                                              |
             | every request is first evaluated by          |
             v                                              v
  +----------------------+                        +-------------------+
  |    Cloud Armor       |                        |   Astro 6 SSR     |
  |  security policy     |  -- if denied -->  X   |   container       |
  +----------------------+    (134 bytes,         +-------------------+
                              no headers)

A request from your browser arrives at a Google-anycast IP. That IP belongs to a Cloud HTTPS Load Balancer. The load balancer hands the request to a Cloud Armor security policy Google Cloud · 20 May 2026 Google Cloud Armor overview cloud.google.com/armor/docs/cloud-armor-overview Paraphrased: Cloud Armor provides DDoS and application-layer protection in front of services exposed through Google Cloud's external load balancers, with security policies that attach to backend services. for inspection. If the policy says allow, the request is forwarded into a serverless network endpoint group Google Cloud · 20 May 2026 Serverless network endpoint groups overview cloud.google.com/load-balancing/docs/negs/serverless-neg-concepts Paraphrased: A serverless network endpoint group lets an external HTTPS load balancer route traffic to a Cloud Run, App Engine, or Cloud Functions service, which is how Cloud Armor policies are attached in front of serverless backends. which routes it to our Cloud Run service Google Cloud · 20 May 2026 What is Cloud Run cloud.google.com/run/docs/overview/what-is-cloud-run Paraphrased: Cloud Run is a managed compute platform that runs container images on Google's infrastructure, scaling to zero when idle and out to many instances under load. — the container that actually renders the page. If the policy says deny, the load balancer answers the browser itself, with the world’s most minimal HTML document.

We brought up this stack so that the site could survive everything from a hug of death to an actual attack. Most of the time it sits there doing nothing. Once in a while, it does its job a little too well.

What Cloud Armor is and why we have one

Cloud Armor is Google Cloud’s web application firewall. It attaches to backend services as a security policy Google Cloud · 20 May 2026 Google Cloud Armor security policy overview cloud.google.com/armor/docs/security-policy-overview Paraphrased: A Cloud Armor security policy is an ordered list of rules attached to one or more backend services; each request is evaluated against the rules in priority order until one matches. — an ordered list of rules, each of which either lets a request through, denies it, throttles it, or sends it to a captcha. Rules can be hand-written, or you can switch on one of the preconfigured rule banks Google Cloud · 20 May 2026 Tune Cloud Armor preconfigured WAF rules cloud.google.com/armor/docs/rule-tuning Paraphrased: Preconfigured WAF rules are sourced from the OWASP Core Rule Set and exposed in Cloud Armor as named signature banks that can be evaluated with a sensitivity level from 0 to 4. that ship with the product.

The preconfigured banks are the OWASP Core Rule Set OWASP CRS Project · 20 May 2026 OWASP CRS Project coreruleset.org Paraphrased: The OWASP Core Rule Set is a set of generic attack-detection rules for ModSecurity-compatible web application firewalls, designed to protect against the OWASP Top 10 and related web application threats. . CRS is a community-maintained collection of generic detection rules for the kinds of attacks listed in the OWASP Top Ten OWASP Foundation · 20 May 2026 OWASP Top Ten owasp.org/www-project-top-ten Paraphrased: The OWASP Top Ten is a community-maintained list of the most critical security risks to web applications, updated periodically and widely used as a baseline for application security programs. — SQL injection, cross-site scripting, remote-file-inclusion, path traversal, scanner fingerprints, header smuggling, the lot. If a request looks too much like one of these attacks, the bank fires and the policy decides what to do about it.

It is a very, very good idea to have one of these in front of a site. It is also the kind of thing that can have a bad day.

Why the bouncer mistook you for a hooligan

CRS organises its rules into paranoia levels OWASP CRS Project · 20 May 2026 Paranoia Levels coreruleset.org/docs/2-how-crs-works/2-2-paranoia_levels Paraphrased: CRS groups rules into paranoia levels from 1 to 4. Higher levels catch more attacks but generate more false positives; level 1 is the baseline recommended for most production sites. , from one to four. Level one is the baseline — broad strokes only, the rules with the lowest false-positive rate. Level four is the deepest end of the pool, and is designed for sites that would rather block a real visitor than let a real attack through. Most production deployments live at level one. Ours did.

The problem is that even level one has opinions. A reasonable, well-mannered Chrome browsing from a freshly opened profile to a marketing site can, depending on which extensions are installed, which language preferences are in the request, and what shape the accept-encoding line happens to be that day, look enough like the leading edge of a scanner to score above a threshold somewhere. The header combinations are not malicious. They are just unusual. CRS does not get to distinguish unusual from suspicious without help — that is what tuning is for.

In our case, the home page request was the one that crossed the threshold. The favicon was not. Static-asset paths were explicitly exempted in our policy, which is why the icon loaded while the page itself was denied on the same connection. The Cloud Armor response is exactly what we saw in the request capture: a tiny text/html body, served by the load balancer itself, with neither the server: Google Frontend line that a Cloud Run response would carry nor any trace identifier to grep on. A 403 with no return address. The signature of a WAF block.

It is not a bug. It is a very specific kind of false positive, and it is the WAF’s most common one.

What we did about it

The first move was to stop blocking real visitors. Cloud Armor supports a per-rule preview action Google Cloud · 20 May 2026 Monitoring Cloud Armor security policies cloud.google.com/armor/docs/monitoring Paraphrased: When a rule is set to "preview" the action is not enforced; the matching request is logged as a "Previewed Request" so operators can baseline rule behaviour against real traffic before turning enforcement on. . A rule in preview still evaluates; if it would have matched, the policy writes a log entry saying so, but the request is not blocked. It is the WAF equivalent of putting the bouncer in observer mode for the night so we can find out exactly who was getting bounced and why.

The change is one line in the Terraform that owns the policy:

module "icecreamboets_armor" {
  source = "git::ssh://...//modules/service/security/cloud-armor-cloudrun?ref=v1.5.5"

  // OWASP rules in PREVIEW (log-only) while we triage false-positives.
  // Customer reports of intermittent 403s on the home page (2026-05-20)
  // — favicon and /_astro/* are exempted via framework_allow_paths and
  // load fine on the same connection, confirming an OWASP CRS rule is
  // firing on root navigation. Flipping to preview restores traffic;
  // re-enforce after building an `excluded_rules` list from the
  // resulting preview logs.
  owasp_preview     = true
  owasp_sensitivity = 1

  // ...rate limiting, allow-lists, framework path passthrough...
}

We ran tofu apply. Cloud Armor security policies propagate at the edge in seconds. Five fresh probes against the home page came back 200 200 200 200 200. Visitors are unblocked.

This is not the end of the story. Preview mode is a triage tool, not a destination. Leaving CRS in preview forever would mean leaving the actual attacks it is meant to catch un-blocked too. The next step is to read the preview-mode logs, identify the specific rule ids that were firing on the home page, write a tightly-scoped exclusion for those rules — excluded_rules in CRS-speak — and then flip the rest of the bank back to live enforcement. That work happens over the next week.

A small word about whose site this is, anyway

You will have noticed that this post sounds slightly less like a sentence about ice cream and slightly more like a sentence about backend systems. That is because the site itself has just been handed over to a sister team, Protoworks, who are rebuilding it from the ground up — atomic design system, accessibility-first components, modern Astro 6 SSR Astro Docs · 20 May 2026 Content collections docs.astro.build/en/guides/content-collections Paraphrased: Astro content collections organise content from local files, remote sources, or live APIs behind a typed loader, with optional Zod-validated schemas, exposed through getCollection and getEntry at both build and request time. , the works. Same Boets. Same trucks. Same ice cream. Slightly fancier internals.

They are the ones who put the bouncer at the door. They are also the ones who pulled him back when he got a bit keen. They are, as we like to say, our boets too.

The apology, properly

If you came here looking for an ice cream and got told you were forbidden, we are sorry. The truck was not closed. The site was not down. You were not blocked. Our front-door security guard read the shape of your request and made a wrong call, and that wrong call is on us, not on you.

We have changed his settings. We will tune them, properly, this week. And the next time we make a change at the edge of the network, we will run it in preview for a few days first — exactly like we are doing now — instead of trusting that the defaults will be kind to everyone.

Thank you for putting up with us. Come back tomorrow. The 200s are flowing.