There are two separate promises hiding under the word "monitoring," and conflating them is how teams end up with dashboards nobody looks at and no phone that ever rings. The first promise is detection: when the thing breaks, a human finds out without having to be watching. The second is prevention that holds: the safety rail you set on day 0 is still there on day 90, after a dozen hasty changes that each had a reason to remove it.
This is part seven of building a GCP environment security-first. The record of who-did-what is on and queryable. Those answer questions after the fact. This part is about the present tense, knowing now, and about policy that does not quietly rot.
The floor: wake me if it's down
The honest label for the alerting I run is the "wake me if it's down" tier. It is not analytics, it is not anomaly detection, it is not clever. It is the floor, and having the floor beats having an elaborate ceiling you never finished building.
The coordinator already emits rich signals on its own: structured request logs, per-node join and leave events, and a telemetry line every ten seconds carrying open connections, active nodes, event-loop lag, and memory. All of it ships to Cloud Logging. That is a lot of signal, and none of it, by itself, tells a human anything, because nothing was watching it. Signal that no one is subscribed to is not monitoring. It is just logs.
So the floor adds three pieces, and they are worth naming by function because the functions port to any cloud:
An external uptime check. A probe hits the coordinator's /healthz from multiple locations around the world, every minute, over TLS. Crucially it runs from outside the system, so it exercises the whole path, the CDN, the load balancer, Cloud Armor, and the VM behind them, not just the box. A health check that runs inside the thing it is checking cannot tell you the edge is broken. And it is not satisfied by a bare 200: it matches on the body, confirming the coordinator's health endpoint actually returned its "status":"ok", so an edge error page that happens to carry a 2xx does not read as healthy. The strength of a check is in what it refuses to accept as "up."
An alert policy with a deliberately dumb threshold. The incident opens when the check fails from more than one location, not one. A single failing probe is usually a flaky probe, a transient network hiccup at one vantage point, and paging on it trains you to ignore the pager, which is the worst outcome. Requiring multiple locations to agree is the difference between a signal and noise. The incident auto-closes once the check is healthy again, so recovery does not leave a stale page to clear by hand.
Notification channels. Email always, and a Slack channel the moment its id is supplied. The interesting detail here is a small honesty about tooling: the Slack integration is created through a Console OAuth flow that keeps its token internal, so Terraform cannot cleanly create that channel. It references the Console-created channel by id instead. That is a real seam between infrastructure-as-code and a thing IaC cannot own, and pretending it does not exist is how you get a plan that never converges. Name the seam, reference across it, move on.
That is the whole floor. No dashboards yet, no log-based metrics yet. I would rather ship the tier that makes a phone ring than stall on the tier that makes a pretty screen.
The natural next rung, alert on the audit trail
The floor watches whether the service is up. It does not watch whether the service is safe. Those are different questions, and the bridge between this part and the two before it is short: the same alerting machinery that pages on a failed health check can page on an audit event.
In part six I built a query that answers "every read of the one secret." The active version of that query is an alert: fire when a principal that is not the coordinator reads the Turnstile secret. Same for IAM, fire when a SetIamPolicy happens outside a known change window. This turns the audit trail from a thing you have to remember to interrogate into a thing that taps you on the shoulder. I am honest that this rung is designed, not yet fully shipped, because it depends on the durable store from part six being in place first, and it is the most valuable place to spend the next unit of effort. A record that alerts is worth ten records you have to remember to read.
Guardrails that survive the next hasty change
Detection is half. The other half is the promise that the safety decisions I made on day 0 are still standing after the project has been through real churn. A guardrail that any hurried change can silently remove is not a guardrail, it is a suggestion.
The security-first move is to encode the rails as policy and structure, so that removing one is loud, deliberate, and visible in review, never a quiet side effect. A few that are actually in place:
Org policy above the project. The organization enforces a domain restriction on IAM members, so by default no principal outside the org's own domain can be granted anything. That rail lives above my project and is not something a change inside the project can casually undo. Where I genuinely needed to relax it, to let public traffic reach a public service through the load balancer, I did not just flip it off. I wrote an explicit, project-scoped override with the compensating control stated next to it in the code: the public entrypoint is only the load balancer and Cloud Armor, and the workload behind it has no path from the internet except through that edge, by an ingress that accepts load-balancer traffic only, or, once it became the VM that replaced the service, the absence of any public IP at all, so the edge cannot be bypassed. The relaxation and the reason it is safe live in the same file. Anyone who reads the grant reads why it is not an open door.
Belt-and-suspenders deny at the network. The firewall carries an explicit deny-all sitting on top of GCP's implied deny, precisely so that it is audit-visible and survives a future stray allow rule created above it. The implied deny is invisible and easy to forget. The explicit one shows up in every plan and every audit, which is the point: a rail you can see is a rail someone will think twice before removing.
Destroy that cannot cascade. APIs are configured to not disable on destroy, deliberately, because disabling an API tears down its service agents and can orphan resources owned by other parts of the system. A teardown of one leaf must never be able to take the platform down with it. That is a guardrail against a mistake I could plausibly make at 2am.
The powerful identity that does not exist yet. The keyless CI runner and its federation are written and deliberately left disabled, so there is no idle service account sitting around holding editor-level roles waiting to be misused. The rail here is absence: the most powerful principal in the system is one I have chosen not to instantiate until CI actually needs it. You cannot misuse an identity that was never created.
The common thread is that each of these is either enforced from a layer the routine change cannot reach, or made so visible that removing it is a conspicuous act rather than an accident.
The honest gap
I will not pretend this is complete, because the scorecard is next and it deserves the full accounting. But the shape of the gap belongs here too. What I have are project-level guardrails plus one org-level restriction I happen to inherit. What I do not have is a real org-wide guardrail posture: a folder hierarchy with policies applied by structure, a consistent set of constraints every future project gets for free, config-drift detection that tells me when live state has wandered from the code. Single project today means the guardrails are hand-placed rather than inherited, and hand-placed rails are exactly the ones a hasty change can pull. There is also no threat-detection layer watching for the patterns a human would miss. Those are real absences, named honestly, and the next part scores them without flinching.
Previous: making the record queryable. Next: the honest scorecard.
The same decision on the other cloud: alerting and guardrails on AWS.