← back to the garden

evergreen

The Root of Trust Comes Before the Automation

planted · March 8, 2026
#security-first-gcp#security#gcp#cloud#day-0#terraform

Every infrastructure-as-code tutorial starts the same way: write a Terraform file, run apply, watch the cloud fill up with resources. I want to argue that the first real decision on day 0 happens before that file exists, and that getting it wrong quietly undermines everything you build afterward.

The decision is about state. Terraform keeps a record of every resource it manages, and that record is the single most privileged artifact in the whole system. It holds resource IDs, relationships, and often generated values you never see in the console. Anyone who can read it learns the shape of your infrastructure. Anyone who can write it can convince your automation that a resource does not exist, or exists differently, and the next apply will happily reconcile reality to match the lie. The state file is not a build artifact. It is the map of the kingdom, and it needs to be treated like one.

The principle: what secures everything cannot be secured by the same hand

Here is the trap. If your automation manages the very thing that protects its own state, then a bad plan, a fat-fingered destroy, or a compromised runner can delete the protection and the state in one motion. The tool that secures everything must not also be able to delete the thing that secures it. That is the root of trust, and a root of trust you can casually revoke is not a root of anything.

Concretely, this means the store that holds Terraform state, and the key that encrypts that store, have to live outside Terraform. Not in a separate workspace. Outside the automation entirely. They come into existence by a different mechanism, owned by a human, so that no apply anywhere in the tree can reach back and remove the floor it is standing on.

This is the part people skip because it feels backwards. We adopt infrastructure-as-code precisely so that nothing is created by hand, and here I am saying the first resources must be created by hand. The resolution is that these are not application resources. They are the trust anchor, and a trust anchor you can regenerate from automation is just another mutable resource.

The receipt: a bootstrap script, not a Terraform leaf

On GCP the real setup looks like this. There is one plain shell script, run once by an operator, that creates exactly the pre-state minimum and nothing more:

  1. The project and its billing link.
  2. A small set of baseline APIs, only enough to create a key and a bucket.
  3. A KMS keyring and key, dedicated to encrypting state.
  4. The state bucket itself, encrypted with that key, with versioning and a soft-delete window turned on.

That is the whole script. Every step is idempotent, so re-running it detects what already exists and skips it, which matters because a bootstrap you are afraid to run twice is a bootstrap nobody maintains.

The chicken-and-egg problem is the reason this cannot be Terraform. Terraform state needs an encrypted bucket to live in, but the bucket, the key, and the project cannot themselves live in a state that does not exist yet. So bash creates that floor with idempotent CLI calls, and only then does Terraform take over. The header comment on the script says it plainly: the project, the key, and the bucket stay out of Terraform forever, because Terraform must never be able to destroy the key or the bucket encrypting its own state.

The key detail carries most of the weight. The state bucket is encrypted with a customer-managed key, not the provider's default encryption. Default encryption is real encryption, and for most buckets it is entirely fine. But a customer-managed key gives you something default keys cannot: the key is a resource you control, with its own IAM, its own rotation schedule, and its own audit trail. Only the storage service's own agent is granted encrypt and decrypt on it, and that grant is the narrow bridge that lets the bucket function at all. The key rotates on a fixed schedule so that no single key version protects state forever. If the key is ever destroyed, the state is unrecoverable, which is exactly why it is not something an automated destroy can touch.

Two more properties on the bucket earn their place. Versioning means an overwrite or a corrupt apply does not erase history; the previous state object is still there to roll back to. The soft-delete window means even a deletion has a grace period before it is permanent. Both exist so that the worst case, an apply that mangles state, is recoverable rather than terminal.

One thing I do not have to configure at all is locking. The GCS backend takes a lock on the state object automatically while an apply runs, so two concurrent applies cannot stomp on each other, and there is no separate lock table to stand up, unlike the S3-plus-DynamoDB pattern the AWS twin has to wire by hand. It is one less moving part in the trust anchor.

One structural choice is worth calling out because it is easy to get wrong. There is a single state bucket for the entire tree, and every component gets its own isolated slice of it through a distinct prefix. Bootstrap, the shared platform, each application: separate state, one bucket, one key. That keeps the trust anchor singular, which is the point. You want one root to reason about, not a scattered handful.

What this buys, and what it costs

The payoff is a clean seam. On one side, a tiny amount of human-run bash that establishes trust and is not expected to change. On the other side, everything else, fully automated, unable to harm the ground it runs on. When you later hand applies to a CI runner (the subject of the next part), the worst a compromised runner can do is bounded, because the runner literally cannot reach the key or bucket that anchors the system. The blast radius was decided here, on day 0, before the first real resource existed.

The cost is honest and small. You run a script by hand once, and you accept that four resources are not captured in your infrastructure-as-code. That offends the purist instinct that everything should be codified. I think the instinct is right almost everywhere and wrong here. The one thing you must not codify into a tool is that tool's own ability to delete its foundation. A short, readable, idempotent script that a human runs deliberately is not technical debt. It is the deliberate boundary that makes all the automation above it safe to trust.

Start here, and everything downstream inherits a floor it cannot fall through. Skip it, and you will not notice until the day an apply goes wrong and takes the map with it.


Next in this series: Identity Before Infrastructure.

The same decision on AWS: The Root of Trust Comes Before the Automation.