There is a temptation, on a fresh AWS account, to start with the interesting thing. The API. The bucket that holds the product. The function that does the work. That is the second thing. The first thing is less exciting and more important: a safe place to keep the record of everything you are about to create.
This is the root-of-trust idea, and it is worth stating plainly before any service enters the conversation. Infrastructure-as-code has a memory. That memory is the state file: a description of every resource, its identifiers, and often its secrets. Whoever can read the state can read a map of your account. Whoever can silently corrupt the state can make your automation destroy the wrong thing on the next run. So the state is not a byproduct of the build. It is the most sensitive artifact in it, and it deserves to be the first thing you make defensible.
What "defensible state" actually means
Three properties, and none of them are optional.
It survives. A single local terraform.tfstate on your laptop is one failed disk away from an account you can no longer manage by code. Remote storage, with versioning turned on, means the record is durable and every past version is recoverable.
It cannot be read by accident. The state describes private resources and can contain sensitive values inline. It should be encrypted at rest and closed to public access completely, so a misconfigured permission somewhere else never turns your infrastructure map into a public document.
Two runs cannot race. This is the one people skip. If two applies run at the same time, against the same state, they interleave writes and you get a corrupted record that no longer matches reality. You need a lock: a way for one apply to say "I am working on this" and for the second to wait rather than trample it.
Notice that I have not named a product yet. These are the requirements. The service is only the means.
The means, on AWS
Terraform's S3 backend covers all three. The record lives in an S3 bucket, and the backend block is where you pin the safety properties:
terraform {
backend "s3" {
bucket = "my-project-terraform-state"
key = "my-project/terraform.tfstate"
region = "ap-south-1"
encrypt = true
use_lockfile = true
}
}
encrypt = true forces server-side encryption on the state object. On the bucket itself I turn on versioning, so every apply leaves the previous state recoverable, and the full public-access block, so the bucket is closed at every level regardless of what any future policy tries to do.
use_lockfile = true is the part I want to dwell on, because it is a recent and genuinely nicer answer to the race problem. For years the standard pattern was a separate DynamoDB table that Terraform would write a lock item into before an apply and delete after. It worked, but it was a second resource to create, to grant permission on, and to remember. S3 now supports conditional writes, so the lock can be a small object in the same bucket as the state. One store, not two. If you look at older setups of mine, you will still find the DynamoDB lock table in the CI permissions, left over from before the migration. That is honest history: the lock moved, the grant that once fed it took a while to be cleaned up.
The gotcha the brief on my own site tells on itself about
Here is the uncomfortable receipt. The very site you are reading this on does not do this yet. Its Terraform still keeps state locally, with the S3 backend sitting commented out in the provider file:
# Uncomment below to use S3 backend for state management
# backend "s3" {
# bucket = "your-terraform-state-bucket"
# ...
# }
That is the gap between knowing the principle and having spent the twenty minutes to apply it. A single-maintainer static site is the lowest-stakes place for local state to exist, which is exactly why it survived. But it is still debt. The moment a second person needs to apply, or the laptop changes, that convenience becomes a problem I have to solve under pressure instead of on day 0. The project where I did it right, a small scanning API, has the encrypted, locked, versioned backend from its first commit, and I have never once thought about state since. That is the tell. Done on day 0, this decision disappears. Deferred, it waits for you at the worst moment.
Why the state store lives slightly apart from the build
There is a subtle ordering problem that the root-of-trust framing solves cleanly. The bucket that holds your state cannot itself be fully managed by the Terraform whose state it holds, because that Terraform would have to exist before its own backend does. You resolve this by treating the state store as bootstrap: created once, by a small separate step or by hand, and then left alone. It is deliberately not in the blast radius of the everyday automation.
That separation is the whole point. You do not want the tool that can delete a hundred resources in one command to also be able to delete the record of what those resources are. Keep the thing that remembers slightly out of reach of the thing that forgets. If an apply goes catastrophically wrong, the state, its history, and the ability to recover are all on the other side of a wall from the mistake.
What this buys you
The payoff is quiet, which is why it is easy to undervalue. When state is encrypted, versioned, and locked from the start:
- A bad apply is recoverable, because the previous state version is one command away.
- Two people, or a person and a CI pipeline, cannot corrupt the record by running at the same time.
- The most sensitive description of your account is not sitting in plaintext on a disk or, worse, in a readable bucket.
None of that is a feature you demo. It is the floor everything else stands on. Every later episode in this series, least-privilege identity, deny-by-default networking, encryption defaults, the audit trail, assumes the record of what you built is itself trustworthy. If the root is soft, the rest is theatre.
So before the interesting resource, make the boring one. Encrypt it, version it, lock it, and keep it a step removed from the automation that could otherwise take it down with everything else. It is the smallest first move in the whole build, and it is the one that makes every move after it safe to get wrong.
Next in this series: Least Privilege Starts at Resource One
The same decision on GCP: Secure the State Before You Secure Anything Else