There is a question every account owner eventually gets asked, usually at the worst possible time. Who changed that policy. Who read that secret. Who deleted that bucket, and when, and from where. The uncomfortable part is that the answer does not exist unless you decided, in advance, to record it. An audit trail is retroactive by nature. You cannot audit a week you never wrote down.
This is the control that almost everyone enables the day after the incident. I want to make the case for enabling it on day 0 instead, and then actually build it, because in my real infrastructure it was the honest gap. Everything else in this series was already running. This one was not.
Application logs are not an audit trail
The thing worth getting straight first, because it is the mistake I see most often, is that having logs is not the same as having an audit trail.
My infrastructure already logged plenty. Every Lambda writes to its own log group. The API gateway records access logs. I had imported the auto-created log groups just to force a bounded retention on them, thirty days, so they would not accumulate forever. The execution logging was deliberately capped at error level with no data trace, precisely so it would never record an API key header or the contents of an uploaded file. That is careful, real logging, and I wrote about the redaction decision in the previous part.
But all of that is data-plane logging. It answers "what did my application do." It tells you a request came in, a scan ran, a response went out. It says nothing about the control plane, the account itself. If someone assumes a role, edits an IAM permission, disables a trail, pulls a secret, rotates a key, or deletes a resource, none of my application logs would show a trace of it. The identity acting on the account is invisible to the logs of the things the account runs.
That is the distinction that matters. Application logs answer what your code did. An audit trail answers what any identity did to the account, using which credentials, from where, at what time. The first is about your software behaving. The second is the only thing that helps when the question is about a person, an automation, or a leaked key.
A grep through my Terraform confirmed the gap plainly. No trail resource anywhere. The control plane was unrecorded. If a key had leaked, I would have had no way to reconstruct what it touched.
What I am turning on
On AWS the audit trail is a managed service that records control-plane API calls across the account. Turning it on is a few resources. Turning it on well is a set of decisions, and the decisions are the point.
Make it account-wide and multi-region. A trail that only watches the region you use leaves every other region as a blind spot. An attacker with a leaked key does not care which region you consider home. They will spin something up wherever monitoring is thinnest. So the trail covers all regions, including the ones I never intend to touch. Especially those.
Record management events, and choose data events deliberately. Management events are the control-plane actions: policy changes, role assumptions, trail configuration, key operations. They are effectively free and there is no reason not to capture all of them. Data events are per-object activity, every object read, every function invoke, and those cost real money at volume. So I do not log all of them. I log the ones that would matter in an investigation: reads of the secret, object access on the sensitive bucket. Logging everything is not more secure, it is just a larger bill and a noisier haystack.
Turn on log file validation. This is the part people skip and it is the part that makes the trail worth trusting. Validation produces signed digest files, so you can later prove a given day of logs was not altered or deleted after the fact. Without it, a sufficiently privileged attacker could edit the record of their own activity. Tamper-evidence is what separates a trail you can rely on in an investigation from a pile of JSON you hope nobody touched.
Land it in a dedicated, locked-down bucket. The trail delivers to object storage, and that bucket gets the full deny-by-default treatment from earlier in this series: public access blocked at every level, its own encryption, and a bucket policy that only lets the trail service write to it. The delivery role is scoped to exactly that, nothing broader. The record of who-did-what is itself a target, so it is stored more carefully than the things it records.
Why day 0 and not day N
The reason this control gets deferred is honest and worth naming: it produces nothing you can see. It ships no feature. On the day you enable it, it changes precisely nothing about how your system behaves. It sits there, quietly writing down events you hope you will never need to read.
And then one day you do need to read them, and the only version of this decision that helps is the one you made months earlier. The trail you enable during an incident starts recording from that moment forward. It cannot tell you what happened last week, or how long the leaked key had been in use, or which resources were touched before you noticed. All of that lives in the history you either captured or did not.
That is the whole argument. This is not a control that gets better with urgency. It is a control whose entire value is the history it has already accumulated by the time you have a question. Turning it on today is cheap. It is a handful of Terraform resources and a few dollars a month. Turning it on after you needed it is free of charge and worth nothing, because the window you cared about is already gone.
What I have, and what is still missing
So the trail is on. It is account-wide, multi-region, validating its own integrity, and delivering to a bucket that is locked down harder than anything else I run. If a credential leaks tomorrow, there will be a record of what it did.
But right now that record is a growing pile of compressed JSON in object storage, partitioned by date, completely durable and completely unreadable by a human under pressure. Having the data and being able to ask it a question are two different things, and the second one is the part that actually gets used at 2am. That is the next build.
Previous: Encrypt at Rest, Redact in the Logs Next: Making the Record Answerable
The same decision, the other cloud: The Record of Who Did What on GCP