At the end of the last part I had turned on the account-wide audit trail. It was recording every control-plane action, across every region, validating its own integrity, and landing in a locked-down bucket. That is a real accomplishment and it is also, on its own, close to useless.
Because what I had was a bucket filling with compressed JSON, partitioned by date, one object after another, growing every hour. It is durable. It is tamper-evident. And if you handed me that bucket and asked "did anyone read the secret last month," I would have no honest way to answer except to download gigabytes of files and grep through them by hand. A record you cannot question is a receipt you filed without ever reading. It feels responsible and it helps nobody.
This is the part almost nobody builds on day 0, and I want to be straight about why. It is the genuine greenfield layer. It has no payoff on the day you build it. It only pays the day you have a question, and by then, if you skipped it, you are building it under pressure with the incident clock running.
The console is not the answer
AWS gives you a console view of recent events, and it is fine for a spot check. It is also a trap if you mistake it for the audit layer. The event history only reaches back ninety days, and it filters weakly. "Show me every access to this one secret across the last six months" is not a question it can answer, because the data older than ninety days is not there and the query it offers is not that expressive.
The full history is in the bucket. All of it, as far back as the trail has been running. The console just cannot reach into it. So the job is to put a real query layer over the bucket, one that can read the entire history and answer a precise question in seconds.
Schema on read, over object storage
The tool that fits the shape of this problem is a serverless query engine that reads directly from object storage and charges per amount of data scanned. No cluster to run, no server to keep warm, no database to load the logs into. You define a table that describes the shape of the JSON, point it at the trail's prefix in the bucket, and query it with ordinary SQL. The engine reads the files in place.
Two design decisions carry almost all the weight here.
Partition by date, and by region. This is the cost lever and it is not optional. The trail already lays its files out in a date-based path structure. If you define the table so the query engine understands that layout, then a query scoped to "last week" reads only last week's files. If you do not, every single query scans the entire bucket, the whole history, every time, and both your bill and your patience run out fast. Tight partitions turn a full-history scan into a read of a single day. On a security question, you almost always know roughly when, so you almost always benefit.
Model the nested identity block honestly. The hardest part of the schema is that the record of who acted is deeply nested JSON. Whether the caller was a user, an assumed role, a federated session, or a service, and what session it came from, all lives in a nested structure. Getting that structure right in the table definition is what lets you ask "which role did this" rather than just "which raw API call happened." It is fiddly, and it is worth doing once, carefully, because every useful question routes through it.
The first questions I asked it
This is the part that made the whole build feel real, so here are the actual first questions I ran, described by what they do rather than as copy-paste.
Who touched IAM last week. Filter the events down to the identity service as the source, keep only the mutating calls, the creates and attaches and puts and deletes, over a seven-day partition. This is the "did anyone change permissions" question, and it is the one you most want a fast answer to. It came back in seconds and read almost nothing, because the partition kept it to a week.
Every access to the secret. Filter on the secret-retrieval event for that one secret's identifier, across a wide date range. This is the question the console literally cannot answer past ninety days. Here it is a single query. If a key ever leaks, "what did it read, and when did the reads start" is the difference between a scoped cleanup and a full credential rotation across everything.
What did this role do. Filter on the assumed-role identity in that nested block, and you get the full activity of one identity over a window. This is how you scope an investigation once you suspect a particular credential, instead of assuming the worst about all of them.
Root usage and logins without a second factor. Any use of the account root, or any console sign-in that did not carry multi-factor, is a small set of events that should be close to empty. Being able to confirm it is empty is its own kind of quiet reassurance.
The honesty about this layer
I said this was the greenfield part and I want to hold to that. Almost nobody builds a queryable audit layer before an incident, and the reason is completely rational in the short term. It ships no feature. It has no user. On the day you finish it, the only thing you can do with it is answer questions you do not yet have. Every incentive points at doing literally anything else.
The trap is that its value is entirely a function of how much history it has already accumulated when you finally need it. Build it today and in six months it can answer six months of questions. Build it during the incident and it can answer nothing about the period you actually care about, because that period is the past, and the past is only in the bucket if the trail was already running and only queryable if you had already done this.
The cost of doing it now is genuinely small. The query engine charges per data scanned, and with tight partitions a typical investigative query scans a sliver and costs a rounding error. Lifecycle the oldest log objects down to colder, cheaper storage and the standing cost of keeping the full history is close to nothing. The tamper-evident digests from the trail mean the answers hold up if anyone ever questions them. It is, dollar for dollar, some of the cheapest security work in the whole series.
Where this leaves the account
The account can now be asked questions about its own past, precisely, over its full history, with answers you can trust. That is the entire point of the two-part build, and it closes the one real gap this infrastructure had.
What it does not do is tell me the moment something goes wrong. Querying is something I do when I already suspect a problem. The next part is the other direction: knowing when something breaks without having to go look, and the guardrails that stop the mistake before the query is even needed.
Previous: Before the First Incident Asks, Write Down Who Did What Next: Know When It Breaks, and Guardrails That Survive a Mistake
The same decision, the other cloud: Making the Record Queryable on GCP