Turning the audit trail on, which was the previous part, gives you a satisfying feeling and a false one. The events are being recorded. The problem is where, and for how long, and whether you can ever get a straight answer out of them.
By default the trail lands split, the same way its two categories split. The always-on Admin Activity goes to the _Required bucket, immutable and kept for 400 days with nothing to configure. The Data Access logging I deliberately switched on goes to _Default, retained for 30 days and then gone. You can scroll either in the Logs Explorer. What you cannot do is the thing you will actually need on a bad day, which is ask a precise question across a meaningful window and get a table back. "Who read the secret last quarter" is a 90-day question against a 30-day store, and even "who changed an IAM policy last year" runs into the 400-day edge of the durable half, both on a tool built for tailing, not for analysis. Older than that, and the answer is simply gone.
This is the genuine day-0 gap, and I want to be honest that it is the part almost every team skips. Turning logging on feels like finishing. It is not. A record you cannot query is a receipt stuffed in a drawer: technically you kept it, but you will never read it, so it may as well not exist. This part closes the gap on the real infra, and I will show the design rather than a click-path.
What "queryable and durable" actually requires
Two properties are missing and they are separate.
Durable means the record outlives those defaults, the 30 days for Data Access and even the 400 for Admin Activity, by enough that a question you think of in April can still be answered about the year before. Compliance windows are measured in months and years, not weeks. An incident discovered late, and they are usually discovered late, needs a trail that reaches back past the discovery.
Queryable means you can express a real question, filter by principal, by method, by resource, over a range, join and count, and get a result set, not a scroll. The default store technically supports filtering, but not at the shape or scale of a real investigation, and not once the window has expired.
GCP gives you two honest ways to get both, and the choice between them is the actual design decision.
The first is a log bucket with Log Analytics enabled. You create a dedicated log bucket, set its retention to something long, and route audit logs into it with a sink. Enabling Log Analytics on that bucket makes it queryable with SQL through a linked dataset, with no data leaving Cloud Logging and no second copy to secure. Retention on the bucket is the durability knob.
The second is a sink to BigQuery. A log sink with a BigQuery dataset as its destination streams matching entries into date-partitioned tables you query as normal BigQuery. You get the full weight of BigQuery, and you also get a second data store to lock down, and ingestion and storage costs that scale with volume.
For a single-project setup at my scale, the log bucket with Log Analytics is the leaner choice: one store, one thing to secure, retention as a dial, SQL when you need it. I will describe the design in those terms, and the BigQuery route is the same shape with a heavier, more powerful destination, which is why the AWS twin, routing to object storage plus a query layer, reads so similarly.
The sink is the load-bearing part
The mechanism that moves the record from the ephemeral default store to the durable, queryable one is a log sink, google_logging_project_sink in Terraform. A sink is three things: an inclusion filter that decides which log entries it carries, a destination, and an identity it writes as.
The filter is where the security thinking lives, and it is the mirror image of the scoping I did when turning the trail on. I do not route everything. I route the audit logs specifically, the cloudaudit.googleapis.com log streams, Admin Activity and the Data Access categories I enabled, because those are the who-did-what record. Application logs, request logs, the coordinator's telemetry line, all of that is operational noise for this purpose and belongs in its own lifecycle. A sink that grabs everything is the same mistake as an audit config set to allServices: it buries the signal you built this for under volume you did not.
A subtlety worth stating: a sink routes matching logs to the new destination, but by default those same logs also keep flowing to _Default. That is fine. The durable store is additive. What I do not do is get clever and exclude audit logs from _Default to save a little storage, because the default store is the fast path for tailing and the durable store is the slow path for asking, and they serve different moments. Pay the small double-storage cost and keep both.
The sink writes as its own service account, which GCP creates for it, and that account needs exactly one grant on the destination: permission to write. Least privilege reaches even here. The sink can write the record and do nothing else, which matters, because the sink's identity is, by construction, a principal that touches the audit store, and a principal that can rewrite the audit store is a principal that can erase the trail.
The first questions
A record is only worth building if it answers questions you will really ask. Here are the first three I built this to answer, and they are not hypothetical, they are the questions the quiet-room incident always asks.
Who touched access controls last week. IAM policy changes surface as SetIamPolicy calls in Admin Activity. The query filters audit entries to that method name over a seven-day range and projects the principal, the timestamp, the resource, and the before-and-after in the request. On a calm week this returns a short, boring list, my own applies, and that is the point: you learn the shape of normal so that abnormal is visible. The first time this list has a row you did not expect, you will be very glad the window is longer than 30 days.
Every read of the one secret. Because I enabled DATA_READ on Secret Manager on day 0, every AccessSecretVersion on the coordinator's Turnstile key is now in the durable store. The query filters to that method and that secret and returns who read it and when. In steady state this is one service account, the coordinator's, on a predictable rhythm. Any other principal reading that secret, a human, a different service, is now a visible, queryable, retained event instead of an invisible one. This is the single query that most justifies the whole exercise.
Who has been looking, not just touching. The ADMIN_READ logging on IAM means reconnaissance, someone enumerating roles and policies, is now answerable too. It is a quieter signal and a genuinely useful one, because looking usually precedes touching.
The honest cost, and what is still missing
The durability is not free, and I would rather name the number-shaped truth than pretend. Data Access logs are the volume driver, which is exactly why they are off by default, and routing them into a retained store means paying to keep that volume for the retention you chose. The discipline from part five, scope the audit config narrowly, is what keeps this bill sane: because I log secret reads and sensitive-bucket reads rather than all reads, the durable store holds a stream measured in the sensible range, not the runaway one. Scoping at the source is what makes retention affordable at the destination. The two decisions are one decision.
What this does not yet do: nothing watches the trail. I can now ask "who read the secret," but nothing tells me when the answer changes. Wiring an alert onto an audit event, fire when a principal that is not the coordinator reads the Turnstile key, is the bridge from a queryable record to an active one, and it belongs with the alerting and guardrails part, which is next. A record you have to remember to query is better than no record, and worse than a record that taps you on the shoulder.
Previous: the record of who did what. Next: know when it breaks, and guardrails that survive a mistake.
The same decision on the other cloud: making the AWS trail queryable.