← back to the garden

evergreen

The More You Log, the More You Leak

planted · April 26, 2026
#security-first-aws#security#aws#cloud#day-0#terraform

Two decisions in this part, and they pull in opposite directions, which is exactly why they belong together. The first is easy and almost nobody argues with it: encrypt everything at rest, by default, everywhere. The second is harder because it feels like giving something up: log less than you can, on purpose, because the most detailed logs are also where your secrets go to leak. Observability and confidentiality are in tension, and day 0 is when you decide where the line sits, before a verbose default has quietly been copying sensitive data into a log group for months.

Encryption at rest as a default, not a feature

There is a version of encryption that is a project: a ticket, a review, a rollout. There is another version that is just how buckets are made. The second one is the goal. On day 0, encryption at rest is not something you turn on for the sensitive data. It is the default state of all data, so you never have to correctly guess in advance which data will turn out to be sensitive.

In practice this is a few lines per bucket and then never thinking about it again:

resource "aws_s3_bucket_server_side_encryption_configuration" "uploads" {
  bucket = aws_s3_bucket.uploads.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

Every bucket gets this, the ones holding uploads, the one holding Terraform state, the one holding a static site. The state backend gets encrypt = true for the same reason. Uniform encryption removes a whole category of judgment call. You are never in the position of having decided a bucket was "not important enough to encrypt" and being wrong. The cost is essentially zero and the default is protection, so the default is on.

That is the whole of the first decision. It is short because it should be. Encryption at rest is settled; the interesting work is the second decision.

The tradeoff nobody frames on day 0

Here is the part that does not show up in most checklists. Logging is presented as an unqualified good. More logs, more visibility, more ability to debug and investigate. And logs are good, right up to the point where the thing you logged was a secret. The uncomfortable truth is that the highest-fidelity observability settings are precisely the ones that capture the data you are working hardest to protect: credentials in request headers, personal information in request bodies, the actual content users uploaded.

Verbose logging does not feel like a leak. It feels like diligence. But a log group that has been quietly recording an API key header on every request for six months is a secret store you did not know you created, with none of the protections you would put on a real one, and a retention you probably never set. The leak is slow, invisible, and entirely self-inflicted.

So the day-0 decision is not "how much can I log." It is "what must I never log," decided before the verbose default has been running long enough to matter.

Drawing the line in code, with the reason next to it

On the scanning API, the request path handles two things you absolutely do not want in a log: an API key, passed as a request header, and uploaded file content, passed as a request body. The gateway is perfectly capable of logging both. Turning on full request tracing would capture them. So the configuration deliberately does not, and it says why, right where the setting lives:

# Execution logging at ERROR level only. Deliberately NOT INFO and NOT
# data_trace: INFO/full-trace would log request headers (the X-API-Key) and
# request/response bodies (uploaded file content), both sensitive.
resource "aws_api_gateway_method_settings" "main" {
  settings {
    logging_level      = "ERROR"
    metrics_enabled    = true
    data_trace_enabled = false
  }
}

ERROR and not INFO. Data trace explicitly off. That comment is not decoration. It is the reasoning preserved at the point of decision, so that six months from now, someone debugging a tricky request cannot flip this to full tracing "just to see" without reading the sentence that says doing so writes API keys and file contents into a log. The comment is a tripwire for a future well-intentioned mistake. Half the value of the decision is that the next person meets the reasoning before they undo it.

The same care shapes what the access log records. Access logs are useful, and this one is on, but it logs a chosen set of fields rather than everything available:

format = jsonencode({
  requestId          = "$context.requestId"
  sourceIp           = "$context.identity.sourceIp"
  httpMethod         = "$context.httpMethod"
  path               = "$context.path"
  status             = "$context.status"
  userAgent          = "$context.identity.userAgent"
  requestTime        = "$context.requestTime"
  integrationLatency = "$context.integrationLatency"
})

Method, path, status, source address, timing. Enough to answer who called what, when, from where, and whether it worked. No headers. No bodies. The source address is here on purpose, because a network allowlist elsewhere keys on exactly that field, so this log is what lets me verify the allowlist is doing its job. That is the balance the principle asks for: log enough to operate and investigate, and stop precisely before the fields that carry secrets. Observability is not the enemy of confidentiality. Unbounded observability is.

Bound the logs you did not choose to create

There is a second, quieter leak, and it is about time rather than content. Several services create their own log groups on first use, and the default retention on an auto-created group is often "keep forever." Nobody decided that. It is just what happens when you did not decide anything. Forever-retention means sensitive-adjacent data, source addresses, request paths, error details, accumulates without bound, and the older it gets the less anyone remembers it is there.

The fix is to take ownership of those groups even though you did not create them, and put a cap on them:

import {
  to = aws_cloudwatch_log_group.api_lambda
  id = "/aws/lambda/${aws_lambda_function.api.function_name}"
}

resource "aws_cloudwatch_log_group" "api_lambda" {
  name              = "/aws/lambda/${aws_lambda_function.api.function_name}"
  retention_in_days = 30
}

That import block adopts a log group the platform made on its own and brings it under management for the sole purpose of setting a retention. Thirty days is enough to debug a recent problem and short enough that logs are not a growing liability. Doing this for the auto-created groups, the function logs and the gateway execution logs, means there is no corner of the account quietly hoarding request data with an indefinite lifespan. Retention is not just cost control. It is limiting how long a leak, if one ever slips into a log despite the care above, can sit there waiting to be found.

The posture, stated plainly

Encrypt everything, because guessing which data matters is a game you will eventually lose. Then log deliberately: capture enough to run and investigate the system, refuse the settings that would record credentials and payloads, write the reasoning next to the setting so nobody quietly reverses it, and cap retention even on the log groups you did not ask for. The two decisions look opposed, protect the data versus observe the system, and the resolution is to do both with intent rather than letting either default run unchecked. On day 0 that is a comment and a few config lines. Discovered later, it is a log group full of keys and an awkward conversation about how long they have been sitting there.


Previous: Deny by Default: Nothing Is Reachable Unless You Say So

Next in this series: The Record of Who Did What

The same decision on GCP: The More You Log, the More You Leak