The series was supposed to be done. Part 8 shipped the grid to pi.inoltro.tech, showed the production numbers, and closed on a thesis I still believed: serverless-first is the right default, and the one component that broke the rule, a stateful WebSocket coordinator pinned to a single Cloud Run instance, broke it for a reason I could say in a sentence.
Two days later that instance wedged in production. This part exists because the series runs on exactly one rule, honesty over polish, and a build log that ends on a deploy which then falls over owes you the next chapter.
The symptom
The coordinator started refusing connections under sustained real load. Not a crash, not an out-of-memory kill, nothing in the logs that looked like my code failing. Requests would queue for about ten seconds and then come back 429, "Rate exceeded," while the container underneath sat there apparently fine. New browser tabs trying to join the grid hung on the WebSocket handshake and gave up. The site served its pages perfectly and then quietly declined to grow the grid.
My first assumption was the obvious one: a leak, a runaway loop, or some slow accumulation in the in-memory maps that part 4 was so proud of keeping small. So I instrumented it.
It wasn't the code
I added a line that logged the real health of the process every ten seconds: event-loop lag, resident memory, heap, open connections, active nodes. Then I watched it under the exact load that triggered the wedge.
The numbers were boring, which was the point. Event-loop lag stayed in the single digits to low tens of milliseconds. Resident memory was flat. Open connections matched active nodes exactly, so nothing was leaking sockets. The container was not struggling. It had headroom to spare at the moment Cloud Run started returning 429s on its behalf.
That reframed the whole problem. The thing refusing requests was not my process. It was the platform in front of it.
The diagnosis
Here is what I missed in part 4. I chose min = max = 1 to prevent split brain, and it does. But a single Cloud Run instance is still a Cloud Run instance, and Cloud Run's model is built around short requests it can admit, count against a concurrency limit, and complete. A WebSocket that stays open for the whole life of a browser tab is the opposite of that. Every connected node holds one long-lived request slot for minutes or hours. As real nodes accumulated, the platform's request accounting filled up and its admission control began shedding new connections, even though the process behind it was idle enough to take hundreds more.
Part 4 caught the split-brain failure mode and guarded against it. It did not catch this one: that a single pinned Cloud Run instance and a pile of long-lived WebSockets are a bad fit at the platform layer, independent of anything my code does. A single always-on stateful process holding many sockets is not a serverless workload wearing a constraint. It is a virtual machine.
The fix
So the coordinator moved to a plain VM. A GCE e2-micro running Container-Optimized OS: the same container image, the same single-process in-memory design, behind the same load balancer, the same Cloud Armor policy, the same Turnstile gate. Only the substrate changed.
The single-instance rule got safer in the move, not weaker. On Cloud Run, max = 1 was a setting I trusted the platform to honor. On the VM it is structural: the coordinator is an unmanaged instance group of size one, with no autoscaler and no managed-instance-group controller that could ever start a second. There is nothing to nudge. Two instances is not a state I can reach by accident.
It is also cheaper. An always-on Cloud Run instance with CPU always allocated ran about forty dollars a month. The e2-micro is around six. I had been paying a premium for a serverless platform to behave like an always-on VM, badly.
Then I load-tested it to seven concurrent browser nodes, more than the load that wedged Cloud Run, and it held all of them: memory flat, no queue, and not a single 429.
The part that bit back
The migration was not clean, and the way it failed is worth keeping.
After the cutover, the site served every page with a 200, health checks were green, and the grid registered zero nodes. Every browser that tried to join hung for five seconds and was refused. The pages were alive and the compute was dead.
The cause was a networking decision I had made for good reasons. The VM has no public IP, and it reaches Google's own APIs over Private Google Access, so it never needs a route to the general internet. Except the coordinator verifies each node's Cloudflare Turnstile token by calling Cloudflare's servers, and Cloudflare is not a Google API. Private Google Access does not route it. So every token verification ran to its five-second timeout, failed closed exactly as designed, and refused the node. The one outbound call the coordinator makes was the one call the network could not deliver.
The fix was a Cloud NAT gateway, giving the VM a single outbound path while keeping it with no inbound public IP. A bogus token that used to hang for 5.2 seconds now gets rejected in half a second, and real tabs register. It is a small thing, but it is the honest texture of infrastructure: you close the network down for security, and you find out later which door you needed open.
Does the thesis survive?
Mostly, and it comes out sharper.
Part 4 said that when a component is genuinely stateful, small, and singular, one deliberate server beats a distributed workaround. That is still true. What I got wrong was calling a pinned Cloud Run instance "one deliberate server." It is a serverless runtime asked to hold still, and under long-lived connections at volume the runtime's own behavior leaks through. The actually-deliberate server, for this shape, is a VM. The judgment held; the implementation of it moved down a layer.
And the larger default survives intact, because this is exactly how it is supposed to fail. Serverless-first says follow the default until you can articulate precisely why you shouldn't. In part 4 I articulated the split-brain reason and stopped one step short. Production supplied the rest of the sentence. That is not the thesis breaking. That is the thesis working on a longer timescale than a single deploy.
I also did the thing I should have done before the wedge, not after: put an uptime check and an alert on the coordinator, so the next time it goes quiet I hear about it from a page, not from a screenshot of an empty grid.
The series still ends where part 8 left it: not a business, but a real capability, honestly measured. This is the footnote the deploy earned by falling over. The default was right and the pinned instance was right. The platform under it was one layer too clever for a socket that never hangs up.
Previously: Shipping it, and the honest scorecard.