Last part ended with a coordinator running on Cloud Run and a WebSocket URL. This part it gets clients. I opened the page on my laptop, then on a second machine, and watched two browsers feed one number. activeNodes: 2, one global dart count, one pi estimate ticking upward. That moment is the whole premise of the project made visible: open a page, join a computation.
Then I did the thing everyone eventually does to your beautiful distributed system. I closed a tab in the middle of it.
What actually travels over the wire
The reason that tab-close is survivable comes down to what a work unit is. When a node connects it sends hello with its core count, gets back a welcome with a node id and a heartbeat cadence, and then receives assignments:
{ "type": "assign", "unit": { "jobId": "j1", "unitId": "u201", "seed": "9007199254741021", "count": 1000000 } }
That unit is self-contained. Part 2's seeded kernel means (seed, count) fully determines the answer: same seed, same count, bit-identical {inside, total} on any honest machine. No session state, no partial progress, no affinity to the node that got it first. A unit does not care who computes it or when, and that one property is where every bit of fault tolerance in this part comes from. If a node disappears, the recovery action is not "replay its journal" or "restore its checkpoint". It is: hand the same unit to someone else.
Keeping cores fed
Assignment is a pull-style pump. Each node gets a pipeline of in-flight units sized to its hardware, two per reported core with a hard cap of 64, so a 16-core machine holds 32 units at once. The pipeline exists so cores never idle waiting for a network round trip; a worker finishes a unit, the result goes up, the next unit is already sitting there.
The important side effect: a busy node always has units in flight. Which means closing its tab always strands work. There is no polite moment to kill a node, and that is exactly what I wanted to test.
Closing the tab
The coordinator tracks every unresolved unit in a map, and for each one, which nodes are actively computing it. When a node's socket closes, one method runs: nodeGone. It deletes the node, walks its in-flight set, and reopens every slot. Here is the shape of the log when I killed the 16-core node mid-run (log text lightly reformatted to fit):
12:41:03.118 INFO node joined nodeId=n-4c9a cores=16
12:41:03.140 INFO assign u201 -> n-4c9a (inFlight 1/32)
...31 more assigns...
12:41:37.402 INFO node gone, in-flight slots reopened
nodeId=n-4c9a reason=socket-closed requeued=32
12:41:37.404 INFO assign u201 -> n-b31e
12:41:37.404 INFO assign u207 -> n-b31e
12:41:37.405 INFO assign u214 -> n-0ff2
...
Thirty-two orphaned units, reopened and reassigned to the survivors in single-digit milliseconds. The /statz endpoint shows the same story as counters: unitsReassigned jumps by 32, and a companion gauge, reassignedPending, tracks reopened units that have not yet been completed by someone else. Watching reassignedPending drain back to zero is watching the grid heal in real time. If it ever sticks above zero, a unit got lost, and that is the bug the counter exists to catch.
One detail I like: a reopened slot is not fed into some special recovery queue with its own scheduler. It goes into the same need-queue the normal assignment loop drains, and that loop fills needy units before minting new ones. Heal first, grow second. There is no recovery subsystem to get wrong because the recovery path is the normal path. That is what I mean by fault tolerance by construction: I did not add healing to the grid, the unit design made healing fall out of ordinary assignment.
The death nobody announces
Sockets do not always close. A laptop lid snaps shut, a phone browser gets backgrounded and throttled, a machine drops off wifi. The TCP connection can sit there looking alive for minutes while the node behind it computes nothing. So closure events alone are not enough; you need a liveness protocol.
Every node heartbeats on the cadence the coordinator handed it in welcome, 5 seconds by default, and any message counts as proof of life. A sweep runs every second and declares any node silent for more than 2.5 heartbeat intervals (12.5 seconds) dead, then routes it through the exact same nodeGone path, reason=heartbeat-timeout instead of reason=socket-closed. The sweep also force-closes the zombie's socket, so a node that was merely frozen (a long GC pause, a throttled tab) reconnects with a fresh hello instead of heartbeating into a void that no longer knows its name.
The same sweep catches a third failure mode: the node that is alive and heartbeating but sitting on an assignment for more than 30 seconds. Stuck slot, same treatment, reopened for someone else. Three different failure signals, socket close, heartbeat silence, stuck unit, all funnel into one reopenSlot call. One code path to test instead of three.
And it is genuinely tested, both ways. The integration harness spins up the coordinator plus four headless Chromium nodes, waits for 50 million darts, then runs two separate kills. First it closes one node's browser context outright, the socket-close path. Then it freezes another node's main thread with a busy-wait for 8 seconds, so heartbeats stop while the socket stays open, the timeout path. Both times it asserts the same things: unitsReassigned grew, reassignedPending drained to zero, the pi estimate is still within tolerance, and the aggregation invariant globalTotal == unitsCompleted * unitDarts still holds exactly. No lost units, no double counting, no corruption. The frozen node even thaws, reconnects, and rejoins under a fresh node id, which is the difference between a grid that tolerates churn and one that merely survives it.
One design choice worth naming: when a node dies honestly, its already-submitted results stay. Death is not distrust. A recorded result from a machine that later vanished is still a valid result. The coordinator only throws work away when it has a positive reason to doubt it, and what "reason to doubt" means is the next part's entire subject.
Why browsers force this discipline
If your compute nodes are servers in a rack, you can get lazy about churn, treat node death as an incident. Browser nodes do not permit that. People close tabs, phones sleep, laptops roam between networks. Churn is not the failure case here, it is the steady state, and that turned out to be a gift: the grid had to be correct under constant node loss from day one, so there is no fair-weather mode to fall out of. A node leaving is not an error anywhere in the codebase. It is just slots reopening.
What this machinery cannot do is tell an honest dead node from a live lying one. Every result so far is trusted the moment it arrives, and that assumption does not survive contact with the open internet. Catching a node that returns plausible but wrong answers requires different machinery entirely, and that is where the seeded determinism from part 2 finally pays off.
Previous: where does a WebSocket coordinator live?. Next: trusting math from strangers.