Download Game! Currently 71 players. Last logged in:BullwinBiubiuFloppyZidane

Blitzer's Blog >> 73073

Back to blogs index
Posted: 18 Oct 2026 10:16 [ permalink ]
You just hit a major milestone. By testing this on two completely different
machines, you organically discovered the two most notorious problems in
decentralized computing: Clock Drift and Split-Brain!

Here is exactly what your logs are showing.

You aren't seeing the tiny Delta Proxy [MESH SYNC] updates. Instead, you are
seeing Thump... print on both machines simultaneously. That means both
machines are executing the heartbeat loop independently. They are fighting
over the monster!

Why did this happen?

Clock Drift: Machine 2's system clock is likely a few seconds ahead of Machine
1. When Machine 2 read Machine 1's timestamp in the VFS, it subtracted it from
its own Date.now(). Because of the clock mismatch, the result was immediately
greater than 4000ms. Machine 2 thought Machine 1 was dead and hijacked the
lock!

Split-Brain: Even though Machine 2 claimed the lock, we never told Machine 1
how to yield. Machine 1's local setInterval just kept running, blithely
overwriting Machine 2's lock. Both engines think they are the authoritative
master.

The Fix: The Yield & Relative Time

We can fix this instantly by making two changes to our watchdog:

The Yield: If an engine notices someone else's ID on the lock file, it must
gracefully call clearInterval and step down.

Relative Time: Instead of comparing timestamps between two different
computers, an engine should simply track how many seconds have passed locally
since the file was last modified.

Open harness.js, scroll down to the DistributedHeartbeat class, and replace
only the _watchdogTick method with this battle-hardened version:

_watchdogTick() { const hbDir = 'mesh/proc/heartbeats/'; Object.keys(this.vfs).
forEach(path => { if (!path.startsWith(hbDir)) return; try { const data =
JSON.parse(this.vfs[path]); const objectId = '/' + path.replace(hbDir,
'').replace('.json', ''); // 1. SPLIT-BRAIN RESOLUTION (The Yield) if
(data.owner_engine !== this.engineId) { // If we were running it, but someone
else claimed the lock, step down! if (this.activeBeats.has(objectId)) {
console.log(`[HA-Cluster] Lost lock for ${objectId} to ${data.owner_engine}.
Yielding!`); clearInterval(this.activeBeats.get(objectId)); this.activeBeats.de
lete(objectId); } // 2. CLOCK DRIFT FIX (Relative Time Tracking)
this.lastObserved = this.lastObserved || {}; this.lastObservedLocalTime =
this.lastObservedLocalTime || {}; const lastKnownTick = this.lastObserved[objec
tId] || 0; if (data.last_tick !== lastKnownTick) { // The owner updated the
file! Record our LOCAL time. this.lastObserved[objectId] = data.last_tick;
this.lastObservedLocalTime[objectId] = Date.now(); return; // They are alive,
nothing more to do. } // The file hasn't changed. How long has it been
according to OUR clock? const localTimeSinceUpdate = Date.now() -
this.lastObservedLocalTime[objectId]; const isDead = localTimeSinceUpdate >
(data.interval_ms * 2); if (isDead) { console.log(`[HA-Cluster] Node
${data.owner_engine} flatlined. Taking over ${objectId}!`); this._hydrateAndTak
eover(objectId, data.interval_ms); } } } catch (err) {} }); } 

The Real Delta Test

Run make web and refresh both machines.

Type monster on Machine 1.

This time, because the clocks are ignored, Machine 1 will maintain the lock
perfectly. Machine 2 will sit quietly in passive observation mode, and you
will not see Thump... print on Machine 2.

Instead, look closely at Machine 2's output. Does the Delta Proxy successfully
broadcast the hp = 99, hp = 98 state changes to Machine 2's terminal via the
[MESH SYNC] logs?