Download Game! Currently 71 players and visitors. Last logged in:DurtleLmtWandererRanma

Blitzer's Blog >> 71912

Back to blogs index
Posted: 31 Aug 2026 15:39 [ permalink ]
Analyzing your architecture for an MVP prototype with a tight timeline
(today/tomorrow), here are my strategic recommendations:

## **Critical Decisions for MVP**

### 1. **Payload Binary Format  JSON (UTF-8)**
**Choice:** JSON strings stored directly in the append-only log

**Why:**
- **Speed of development**: You can prototype in hours, not days
- **Debugging visibility**: You can inspect raw SAB data with TextDecoder
- **Flexibility**: Easy to change structure without recompiling
- **Zero-copy serialization**: You can still achieve zero-copy reads by just
returning the string pointer
- **Sufficient for MVP**: Your 100k ops/sec target is achievable with JSON +
lock-free design

**Implementation:**
```javascript
/ Store as UTF-8 bytes directly in SAB
const encoder = new TextEncoder();
const payload = encoder.encode(JSON.stringify(obj));
/ Write to append-only log
```

### 2. **UID Format  64-bit Integer**
**Choice:** Monotonically incrementing 64-bit unsigned integer

**Why:**
- **Simplicity**: No generation overhead, easy to debug
- **Atomic operations**: Atomics.compareExchange works natively with 64-bit
- **Hash table efficiency**: Perfect fit for 8-byte slots (32-bit hash +
32-bit offset)
- **Compaction friendly**: Easy to track highest UID during compaction
- **Debugging**: Human-readable (`uid_42` vs `uuid-550e8400-e29b-41d4-a716-4466
55440000`)

**Implementation:**
```javascript
/ Last UID stored in SAB at offset 0
const lastUid = Atomics.add(sab, 0, 1) + 1;
```

### 3. **OpCodes  Minimal Set**
**Choice:** Implement only 4 core operations for MVP

```javascript
/ OpCodes (1 byte)
const OP_INSERT = 0x01;      // { uid, namespace, data }
const OP_UPDATE = 0x02;      // { uid, namespace, partial }  
const OP_DELETE = 0x03;      // { uid, namespace } (tombstone)
const OP_GET = 0x04;         // { uid, namespace } (response via OUT)

/ Payload structure:
/ [uid: u64][namespace_len: u8][namespace: bytes][data_len: u32][data: bytes]
```

### 4. **Compaction  Double Buffer Simple**
**Choice:** Simple A/B buffer switching with immediate failover

**Why:**
- **Lock-free guarantee**: Writer never blocks
- **Simple state machine**: Two buffers, active_flag, compacting_flag
- **Predictable memory**: Fixed sizes for MVP

**State Machine:**
```
STATE_ACTIVE_A (writing to A, B free)
STATE_COMPACTING_A (writing to B, compacting A)
STATE_ACTIVE_B (writing to B, A free/cleared)
```

### 5. **Target Environment  Node.js**
**Choice:** Node.js with worker_threads

**Why:**
- **Fastest prototyping**: Full debugging, console.log, inspect
- **SharedArrayBuffer support**: Full implementation
- **Worker_threads**: Perfect for your SPSC ring buffer architecture
- **File system access**: Easy swap provider implementation
- **Transition path**: Easy to later port to mquickjs or browser

## **MVP Implementation Roadmap (24 hours)**

### **Phase 1: Core Database (4 hours)**

### **Phase 2: Ring Buffer Bus (3 hours)**


### **Phase 3: Worker System (4 hours)**


### **Phase 4: Test & Benchmark (3 hours)**

## **Key Constraints for MVP**

1. **Payload Size**: Keep < 64KB to avoid fragmentation
2. **Max Objects**: 1 million (hash table size)
3. **Max Log Size**: 64MB (compaction triggers at 90%)
4. **Max Workers**: 8 (bus memory allocation)

## **Immediate Actions**

1. **Clone the repo structure** with these core files:
   - `src/core.js` - Database core
   - `src/bus.js` - Ring buffer implementation
   - `src/worker.js` - Worker template
   - `src/benchmark.js` - Performance tests

2. **Start with JSON payloads** - you can optimize to binary later

3. **Implement just INSERT, GET, DELETE** - skip UPDATE (just DELETE+INSERT
for MVP)

4. **Use Node.js 18+** - latest SAB and atomics support

5. **Focus on getting the lock-free ring buffer right** - this is your
performance bottleneck

## **Why This Works Today**

- **Proven patterns**: Your architecture mirrors Disruptor pattern (LMAX) 
- **Known technology**: All Node.js features are stable
- **Debug friendly**: JSON + console.log means you can see everything
- **Performance**: SPSC ring buffers in SAB can do millions of ops/sec

## **Deferred Decisions (Post-MVP)**

1. **Binary format**: Once JSON works, profile and optimize
2. **Compaction**: Implement after basic CRUD works
3. **Swap/Offloading**: Add when memory becomes an issue
4. **mquickjs port**: Only if you need embedded/edge deployment

**Start coding now** - the architecture is solid, and the design choices I've
suggested will get you to a working prototype today. The lock-free core is the
hardest part, but with JSON and simple 64-bit UIDs, you'll have it running in
hours, not days.