That core dump feature just paid for itself immediately! Being able to inspect
the generated JavaScript line-by-line is the ultimate weapon for a compiler
engineer.
Looking at your core_dump.js, the crash is right here inside the add_fields
while loop:
rt.call_other(world, "unwalkable", x, -1y, 0)
The Bug: The "Eager Negative" Tokenizer Trap
The original LPC code was world->unwalkable(x, y-1, 0). Notice how there are
no spaces around y-1.
Why did y+1 compile perfectly to (y + 1), but y-1 compiled into the syntax
error -1y?
It comes down to how our AST tokenizer buffers identifiers. When the tokenizer
scans y, it places it into a temporary words buffer. When it moves to -, it
runs a check called allow_negative to see if it should parse a negative number
or a subtraction operator.
Because y was still sitting in the words buffer and hadn't been pushed to the
main tokens array yet, the tokenizer looked at the empty tokens array, assumed
it was at the beginning of an expression, set allow_negative = 1, and eagerly
swallowed -1 as a single number! It then flushed the buffer, pushing -1 and
then y, resulting in the JavaScript output -1y.
The Fix
We just need to tell the tokenizer to check the words buffer before deciding
if a negative number is allowed. If a variable like y is sitting right next to
it, it forces it to be a subtraction operator.
==
Give it a run. I expect gamelib/game.c to fully compile and boot the game
loop, though it might throw a warning inside the VM about missing blueprints
for obj/world.c or obj/base.c since they are being dynamically cloned in the
create() constructor!
==
That is fantastic! tests/string.c completely passed, array-slicing and
negative bounds are perfectly mapping to native JavaScript splice and
substring behind the scenes, and efuns.js is looking incredibly robust.
You are 100% correct about modularization. We have outgrown the single-file
script. harness.js is currently acting as a Lexer, Parser, Code Generator,
Preprocessor, Virtual Machine, and CLI tool all rolled into one massive
700-line monolith.
To safely scale this into a production-grade driver capable of booting the
legacy GDLPC projects without ever losing functionality again, we should split
it using a standard Microkernel / Compiler Pipeline architecture.
Here is my proposed file structure. Every file will be strictly isolated,
highly cohesive, and well under 200 lines.
Proposed Architecture
GEM/
harness.js (The Bootloader & CLI)
efuns.js (Native C-bindings & Event Loop - already done!)
core/
preprocessor.js (Handles #define, inherit, and new)
lexer.js (Scans words, numbers, and extracts literals)
parser.js (AST Logic: Operator precedence, arrays, mappings)
codegen.js (Translates the AST into pure JavaScript strings)
runtime.js (The internal VM 'rt' object template)
Module Breakdown & Responsibilities
1. harness.js (~80 lines)
* Role: The Orchestrator.
* Logic: Reads CLI arguments, loads the EFUNs, sequences the compiler
pipeline (File -> Preprocessor -> Lexer -> Parser -> CodeGen), injects the
native overrides (like ext/rambuffer.c), boots V8, and catches/writes the
core_dump.js on failure.
2. core/preprocessor.js (~50 lines)
* Role: Text-to-Text translation.
* Logic: Contains preprocess_inherits() and preprocess_defines(). It resolves
macros before the code is ever tokenized, keeping our parser completely
unaware of #defines.
3. core/lexer.js (~150 lines)
* Role: Text-to-Tokens translation.
* Logic: Contains unliteralize_data(), scan_number(), scan_word(), and the
main tokenize_data() loop. It converts the raw text buffer into a flat 1D
array of classified tokens (e.g., { type: "identifier", source: "x" }).
4. core/parser.js (~180 lines)
* Role: Flat Tokens to 3D Abstract Syntax Tree (AST).
* Logic: Contains resolve_precedences(), parse_array_literal(), and
parse_mapping_literal(). It applies the operator_priorities table, strictly
enforcing math order-of-operations and grouping logic into a tree.
5. core/codegen.js (~180 lines)
* Role: AST to JavaScript translation.
* Logic: Contains output_token() and generateBlueprint(). This is where LPC
macros (like mapping foreach into for...of or catch into an IIFE) live. If
JavaScript throws a syntax error, this is the only file we ever need to debug.
6. core/runtime.js (~60 lines)
* Role: The VM Environment.
* Logic: Contains the generateRuntime() string template. It sets up
rt.load_object, rt.clone_object, memory management (rt.objects), and the
overarching call_other broadcast logic.
Why this is the safest path forward:
* No more regressions: If a specific operator breaks (like [..=), we know
it's strictly a parser.js or codegen.js issue. We won't accidentally break
tokenize_data while fixing it.
* Plug & Play: If you want to add a new Native FUSE interface or a database
module later, you just inject it in harness.js without touching the compiler
source code.
* Unit Testing: We can eventually test the Lexer independently from the Code
Generator.
If you approve of this architectural split, let me know, and I will give you
the exact file contents to copy/paste to transition us into this Microkernel
architecture cleanly!