To compile and execute custom functions, the virtual machine must evolve from
a single loop over a flat array into an engine that manages a Call Stack.
When the VM encounters a function, it doesn't execute it immediately. Instead,
the compiler generates the function's body as an independent chunk of bytecode
(a "Code Object"). The VM then uses Call Framesstructures that track the
Instruction Pointer (IP), local variables, and the bytecode chunk for the
current execution context.
Here is how to implement this cleanly in a stack machine architecture.
1. Expanding the Instruction Set
We need four new opcodes to handle the lifecycle of a function: creating it,
storing it in scope, invoking it, and returning control to the caller.
const Opcodes = Object.assign(Opcodes || {}, {
  MAKE_FUNC: 'OP_MAKE_FUNC', // Pushes a callable Code Object to the stack
  STORE:     'OP_STORE',     // Pops a value and saves it to a variable name
  CALL:      'OP_CALL',      // Pops a function and its arguments, pushes a
Call Frame
  RET:       'OP_RET'        // Pops the Call Frame, leaving the return value
on the data stack
});

2. Adapting the Compiler
When the compiler hits a def block, it spins up a fresh array to compile the
function's body independently. It then emits a MAKE_FUNC instruction
containing that new bytecode array, followed by a STORE instruction to bind it
to the function's name.
function compile(ast, bytecode = []) {
  if (!Array.isArray(ast)) { /* ... primitive / load logic as before ... */
return bytecode; }
  const [op, ...args] = ast;

  // Compile a Function Definition
  // AST: ["def", "square", ["x"], ["*", "x", "x"]]
  if (op === 'def') {
    const [name, params, body] = args;
    
    // Compile the body into an isolated chunk of bytecode
    const funcBytecode = compile(body, []);
    
    // Inject an implicit return at the end of the function body
    funcBytecode.push({ op: Opcodes.RET });
    
    // Emit instruction to instantiate the function at runtime
    bytecode.push({ 
      op: Opcodes.MAKE_FUNC, 
      val: { params, code: funcBytecode } 
    });
    
    // Emit instruction to save the function to the environment
    bytecode.push({ op: Opcodes.STORE, val: name });
    return bytecode;
  }

  // Compile standard operations and Function Calls
  // Arguments are evaluated and pushed to the stack first
  args.forEach(arg => compile(arg, bytecode));

  if (['+', '*', '<'].includes(op)) {
    // Built-in ALU operations
    bytecode.push({ op: `OP_${op === '+' ? 'ADD' : op === '*' ? 'MUL' : 'LT'}`
});
  } else {
    // Custom function call: ["square", 5]
    // 1. Load the function object onto the stack
    bytecode.push({ op: Opcodes.LOAD, val: op });
    // 2. Emit the CALL instruction, telling the VM how many arguments to pop
    bytecode.push({ op: Opcodes.CALL, argCount: args.length });
  }

  return bytecode;
}