==[ VON ]==
This is a fascinating project that sits at the intersection of accessibility
engineering, telemetry systems, and LLM-driven automation. The core
ideatranslating a live UI into both a deterministic AST and a natural language
narrativeis essentially building a "semantic bridge" between the visual world
and the reasoning core of an AI agent.

Below Ill expand on each of your three key concepts, offering design
considerations, potential pitfalls, and concrete implementation ideas.

---

## 1. The Structural Snapshot (State as AST)

**Goal:** Produce a minimal, semantic JSON tree that captures *what matters*
about the interface, stripping away all visual/stylistic noise.

### Brainstorming Points

- **What belongs in the AST?**  
  The AST should answer: *What can the agent interact with, and what is the
current meaningful state?*  
  Consider including:
  - **Role & type:** button, link, input, heading, list, etc. (based on ARIA
roles or HTML semantics)
  - **Label / accessible name:** derived from text content, `aria-label`,
`aria-labelledby`, `alt`, etc.
  - **State:** enabled/disabled, checked/unchecked, expanded/collapsed,
focused, selected, visibility.
  - **Value/Content:** for inputs (`value`), for text nodes (the text itself),
for lists (item count).
  - **Relationships:** parent-child hierarchy, but also meaningful groupings
(e.g., a form, a navigation bar, a modal).
  - **Unique identifiers:** stable IDs if available (`id` attribute), or
generated selectors that are resilient to DOM changes.
  - **Interactive capabilities:** what actions can be performed? (click, type,
submit, etc.)

- **How to strip CSS/styling noise:**  
  Simply ignore all CSS properties (display, color, margins, etc.) *except*
those that affect interaction semantics:
  - `visibility: hidden` or `display: none`  mark as `hidden`/`not in
accessibility tree`.
  - `pointer-events: none`  mark as non-interactive even if it looks like a
button.
  - `opacity: 0` or off-screen positioning  possibly mark as visually hidden
but still in DOM? Consider using the browsers accessibility tree as a source
of truth.

- **Leverage the Accessibility Tree:**  
  Browsers already compute an accessibility tree (used by screen readers). You
can access it via:
  - **CDP (Chrome DevTools Protocol):** `Accessibility.getFullAXTree` gives a
rich, semantic tree.
  - **Web APIs:** `getComputedAccessibleNode()` (not widely supported) or
build your own using ARIA and DOM traversal.  
  Using the built-in accessibility tree ensures you inherit all the platforms
heuristics for naming, roles, and states.

- **Example enriched AST node:**  
  ```json
  {
    "id": "patchBtn",
    "type": "interactive",
    "role": "button",
    "label": "Apply Patch & Resume",
    "state": {
      "enabled": true,
      "focused": false,
      "pressed": false
    },
    "actions": ["click"],
    "children": []
  }
  ```

- **Handling dynamic content:**  
  The snapshot is just a single frame. But you might want to include a
timestamp and perhaps a root node representing the top-level viewport or
document.