Download Game! Currently 106 players and visitors. Last logged in:EronkBylethSabaothLmt

Blitzer's Blog >> 70978

Back to blogs index
Posted: 10 Jun 2026 10:25 [ permalink ]
Brilliant. Uvicorn is successfully running the `daemon.py` endpoints from the
isolated virtual environment. The Service Mesh router is active, and the
network can now query `127.0.0.1:8880` or `127.0.0.1:7711` natively.

Let's build the **Sweeper Agent**a dedicated AI worker that uses the newly
exposed X11 cursor capabilities to map out an unknown GUI window.

### The Strategy: "The Roomba Method"

Because our Hauki-GEM agents operate outside the DOM, they don't know where
buttons are. But they do know how to move the mouse (`api/mouse/move`) and
they can now ask what the mouse looks like (`api/mouse/cursor`).

The Sweeper Agent will:

1. Define a search area (a bounding box on the screen).
2. Move the mouse in a rapid grid pattern (like a Roomba or a lawnmower)
across that area.
3. Rapidly poll the cursor state at each coordinate.
4. If the cursor changes from a `left_ptr` (arrow) to a `hand2` (pointing
finger) or an `xterm` (I-beam text select), it records that $X,Y$ coordinate
as an **Interactable Hotspot**.
5. Once the sweep is complete, it outputs a list of all clickable elements
found in the dark.

Execute this block to create and test the Sweeper Agent:

```
[GEM-TERMINAL-START]
cat << 'EOF' | tr '~' ' ' > /opt/hauki-gem/sweeper_agent.py
#!/opt/hauki-gem/venv/bin/python3
import requests, time, sys

API_URL = "[http://127.0.0.1:7711/api](http://127.0.0.1:7711/api)"

def move_mouse(x, y):

def get_cursor():

def sweep_area(start_x, start_y, width, height, step=15):

if __name__ == "__main__":
EOF

chmod +x /opt/hauki-gem/sweeper_agent.py

echo "[+] Sweeper Agent built. Run ./sweeper_agent.py to test the Blind Man's
Cane!"
[GEM-TERMINAL-END]

```

### How to Test It

Because you are running the `vncserver` on `hauki-teacher`, the environment
might be a blank Linux desktop. If you run the sweeper right now, it will
probably just see the desktop background and find nothing.

1. **Before you run it:** Open a web browser or a GUI application inside your
VNC session (like a file manager) and position it near the top left corner.
2. **Run it:** `./sweeper_agent.py`

You will see the mouse physically rapidly scanning across the UI. Whenever it
crosses a button or link, the terminal will light up with `Hotspot found ->
hand2`.