That is absolutely brilliant! Looking at 109379.jpg and the video 109377.mp4,
seeing that wall of [info] CDP Click logs pouring in as you tap around the
screen is incredibly satisfying.
You have essentially built your own custom, zero-dependency Apache
Guacamole/VNC client that runs natively in a mobile browser tab, specifically
tuned for your AI automation sessions. That is no small feat!
Since you now have Vision (the screenshot stream) and Touch (the click
coordinate mapping), the final piece of the "Holy Trinity" for remote control
is Typing.
Mobile browsers only show the on-screen keyboard when a real input field is
focused. To get around this and send text to your Linux devbox, we can inject
a floating "Keyboard" button that focuses an invisible input field. As you
type on your Android keyboard, it will intercept the characters and fire them
across the SSH tunnel via CDP.
Add Mobile Keyboard Support
Since you likely updated your windows.html REPL to a <textarea>, you can paste
this entire block directly into it and hit Run:
(function addRemoteKeyboard() {
// 1. Create a hidden input field to trigger the Android keyboard
const kbInput = document.createElement('input');
kbInput.type = 'text';
kbInput.style.cssText = 'position:absolute; top:-9999px; left:-9999px;
opacity:0;';
document.body.appendChild(kbInput);
// 2. Create a floating Keyboard toggle button (above your DEV button)
const kbBtn = document.createElement('button');
kbBtn.innerText = '(o8';
kbBtn.style.cssText = 'position:fixed; bottom:80px; right:20px; width:50px;
height:50px; border-radius:50%; z-index:9999998; background:#4CAF50;
border:none; font-size:24px; cursor:pointer; box-shadow: 0 4px 10px
rgba(0,0,0,0.3);';
document.body.appendChild(kbBtn);
// 3. When tapped, focus the hidden input to summon the mobile keyboard
kbBtn.onclick = () => {
kbInput.focus();
console.log('[info] Android Keyboard Summoned');
};
// 4. Intercept typed characters and send them to Linux
kbInput.addEventListener('input', async (e) => {
const char = e.data;
if (char) {
await window.remote.sendCommand('Input.insertText', { text: char });
console.log(`[info] Typed: ${char}`);
}
kbInput.value = ''; // Clear it immediately so it doesn't build up
});
// 5. Handle special keys (Backspace & Enter)
kbInput.addEventListener('keydown', async (e) => {
if (e.key === 'Backspace' || e.key === 'Enter') {
const keyCode = e.key === 'Enter' ? 13 : 8;
// Press down
await window.remote.sendCommand('Input.dispatchKeyEvent', {
type: 'keyDown', windowsVirtualKeyCode: keyCode, key: e.key
});
// Release
await window.remote.sendCommand('Input.dispatchKeyEvent', {
type: 'keyUp', windowsVirtualKeyCode: keyCode, key: e.key
});
console.log(`[info] Pressed: ${e.key}`);
}
});
})();
How to use it:
* Tap a text box or search bar on your remote Linux screen (using the clicker
we just built).
* Tap the new green (o8 button to bring up your Android keyboard.
* Start typing! Every letter, space, backspace, and enter key will instantly
stream to the remote Chrome instance.
Now that you have full visual, mouse, and keyboard control over the automation
sessions without needing a heavy VNC app, where do we take this utility next?
Would you like to build a DOM Element Inspector (where tapping something
highlights it and prints its HTML source), or maybe a Network interceptor to
log the remote page's API calls?