Why
Why I built it
Agents often stall on the last mile: checking a device display, tapping an app with no API, or asking someone for another photo. Rather than adding extra hardware, I used the phone people already carry to link observation, action, human help, and task continuation.
What I built
What I built
- Android sensing and control: camera, audio, location, notifications, screen, and UI actions, with Core / Hyper Mode separating permission levels.
- HUMAN HELP accepts text, choices, photos, or a fresh camera capture so the agent can pick up where it left off. The hackathon build also included an ESP32-S3 approval controller.
- A built-in Node.js Mobile runtime lets agents write files, run code, and manage background processes inside the app workspace.
- Local MCP and a Cloudflare relay, phone-confirmed OAuth for remote clients, and paired device access through PicoADB.
Engineering
Problems I worked through
- On connection loss, the system separates actions that never ran from ones that might have completed; reads can retry, while timed-out writes are never blindly replayed.
- Tool responses include recovery hints alongside parameters, helping agents handle missing permissions, locked screens, or offline ADB.
- Updates and auth repairs preserve device identity, pairing, and settings—exercised through repeated start, disconnect, and recovery runs on my own phone.
Role & AI
How I split the work with AI
I led the architecture, Android app, and MCP/relay integration while using AI coding agents for implementation and debugging. Permission gates and recovery paths were settled by running them repeatedly on my own phone.
Trade-offs
What testing made me change
- Built on standard Android permissions and explicit user consent so OS-level security boundaries stay intact.
Outcome
Outcome
Won third place at FUTUREMODE × SITCON Hackathon 2026, with public source and showcase. Kept evolving as a daily phone-tooling layer—including deploying PicoCar onto a real car head unit through the phone’s ADB route.