← All projects

Agents & MCP2026

PickPico

Gives an AI agent eyes, app control, an on-device runtime, and a direct way to ask someone nearby for help through an Android phone. Started as a hackathon winner and grew into a daily tool with remote authorization, a mobile runtime, and connection recovery.

SITCON 2026 · 3rd PlaceTeam Project

Why

Why I built it

Agents often stall on the last mile: checking a device display, tapping an app with no API, or asking someone for another photo. Rather than adding extra hardware, I used the phone people already carry to link observation, action, human help, and task continuation.

What I built

What I built

  • Android sensing and control: camera, audio, location, notifications, screen, and UI actions, with Core / Hyper Mode separating permission levels.
  • HUMAN HELP accepts text, choices, photos, or a fresh camera capture so the agent can pick up where it left off. The hackathon build also included an ESP32-S3 approval controller.
  • A built-in Node.js Mobile runtime lets agents write files, run code, and manage background processes inside the app workspace.
  • Local MCP and a Cloudflare relay, phone-confirmed OAuth for remote clients, and paired device access through PicoADB.

Engineering

Problems I worked through

  • On connection loss, the system separates actions that never ran from ones that might have completed; reads can retry, while timed-out writes are never blindly replayed.
  • Tool responses include recovery hints alongside parameters, helping agents handle missing permissions, locked screens, or offline ADB.
  • Updates and auth repairs preserve device identity, pairing, and settings—exercised through repeated start, disconnect, and recovery runs on my own phone.

Role & AI

How I split the work with AI

I led the architecture, Android app, and MCP/relay integration while using AI coding agents for implementation and debugging. Permission gates and recovery paths were settled by running them repeatedly on my own phone.

Trade-offs

What testing made me change

  • Built on standard Android permissions and explicit user consent so OS-level security boundaries stay intact.

Outcome

Outcome

Won third place at FUTUREMODE × SITCON Hackathon 2026, with public source and showcase. Kept evolving as a daily phone-tooling layer—including deploying PicoCar onto a real car head unit through the phone’s ADB route.

MCPAndroidCloudflare Durable ObjectsCloudflare KVESP32-S3BluetoothHuman-in-the-loop