21 build logs. Months of work. One shipped project. Here's the full journey—and how to set up the Phone Agent on your own device.
The first build log was published months ago. "I'm building an AI agent that controls a phone." No code. No repo. Just an idea and a cracked phone.
Today, after 21 build logs spread across months, the project is shipped.
This wasn't a straight line. There were gaps. Weeks where the log went silent—not because the work stopped, but because life doesn't pause for build logs. I took breaks to survive exam season. I paused to ship 9 portfolio websites. I stepped away when the code refused to cooperate and my brain needed rest.
But every time I came back, the agent was still there. Waiting. And every log picked up where the last one left off.
What We Built
An autonomous AI agent that controls an Android phone using natural language commands. It can parse your words into actions, read the screen, tap buttons, type text, switch between apps, verify financial data, and serve a web interface—all offline.
The Real Timeline
| Phase | What Happened |
|---|---|
| Days 1-4 | Foundation. Gemma 4 + ADB. First working pipeline. |
| Days 5-8 | Vision overhaul. UI tree. OCR. Template matching. |
| Days 9-12 | Accessibility audit. 30 apps scored. |
| Days 13-16 | Multi-app workflows. Task memory. Home reset. |
| Days 17-19 | Financial verification. Accuracy from 80% to 94%. |
| Breaks | Exams. Portfolio sites. Life. |
| Days 20-21 | Web interface. Flask backend. Shipped. |
📖 FULL SETUP GUIDE: How to Install and Use the Phone Agent
Follow these steps to get the agent running on your own Android phone.
Prerequisites
- An Android phone (Android 7 or later)
- At least 6GB of free storage space
- A WiFi connection for the initial download
Step 1: Install Termux
Do NOT install Termux from the Google Play Store—that version is outdated. Install it from F-Droid instead.
- Open your phone's browser
- Go to
f-droid.org - Download and install the F-Droid app
- Open F-Droid and search for "Termux"
- Install Termux from F-Droid
Step 2: Set Up Termux
Open Termux and run these commands one by one. Wait for each to finish before typing the next.
bash
pkg update
pkg upgrade
pkg install python git curl wget
pkg install tesseract
pkg install android-tools
Step 3: Install Ollama
Ollama is the tool that runs Gemma 4 locally on your phone.
bash
curl -fsSL https://ollama.com/install.sh | sh
After installation, start Ollama:
bash
ollama serve
Open a new Termux session (swipe from left, tap "New session") and pull the Gemma 4 model:
bash
ollama pull gemma4:4b
This downloads about 2-3GB. Use WiFi. Be patient.
Step 4: Clone the Phone Agent Repository
bash
git clone https://github.com/Dexter2344/phone-agent.git
cd phone-agent
Step 5: Install Python Dependencies
bash
pip install flask flask-cors requests numpy PyPDF2 lancedb
Step 6: Enable Developer Options and ADB
Go to Settings → About Phone
Tap "Build Number" 7 times until you see "You are now a developer"
Go back to Settings → Developer Options
Enable "USB Debugging"
Connect your phone to itself via ADB:
bash
adb devices
You may need to approve a prompt on your phone. Once approved, you'll see your device listed.
Step 7: Start the Phone Agent
Using the Web Interface (Recommended):
bash
python server.py
Then open your phone's browser and go to: http://localhost:5000
You'll see a chat interface. Type commands like:
"Open WhatsApp and send a message to Mom saying I'll call later"
"Copy my bank balance and send it to Mom on WhatsApp"
Using the Terminal (Alternative):
bash
python agent.py
Then type commands directly into the terminal.
Troubleshooting
Problem Fix
"Ollama not found" Make sure you ran ollama serve in a separate Termux session
"ADB device not found" Check USB Debugging is enabled. Run adb kill-server then adb devices again
"Module not found" Run pip install for the missing module
"Tesseract not found" Run pkg install tesseract
Phone gets hot Close other apps. The agent is CPU-intensive. Take breaks.
Agent misreads numbers The verification layer will catch most errors. For financial data, always double-check.
Using the Web Interface
Once server.py is running and you've opened http://localhost:5000:
Status bar: Green dots mean Ollama and ADB are connected. Red dots mean something is wrong.
Command input: Type what you want the agent to do. Be specific.
Example commands: Click any example to auto-fill and send.
History: Your last 20 commands and responses are stored.
The Repo
👉 github.com/Dexter2344/phone-agent
21 build logs. Months of work. One shipped project. Full setup guide included.
Thank You
To everyone who followed this log—through the daily posts and the weeks of silence—thank you. This was never about shipping fast. It was about shipping honestly.
This was Log #1:
"I'm starting a new project. It's the most ambitious thing I've attempted from a phone."
This is Log #21. We shipped.
Top comments (1)
Interesting project. The verification layer for financial data caught my attention, especially the jump from 80% to 94% accuracy.
I’ve been working on a somewhat similar problem in AI agent tooling (MSCodeBase): an agent can execute the correct-looking action, and the verification can also succeed, while the underlying target or source was actually wrong. In our case, we had a case where symbol resolution selected a test fixture instead of the intended source — the execution itself was valid, but the conclusion was wrong.
That made me curious about your financial-data verification: when you double/triple-read an OCR result and the readings agree, how do you distinguish correctness from a consistent OCR error?
For example, if OCR reads the same wrong value twice, the verification layer would see strong agreement — but not necessarily truth.
Have you tested the verification layer with deliberately introduced but repeatable OCR errors?
Do you keep the evidence behind each verification (what was observed, when, and by which method)? I ask because a previously verified state can become stale when the UI changes, and that evidence helps distinguish “still valid” from “needs re-verification.”
I think there’s an interesting distinction here between verification of an observation and verification of the conclusion drawn from that observation.