DEV Community: Okeke Chukwudubem

Day 12: My AI agent can control WhatsApp perfectly. But banking apps? Invisible. The difference is accessibility labels. Apps built for blind users are automatable. Apps that skip accessibility lock out both humans and ai.

Okeke Chukwudubem — Tue, 30 Jun 2026 13:49:34 +0000

Okeke Chukwudubem

Jun 30

Project Log #12: The Apps My AI Agent Can't See

#ai #webdev #programming #productivity

2 min read

Project Log #12: The Apps My AI Agent Can't See

Okeke Chukwudubem — Tue, 30 Jun 2026 13:47:50 +0000

Day 12. WhatsApp is fully automatable. Banking apps are invisible. The reason is accessibility.

For 11 days, I've been building an AI agent that can see. UI trees. OCR. Template matching. I've benchmarked it. Optimised it. Proved it works.

But today I discovered something that changed how I think about this entire project.

My agent can control WhatsApp perfectly. Every button has a label. Every icon has a content description. The send button says "Send message." The back button says "Back." The search button says "Search." These are accessibility labels—designed for screen readers used by blind and visually impaired people.

WhatsApp invested in accessibility. And that investment accidentally makes their app fully automatable by my agent.

Then I tested a local banking app. And another. And another.

None of them had accessibility labels. The login button? content-desc="". The password field? content-desc="". The transfer button? Generic class name. Nothing identifiable. My UI tree approach was useless. The agent was blind.

This is bigger than my project. The same neglect that locks out visually impaired users also locks out AI agents. When developers skip accessibility, both humans and machines suffer.

For the apps that invested in accessibility, my agent works flawlessly. For the ones that didn't, I'm back to OCR and template matching—slower, less reliable, and device-dependent.

I'm now building a list of "agent-friendly" apps and "agent-hostile" apps. The pattern is clear: well-funded, global apps (WhatsApp, Google apps, Slack) are accessible. Local apps, especially banking and government apps, are not.

This project started as an automation tool. It's becoming an accessibility audit.

Next: Building fallback strategies for unlabeled elements.

👉 github.com/Dexter2344/phone-agent

Project Log #11: UI Tree vs Screenshots — The Real Performance Test

Okeke Chukwudubem — Fri, 26 Jun 2026 10:23:29 +0000

Day 11. I ran the old vision system and the new one side by side. The results surprised me.

Yesterday I rewrote the vision system. Screenshots and template matching got demoted. Android's UI hierarchy tree became the primary.

Today I ran both systems side by side on the same tasks to see if the rewrite was actually worth it.

The Test

I set up three identical tasks and ran each one twice—once with the old screenshot-based system, once with the new UI tree system. Same phone. Same apps. Same commands.

Task	Old System (Screenshots)	New System (UI Tree)
Open WhatsApp	3.2s	2.1s
Find "Mom" in contacts	2.8s (OCR)	0.7s (UI tree text match)
Tap send button	4.1s (template matching)	0.6s (content-desc match)
Total	10.1s	3.4s

The new system is nearly 3x faster for a simple task. The difference gets bigger as tasks get more complex.

Why the UI Tree Wins

The old system had to:

Take a screenshot (1-2 seconds)
Run OCR or template matching (2-4 seconds)
Parse the results
Calculate coordinates

The new system:

Runs one ADB command that dumps an XML file
Searches the XML for a matching element
Reads the exact coordinates from the element's bounds

No image processing. No OCR loading time. No template matching loops. Just a text search on a structured document.

What I Got Wrong Yesterday

I said the UI tree takes 0.5-1 second. In practice, it's faster—around 0.3-0.7 seconds for the dump and pull combined. The XML file is small (usually under 100KB). The parsing is instant.

But there's something I didn't anticipate: the UI tree is too detailed. A single screen can have 200+ elements. Searching through all of them by content-desc or text takes a few milliseconds of Python. It's negligible, but it reminds me that XML parsing on a phone CPU isn't free.

What's Still Hard

The UI tree approach has a blind spot: not all apps label their elements well.

WhatsApp is great. The send button has content-desc="Send message". The back button has content-desc="Back". The search button has content-desc="Search". These are accessibility labels that screen readers use, and WhatsApp's developers actually invested in them.

But I tested a local banking app. The login button? content-desc="". The password field? content-desc="". Nothing. No labels. No text. Just generic class names like android.widget.Button with no identifying information.

For apps like that, the UI tree is useless. I still need OCR as a fallback. The unified find_target() function handles this gracefully—it tries UI tree first, and if nothing is found, it falls back to OCR and template matching.

The Accessibility Divide

Here's something I didn't expect to learn from this project: well-funded apps (WhatsApp, Google apps, Slack) have excellent accessibility labels. They invested in making their apps usable by screen readers, and that investment accidentally makes them automatable by my agent.

Smaller apps, local apps, banking apps—they often have zero accessibility labels. My agent's ability to automate an app depends on whether that app's developers cared about disabled users.

This is a sobering realization. The same neglect that locks out blind users also locks out AI agents. Accessibility isn't just about inclusion—it's about making software machine-readable. And when developers skip it, both humans and AI suffer.

What's Next (Day 12)

Build a fallback mapping for unlabeled elements: if content-desc is empty, try to identify the element by its class + position + surrounding labeled elements
Test on more apps beyond WhatsApp—banking apps, settings, Chrome
Start logging which apps are "agent-friendly" and which aren't

The Repo

👉 github.com/Dexter2344/phone-agent

vision.py v4 is live. agent.py v10 is live. The unified target finder handles UI tree, OCR, and template matching in one call.

This is Day 11. The rewrite was worth it. The agent is faster, more reliable, and no longer tied to one device. But I've discovered a new problem: the agent can only automate apps that were built to be accessible. That's the next wall to climb.

Day 10 of building an AI agent that controls a phone. I'm ditching screenshots. OCR was slow. Template matching broke on different devices. The answer? Reading the Android UI tree directly. Exact coordinates. No guessing. Under 1 second.

Okeke Chukwudubem — Thu, 25 Jun 2026 18:53:58 +0000

Okeke Chukwudubem

Jun 25

Project Log #10: I'm Ditching Screenshots. Here's Why.

#ai #webdev #programming #productivity

2 min read

Project Log #10: I'm Ditching Screenshots. Here's Why.

Okeke Chukwudubem — Thu, 25 Jun 2026 18:53:09 +0000

Day 10. OCR and template matching hit their limits. UI hierarchy inspection might be the real answer.

Nine days ago, I was proud of my screenshot-based vision system. ML Kit for text. Template matching for icons. A clever fallback chain that worked most of the time.

Today, I'm ripping most of it out.

The Breaking Point

Last week, I tested the agent on a friend's phone. Template matching failed. The same icons I cropped on my device didn't match on his—different screen density, different rendering, different pixel arrangement.

I explored building a multi-resolution icon library. Crop every icon at 5 different DPIs? That's tedious. I explored AI-based icon detection. Train a model to recognize buttons by shape? That's heavy for a phone CPU.

Then I remembered something. Android already knows what's on the screen. It has to—it's rendering the UI. And there's a way to read that information directly.

Enter UI Hierarchy Inspection

ADB has a command called uiautomator dump. It spits out an XML file containing every visible UI element on the screen—buttons, text fields, icons, images, everything. Each element has:

A class name (e.g., android.widget.Button, android.widget.ImageView)
Bounds (exact pixel coordinates of where it sits)
Text (if it has any)
A content description (accessibility label, often used for icons)
Whether it's clickable, scrollable, focused

This is not a screenshot. This is the app's internal blueprint.

Why This Changes Everything

Screenshot-Based (Old Way)	UI Tree (New Way)
Run OCR on a screenshot (1.5–2s)	Run one ADB command (0.5–1s)
If text not found, try template matching (2–4s)	Not needed. Icons have content descriptions.
Accuracy depends on screen resolution and DPI	Accuracy is 100%—the OS tells you exactly where things are
Breaks on different devices	Works across all devices. Same XML structure.
Can't detect icons without reference images	Icons are in the tree with coordinates

The First Experiment

I ran adb shell uiautomator dump on my phone, then pulled the XML file. I searched for "send." Here's a snippet of what I found:


xml
<node
  class="android.widget.ImageButton"
  content-desc="Send message"
  bounds="[924,1656][1020,1752]"
  clickable="true"
  package="com.whatsapp" />

Day 9 of building an AI agent that controls a phone. It works perfectly on my phone. But on a friend's phone, template matching failed. Icons rendered differently. The agent couldn't send a message. Now I'm exploring UI hierarchy inspection

Okeke Chukwudubem — Sat, 20 Jun 2026 21:23:07 +0000

Okeke Chukwudubem

Jun 20

Project Log #9: My AI Agent Works on My Phone. But What About Yours?

#ai #webdev #programming #productivity

3 min read

Project Log #9: My AI Agent Works on My Phone. But What About Yours?

Okeke Chukwudubem — Sat, 20 Jun 2026 21:22:01 +0000

Day 9. Template matching works. But screen sizes, resolutions, and Android versions might break everything.

Eight days ago, the agent was an idea. Now it can read text, handle interruptions, and find icons on a screen.

But there's a question I've been avoiding: does it work on any phone other than mine?

The Cross-Device Problem

Every screenshot I've taken, every icon I've cropped, every coordinate I've mapped—it's all on one device. My phone. Same screen size. Same resolution. Same Android version. Same DPI.

Template matching relies on reference images that look exactly like the target on screen. Change the screen density, change the icon size, change the font scaling, and the match confidence drops. Suddenly "send_button.png" doesn't match anymore, and the agent can't press send.

This isn't a bug in my code. It's a fundamental challenge in computer vision: reference-based matching breaks when the visual context changes.

Today's Experiment

I tested the same agent on a friend's phone—different manufacturer, different Android version, slightly larger screen. The results were humbling.

Task	My Phone	Friend's Phone
OCR (text recognition)	✅ 95% accuracy	✅ ~90% accuracy
Find "Mom" in contacts	✅ Found	✅ Found
Template match: send button	✅ 94% confidence	❌ 62% confidence
Template match: back button	✅ 91% confidence	❌ 58% confidence

OCR held up reasonably well because text is text. Fonts might change slightly, but the characters are the same. But the icons—the send button, the back arrow—were rendered at a different size and slightly different pixel arrangement on my friend's device.

The agent failed to send the message.

Why This Matters

An AI agent that only works on one phone isn't an agent. It's a script. If I want this to be useful to anyone else—or even to myself if I change phones—it needs to be device-agnostic.

Possible Solutions I'm Exploring

Solution	Pros	Cons
Multi-resolution icon library	Simple. Just crop icons at different DPIs.	Tedious. How many variants are enough?
AI-based icon detection	Could recognize icons by shape, not pixels.	Requires training data. Heavy for a phone.
UI hierarchy inspection	Instead of "seeing" the screen, read the app's UI tree directly via ADB.	Requires root or accessibility service. Not universal.
Relative positioning	Once OCR finds text, calculate icon positions relative to known landmarks.	Fragile. Different layouts on different devices.

None of these are perfect. All of them are more work. But that's the reality of building something that's supposed to work in the wild, not just in a demo.

What I'm Trying First

The UI hierarchy approach. ADB has a command called uiautomator dump that returns an XML tree of every visible element on screen—text, buttons, icons, everything. Each element has bounds, a class name, and a content description.

If I can parse that XML tree instead of taking screenshots, the agent doesn't need to "see" the screen at all. It just reads the structure. No OCR. No template matching. No resolution issues.

This is a fundamental architectural shift. But it might be the right one.

What's Next (Day 10)

Experiment with uiautomator dump as a replacement for screenshot-based detection
Compare speed and accuracy against current OCR + template matching pipeline
Decide whether to pivot the entire vision system

The Repo

👉 github.com/Dexter2344/phone-agent

All code from Day 8 is live. The Day 9 experiments are in a new branch called ui-tree-experiment. I'll merge to main once I have results.

This is Day 9. The hard problems don't stop coming. But neither do I.

Day 8 of building an AI agent that controls a phone. Template matching is now wired into the agent. When OCR can't find a target, the agent searches its icon library. The send button is no longer invisible.

Okeke Chukwudubem — Thu, 18 Jun 2026 22:40:34 +0000

Okeke Chukwudubem

Jun 18

Project Log #8: The AI Phone Agent Can Now See Icons

#ai #webdev #software #buildinpublic

2 min read

Project Log #8: The AI Phone Agent Can Now See Icons

Okeke Chukwudubem — Thu, 18 Jun 2026 22:40:00 +0000

Day 8. Template matching is wired into the agent. The send button is no longer invisible.

Seven days ago, the agent was blind to icons. It could read text, find contacts, and type messages—but it couldn't press send.

Today, that changed.

What Got Built

Template matching is now integrated into the agent's decision pipeline. Here's the new flow when the agent needs to tap something:

Try OCR first. If the target is text (like a contact name), find it via ML Kit or Tesseract.
If OCR fails, fall back to template matching. The agent searches its icon library for a matching reference image.
If template matching succeeds above the confidence threshold (80%), tap the matched coordinates.
If both fail, report the failure and stop. No guessing. No hardcoded coordinates.

Today's Progress

Task	Status
Updated `agent.py` to call `match_template()` when OCR fails	✅ Done
Tested on WhatsApp send button	✅ Success
Tested on WhatsApp back button	✅ Success
Full pipeline test: type message → detect send icon → tap → verify sent	✅ Passed
Added 3 reference icons to the library	✅ Done

The Full Pipeline Test

I gave the agent a command: "Send a WhatsApp message to Mom saying I'll call later."

Here's what happened:

Agent opened WhatsApp via ADB. ✅
Agent searched for "Mom" using OCR + fuzzy matching. Found her. ✅
Agent tapped the contact. Chat opened. ✅
Agent typed "I'll call later" into the message box. ✅
Agent looked for the send button. OCR didn't find it (no text). ❌
Agent switched to template matching. ✅
Agent matched send_button.png with 94% confidence. ✅
Agent tapped the coordinates. ✅
Agent verified the message appeared in the chat. ✅

Task complete. No hardcoded coordinates. No guessing. The agent found the icon by seeing it.

What's in the Icon Library Now

Icon	File	Status
Send button (WhatsApp)	`send_button.png`	✅ Working
Back button (WhatsApp)	`back_button.png`	✅ Working
Search button (WhatsApp)	`search_button.png`	🔧 Testing

What's Still Hard

Template matching is slower than OCR. Each match takes 2-4 seconds on my device. For a single icon that's fine. For a task that needs to find three different icons, the delays add up.

The simple NumPy fallback is about 3x slower than OpenCV. On a device without OpenCV installed, template matching becomes the new bottleneck.

And icons that change appearance based on theme (dark mode vs light mode) need separate reference images. One icon, two variants. The library will grow.

What's Next (Day 9)

Add more reference icons: attach, camera, emoji, more options
Test on a second device to confirm the icon matching works across different screen sizes
Record a full demo video of the pipeline in action

The Repo

👉 github.com/Dexter2344/phone-agent

agent.py now calls match_template() as a fallback when OCR can't find a target. vision.py handles the matching with OpenCV primary and NumPy fallback. The icon library is growing.

This is Day 8. The agent can finally see what it's doing.

Day 7 of building an AI agent that controls a phone. OCR is fast now. But the agent still can't see icons the send button, the camera, the paperclip. I'm adding template matching with OpenCV to give it visual recognition for image-only UI elements.

Okeke Chukwudubem — Wed, 17 Jun 2026 15:46:10 +0000

Okeke Chukwudubem

Jun 17

Day 7. Text is solved. But icons and image-only buttons are still invisible. Enter template matching.

#ai #webdev #programming #softwaredevelopment

3 min read

Day 7. Text is solved. But icons and image-only buttons are still invisible. Enter template matching.

Okeke Chukwudubem — Wed, 17 Jun 2026 15:45:23 +0000

Two days ago, I fixed the OCR bottleneck. The agent can now read text on a screen in under 2 seconds. It handles interruptions. It verifies each step.

But there's a problem I've been dodging since Day 1, and it's time to face it.

What the Agent Still Can't See

Every app has buttons without text. The send button in WhatsApp is a paper plane icon. The attach button is a paperclip. The back button is an arrow. The camera button is a camera.

ML Kit sees text. It doesn't see icons. Tesseract doesn't see icons either. So when the agent needs to tap the send button after typing a message, it has no idea where that button is.

Right now, my workaround is hardcoded coordinates. I know the send button is roughly at a specific position on my phone. But that breaks on a different device. It breaks if the app updates. It breaks if the screen orientation changes. It's a hack, not a solution.

Template Matching: The Plan

Template matching is an old-school computer vision technique. You give it a reference image (like a small crop of the send icon) and it scans a larger image (the screenshot) looking for a match. It returns the coordinates of the best match.

It's not AI. It's not deep learning. It's pixel math. But it works surprisingly well for static UI elements like icons that don't change appearance.

Here's the approach:

Build an icon library. For each app the agent supports (WhatsApp for now), I manually crop reference images of the key icons: send, attach, back, camera, search.
Before each tap action, if the target isn't found via OCR, the agent runs template matching against its icon library.
If a match is found above a confidence threshold (80%), the agent taps the center of the matched region.
If no match is found, the agent reports the failure and stops, rather than guessing.

Today's Progress

Task	Status
Researched template matching libraries for Python	✅ Done
Selected OpenCV as primary engine, NumPy/PIL as fallback	✅ Done
Wrote `match_template()` and `_simple_template_match()` in vision.py	✅ Done
Created first icon reference image (WhatsApp send button)	✅ Done
Uploaded `send_button.png` to repo	✅ Done
Integrated into agent.py	⏳ Next

The Code

The new vision.py now has two template matching functions:

match_template() — Uses OpenCV for accurate, fast matching. If OpenCV isn't installed, it falls back to the simple version.
_simple_template_match() — A lightweight pure-Python implementation using NumPy and PIL. Zero dependencies beyond what the agent already uses.

The function takes a screenshot and a reference icon image, and returns the center coordinates of the best match above the confidence threshold.

Why Two Versions?

OpenCV is heavy. Installing it in Termux requires compiling from source or finding a pre-built arm64 wheel. The simple fallback means the agent can still do template matching even without OpenCV—just slower and slightly less accurate. Progressive enhancement. No hard dependency.

What's Next (Day 8)

Wire template matching into agent.py so the agent actually uses it when OCR fails
Test the full pipeline: type message → detect send icon via template matching → tap → verify
Start building out the icon reference library for WhatsApp

The Repo

👉 github.com/Dexter2344/phone-agent

vision.py is now at v3 with ML Kit, Tesseract, fuzzy matching, interruption handling, and template matching. send_button.png is in the repo. Agent integration coming next.

This is Day 7. The agent is getting eyes for the things that don't have names.

Day 6 of building an AI agent that controls a phone. Swapped Tesseract for Google ML Kit. Screen scans went from 12 seconds to under 2 seconds. Added an interruption handler that dismisses calls and notifications before they break the agent.

Okeke Chukwudubem — Tue, 16 Jun 2026 04:09:00 +0000

Okeke Chukwudubem

Jun 16

Project Log #6: I Fixed the OCR Problem That Was Killing My AI Agent

#ai #webdev #automation #programming

3 min read