Project Log #16: I Fixed the App-Switching Problem That Kept Breaking My Agent

#ai #programming #webdev #productivity

Day 16. The agent now resets to home between apps. Three-app workflows are working.

Day 15 was a milestone: my agent completed its first multi-app task. It copied my bank balance and sent it to Mom on WhatsApp. Two apps. One task.

But behind the scenes, it was fragile. The agent sometimes opened apps to the wrong screen. The banking app would resume from where it left off instead of starting fresh. A notification from the previous app would cover the new one. The task worked—but only sometimes.

Today, I fixed the foundation.

The Home Screen Reset

Between every app switch, the agent now returns to the home screen first. It's a simple adb shell input keyevent KEYCODE_HOME command. But it solves a cascade of problems.

Before the reset, here's what would happen:

Agent opens Banking App. Reads balance. ✅
Agent opens WhatsApp. But WhatsApp was already open in the background on a different chat. ❌
Agent tries to find "Mom" but it's looking at the wrong screen. ❌
Task fails. ❌

After the reset:

Agent opens Banking App. Reads balance. ✅
Agent returns to home screen. ✅
Agent opens WhatsApp from scratch. It loads the main chat list. ✅
Agent finds "Mom" and sends the message. ✅

The fix is two lines of code. The reliability improvement is massive.

Today's Progress

Task	Status
Added `go_home()` between all app switches	✅ Done
Tested "Copy balance → send to Mom" 10 times	✅ 9/10 success
Tested "Copy address → paste into Maps" 10 times	✅ 10/10 success
Tested 3-app workflow: "Get last transaction from banking app → summarise → email to accountant"	✅ First success
Updated `agent.py` with home reset logic	✅ Done

The 3-App Workflow

The 3-app test was the real challenge. The command: "Get my last transaction from the banking app, summarise it, and send it to my accountant on WhatsApp."

The agent:

Opened the banking app. Read the last transaction: "₦15,000 debit — Electricity Bill — June 19." ✅
Stored it in task memory as last_transaction. ✅
Returned to home. Opened WhatsApp. ✅
Found "Accountant" in contacts. ✅
Retrieved last_transaction from memory. ✅
Typed: "Last transaction: ₦15,000 debit for Electricity Bill on June 19." ✅
Pressed send. ✅

Three apps. One task. Seven successful steps.

What Still Breaks

The banking app OCR is still unreliable. On 2 out of 10 tests, the agent misread a digit in the balance. A "₦15,000" became "₦15,000" in one test and "₦15,000" in another—but on the failures, it read "₦15,000" as "₦15,000" or missed a digit entirely. For sending a balance to Mom, that's a typo. For transferring money, that's a disaster.

I need a verification step specifically for financial data. Something that double-checks the OCR output before storing it in memory. That's the next wall to climb.

The home screen reset adds about 1.5 seconds between app switches. For a 3-app task, that's 4.5 extra seconds. Not a dealbreaker, but noticeable. I'm experimenting with using adb shell am force-stop instead of home reset—it's cleaner but takes longer.

What's Next (Day 17)