AbeMarkn

Posted on Aug 23 Originally published at qiita.com

Automated Hardware Testing on Raspberry Pi Pico - Testing ePaper with a Test Jig, Camera, and Gemini API

#raspberrypi #python #embedded #ai

Introduction

AI has made software development much easier. But for embedded software, testing is still tough.

Embedded devices require pressing physical buttons and visually checking the screen. Because hardware is involved, manual testing is usually required.

I wanted to change that, so I built an automated testing setup!

Now, everything from pressing the buttons to verifying the ePaper display with AI is fully automated.

What I Built

I am building a battery-powered "Time Recorder" (punch clock) using a Raspberry Pi Pico and an ePaper display.

When a button is pressed, the device records the time and displays it on the screen.

In this project, I automated the entire test cycle:

Triggering the physical button press via hardware.
Waiting for the ePaper to update.
Capturing the screen with a camera.
Using the Gemini API to verify the displayed content automatically.

How It Works

Normally, pressing Button A (Fig 1: ★1) displays the recorded timestamp on the ePaper (Fig 1: ★2).

To automate this test, the system runs through these steps:

Custom Test Jig: Built a testing jig (Pico_Tester) to control the button lines (Fig 1: ★4).
Automated Button Press: The test jig triggers the button electronically (a green LED turns on so humans can see it was pressed).
Camera Capture & AI Verification: A Raspberry Pi 5 camera captures the ePaper screen (Fig 1: ★5), and the Gemini API inspects the image.
Automated Report: The AI evaluation and test result are reported automatically (Fig 2: ★3).

Fig 1: Overall System Setup

Fig 2: Test Result Report (Google Antigravity)

Fig 3: Circuit Diagram

Gemini API & Google Antigravity

Here is how AI is used in this test pipeline:

Prompt: Sends structured text instructions with the Base64 image, asking to extract text/status and compare it with the expected output.
Accuracy: Recognized custom 7-segment bitmap fonts, timestamps, and punch records (0002-0003, etc.) with 100% accuracy.
Implementation: Uses Python's standard urllib to call the gemini-3.6-flash REST API directly. No external SDKs required.
Integration: The AI output is parsed and saved into the test result JSON (snapshot info + AI verdict) as test evidence.
Development Speed: Pair-programmed with Google Antigravity to quickly design and deploy API endpoints, concurrency locking, timeouts, and live hardware tests.

Prompt Example (Generated by AIJudge)

This image is a photograph of an ePaper display.
Please read the content displayed on the screen and report in the following format:

1. [Extracted Text]: Transcribe all text, numbers, and symbols shown on the screen.
2. [Date/Time]: State any date and time displayed.
3. [Status]: State the status shown at the bottom (SLEEP/KEEP, READY/START, etc.).
4. [Verdict]: Compare with the expected criteria below and judge if it matches:
   Expected: The ePaper displays the date (MM/DD), punch record (HHMM-HHMM), current time at the bottom, and button guide.

   Answer with "OK" or "NG" and provide a concise reason.

Actual Output from Gemini API

1. [Extracted Text]
  - Date Header: 08/22
  - Left Column (Punch History):
      - 0002-0003
      - 0944-0945
      - 2322-2323
      - 2359-2343
      - 1102-1104
  - Bottom Area:
      - Guide: A=START B=FINISH C=KEEP
      - Clock: 08/22 23:04
      - Status: SLEEP READY

2. [Date/Time]
  - Top Header: 08/22
  - Bottom Clock: 08/22 23:04

3. [Status]
  - "SLEEP READY" is displayed on the bottom right.

4. [Verdict]
  - Result: OK
  - Reason: The date (08/22) at the top, punch records (0002-0003, etc.) on the left, current time (08/22 23:04), and button guide (A=START B=FINISH C=KEEP) at the bottom are all clearly displayed and completely match the expected criteria.

Key Benefits

Now I can define test cases on my PC, and the system runs them completely automatically.

Whenever I add a feature or fix a bug, I just tell the tool: "Run Test_0001", and it tests the hardware and generates a report.

No more pressing physical buttons dozens of times manually! No matter how many bugs I create, the automated setup tests tirelessly without complaining. 🎉

Next Steps

Refine and expand the AI evaluation prompts for different test cases.
Build the remaining firmware features for the Time Recorder now that the test loop is ready.
Use the test jig's INA228 power monitoring module and EEPROM to evaluate sleep current and battery life.

DEV Community