[GCP in Action] LINE Business Card Bot Evolution: Dual-Side Recognition and Merging with Gemini

Pain Point: One Chinese Business Card is Actually Two Business Cards

Business cards in Taiwan often have a common design: Chinese printed on the front and English on the back (or vice versa). Our LINE business card bot's original logic was very simple: receive an image, perform OCR once, and save one record.

This is where the problem lies. When a user sends the front side, the bot saves a record with only the Chinese name. If the user then sends the back side, the bot treats it as "another new business card," resulting in two records for the same person in the database, each missing half the information. Users have to manually compare and delete duplicate data, which is a terrible experience.

This article records how we taught the bot to recognize that "these are two sides of the same business card" and merge the information from both sides into a single complete record.

---

Solution: First Ask, "Is there a back side?"

Instead of writing rules to guess if two images are the same business card, we chose a more direct approach: Ask the user.

The workflow is designed as follows:

1. The user sends the front of the business card, and the bot performs OCR as usual.
2. After OCR is complete, the bot does not save immediately. Instead, it replies with "📇 Front side data recognized. Is there a back side to this card?" and provides two Quick Reply buttons.
3. User clicks "No, save directly" → Save according to the original process and finish.
4. User clicks "Yes, there's a back side" → The bot remembers the front image and waits for the next image.
5. Once the back side image arrives, both images are sent to Gemini together to be merged into a single record before saving.

We manage this waiting state using the user_states memory dictionary already present in the project, and add a 5-minute timeout. If a user clicks "Yes, there's a back side" but then ignores it or does something else, the process is treated as abandoned after 5 minutes, preventing the entire workflow from getting stuck.

user_states[user_id] = {
    'action': 'pending_backside_confirm',
    'card_obj': card_obj,
    'front_image_bytes': image_content,
    'expires_at': time.time() + PENDING_BACKSIDE_TIMEOUT_SECONDS
}

---

Core: Let Gemini See Two Images at Once and Merge Them

The most critical technical decision was: should we recognize the front and back sides separately and then write code to merge them? We chose another path: wrapping both the front and back images in a single generate_content request and letting Gemini handle the judgment directly.

The reason is simple: merging Chinese and English names into a format like "Wang Daming David Wang" using string rules is prone to being messy and inaccurate. Semantic-level integration is more likely to fail with hardcoded rules, so letting Gemini handle it directly is much easier.

In app/gemini_utils.py, we added generate_json_from_two_images, which reuses the existing NAMECARD_SCHEMA structured output, but this time the contents includes two image Parts:

def generate_json_from_two_images(
        front_img: PIL.Image.Image,
        back_img: PIL.Image.Image,
        prompt: str) -> object:
    model = GenerativeModel(
        "gemini-3-flash-preview",
        generation_config={
            "response_mime_type": "application/json",
            "response_schema": NAMECARD_SCHEMA
        },
    )
    front_part = Part.from_data(
        data=pil_to_bytes(front_img), mime_type="image/jpeg")
    back_part = Part.from_data(
        data=pil_to_bytes(back_img), mime_type="image/jpeg")
    response = model.generate_content(
        [prompt, front_part, back_part],
        stream=False,
        labels={"client_id": "namecard"}
    )
    return response

The prompt also just adds a merge instruction after the original IMGAGE_PROMPT:

DOUBLE_SIDED_IMAGE_PROMPT = IMGAGE_PROMPT + """
These two images are the front and back of the same business card; please integrate them into a single complete record.
If both Chinese and English appear in the same field (such as name or company), please present them merged
(e.g., "Wang Daming David Wang");
if a field appears on only one side, use the value from that side; ignore obviously redundant information.
"""

One API call means we don't have to write or maintain any merging rules ourselves.

---

Two Easily Overlooked Pitfalls

During the overall code review before the feature went live, we caught two details that are easily overlooked but can really cause issues.

Pitfall 1: Timing of Duplicate Checks

Originally, the duplicate check (comparing if the email already exists) was done immediately after OCR. However, after the double-sided recognition went live, if the front side happened to have a duplicate email from an old record, the process would prematurely determine it "already exists" and end. This would mean any new email on the back side would never be seen.

The fix was to move the duplicate check later, performing it only after "single-side chosen not to merge" or "double-sided merge complete." This ensures we are always comparing the final version of the data:

async def _finalize_and_save_card(card_obj, event, user_id) -> None:
    existing_card_id = firebase_utils.check_if_card_exists(card_obj, user_id)
    if existing_card_id:
        # ... Reply already exists
        return
    card_id = firebase_utils.add_namecard(card_obj, user_id)
    # ... Reply save successful

Both the single-sided and double-sided merge workflows eventually converge to call this shared function, ensuring the duplicate check is executed only once when the data is finalized.

Pitfall 2: Don't Clear All States Indiscriminately

The user_states dictionary is actually shared by several features: editing memos, modifying fields, and this back-side recognition. The initial implementation, for convenience, would delete the entire state whenever a residual state was detected before processing a new event.

The problem is: if a user is "editing the phone field" and waiting to input a new number, but accidentally sends an image, this logic would clear the editing_field state as well, silently canceling the user's original editing operation.

The fix was to only clear the two states related to the back-side recognition process and leave other states untouched:

if state.get('action') in (
    'pending_backside_confirm', 'awaiting_backside_image'
):
    del user_states[user_id]

We later found a missing branch in the same logic; the part handling "operation expired" replies also cleared everything initially. It was only truly fixed after unifying it with the same selective judgment.

---

Incidental Resource Cleanup: Don't Let Back-Side Images Linger in Memory

The awaiting_backside_image state stores not just text, but also the raw byte data of the front image. If a user disappears after being asked "Is there a back side?", this data would theoretically stay in the process memory because the original design only checked and cleared timeout states during the "user's next interaction."

We added a sweep_expired_states() function, which runs immediately when a Webhook comes in. It clears all expired temporary states for all users, so we don't have to wait for the specific user to return for passive cleanup:

def sweep_expired_states() -> None:
    now = time.time()
    expired_user_ids = [
        user_id for user_id, state in user_states.items()
        if 'expires_at' in state and state['expires_at'] <= now
    ]
    for user_id in expired_user_ids:
        del user_states[user_id]

Whenever any user sends a message, it performs garbage collection for all users, ensuring that those who abandoned the process don't leave behind memory-consuming remnants.

---

Summary and Benefits

This double-sided recognition and merging feature makes the LINE business card bot much more aligned with the actual usage habits of Taiwanese users:

1. Single Recognition, Complete Data: Both sides are sent to Gemini at once, automatically merging Chinese and English fields, eliminating the need for manual duplicate comparison.
2. Non-Intrusive: Ignoring prompts, timeouts, or temporarily doing something else will naturally revert to single-sided storage without getting the workflow stuck.
3. Duplicate Checks Target Final Data: Ensures that comparisons are always made against the merged, complete version, so new information appearing only on the back side isn't missed.
4. Independent State Machines: The temporary state for back-side recognition only affects itself and doesn't interfere with other ongoing user operations.
5. Clean Memory Usage: Proactively cleaning up timeout states ensures that users who abandon the process don't leave an invisible memory burden.

The complete code has been pushed to GitHub, feel free to check it out!