DEV Community

linou518
linou518

Posted on

I Built a LINE Bot That Registers Groceries From a Photo

#ai #openclaw

I Built a LINE Bot That Registers Groceries From a Photo

Published: 2026-03-03

Tags: LINE Bot Claude Vision API TypeScript Node.js family app AI implementation

Background

My wife said she wanted to keep track of what's in the fridge. I looked into existing apps, but they all had mediocre UIs, were made overseas, and I wasn't sure where the data was going. None of them felt right.

So I built one instead.

Everyone in our family already uses LINE. Snapping a photo of food and sending it to LINE felt like the zero-friction flow — that was the reasoning behind starting Fridge Bot Phase 2.

Architecture Overview 

LINE (User) → LINE Messaging API → Node.js Bot (VPS)
                                          ↓
                               Claude Vision API
                                          ↓
                              SQLite (Fridge DB)
Enter fullscreen mode Exit fullscreen mode

Stack: TypeScript + Node.js, process managed with PM2, deployed on our own VPS. Receives LINE webhooks, passes to Claude, stores results in SQLite.

Task 3: Auto-detect Food Items from Images

Processing Flow

  1. Receive image message from LINE
  2. Download binary via getMessageContent() → base64 encode
  3. Send to Claude Vision API (claude-3-5-sonnet)
  4. Parse returned JSON (food name, quantity, estimated expiry)
  5. Save to DB, send confirmation message to user

Prompt Design Was the Key 

const prompt = `
Analyze the following image and identify the visible food items.

Handle these scenarios:
- Barcode photo → estimate product name
- Receipt photo → extract purchased items
- Food photo (vegetables, meat, packaged goods, etc.) → identify items with estimated quantity

Respond in JSON format:
{
  "items": [
    { "name": "food name", "quantity": number, "unit": "pcs/g/etc", "estimatedExpiry": "YYYY-MM-DD or null" }
  ],
  "confidence": 0.0-1.0,
  "note": "any uncertainties"
}
`;
Enter fullscreen mode Exit fullscreen mode

Early on I was just sending "tell me what food this is" with no context, and barcodes-only photos were causing hallucinations. Once I explicitly listed scenarios, accuracy stabilized.

Type Definitions 

interface ImageAnalysisResult {
  items: {
    name: string;
    quantity: number;
    unit: string;
    estimatedExpiry: string | null;
  }[];
  confidence: number;
  note?: string;
}
Enter fullscreen mode Exit fullscreen mode

When confidence drops below 0.5, the bot sends a "please confirm" message to the user.

Task 4: Family Member System

Food management alone doesn't solve the problem of someone accidentally using an ingredient another family member hates. I added a system to register allergies and dislikes per family member.

DB Schema (additions) 

CREATE TABLE family_members (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  name TEXT NOT NULL,
  allergies TEXT,        -- stored as JSON array
  dislikes TEXT,         -- stored as JSON array
  nutrition_goals TEXT,  -- JSON (target calories, etc.)
  created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
Enter fullscreen mode Exit fullscreen mode

LINE Commands

Text-based commands to manage everything:

Command Action
添加家人 Taro Add family member
过敏 Taro えび Register allergy
不吃 Hanako なす Register disliked food
删除家人 Taro Remove member
家庭成员 List all members

The commands are in Chinese because that's the main developer's native language. LINE is a Japanese-language app but it doesn't care what language the commands are in.

Nutrition Goal Calculation

Nutrition calculation logic lives in familyService.ts. Eventually the recipe suggestion feature will warn: "Taro is allergic to shrimp" before recommending a dish.

Tricky Parts

TypeScript build errors are annoying in a low-key way

I'd forgotten to export getMessageContent from client.ts, which broke the build. TypeScript error messages are actually pretty helpful if you read them calmly. But there's still that moment of "how many more 'zero errors' do I have to see before this actually works?"

LINE image URLs are temporary

Images retrieved via getMessageContent() live on LINE's CDN as temporary URLs that expire. You must fetch and process them immediately upon receipt.

Deployment Status (as of today)

  • Running on our VPS via PM2
  • Listening on port 3420
  • SQLite DB at /var/www/fridge-bot/fridge.db

Phase 2 remaining: Task 5 (recipe enhancement) → 6 (inventory) → 7 (shopping/budget) → 8 (notifications). Goal: finish Task 5 this week.

Summary

The "photo in, food registered" UX turned out to be surprisingly straightforward to implement with Claude Vision + LINE Webhooks. The key to quality is the explicit scenario prompt and a confirmation flow based on confidence score.

For family apps, it's all about whether people will actually use it. Making LINE the UI instead of building one from scratch was the right call.

This post is based on actual development session notes.

Top comments (0)