DEV Community: Oliver S

From $USD to €EUR to MXN$: A Self-Improving AI Agent That Speaks All Currencies

Oliver S — Tue, 02 Sep 2025 09:29:16 +0000

Project: Multi-Currency Invoice

What this AI agent actually does?

This self-improving AI agent takes multi-currency invoices, extracts all data, and automatically normalizes all monetary values to a target currency (header currency) using historical exchange rates based on the invoice issue date. The crazy part? It gets smarter the more you use it.

Full code open source at: https://github.com/Handit-AI/handit-examples/tree/main/examples/multi-currency-invoice

Let's dive in!

What this AI agent actually does?
1. It gets smarter the more you use it (Best Part)
- Before Handit
- After Handit
2. Architecture Overview
3. Stack
- Tech Stack
- External APIs
4. Conclusions

1. It gets smarter the more you use it (Best Part)

I'm going to start with the best part — the cherry on top

Here’s a really cool thing — this AI agent actually gets better over time. Here is the secret weapon Handit.ai

Every action, every response is fully observed and analyzed. The system can see:

Which fields contain monetary values
What currencies are being used
When conversion is needed
How to apply real-time exchange rates
Whether normalization was successful
LLM output accuracy
LLM output format
And more...

And yes sir! When this powerful tool detects any mistakes it fixes automatically.

This means the AI agent can actually gets smarter the more you use it. If the LLM extracts the wrong field or generates incorrect outputs, Handit.ai tracks that failure and automatically adjusts the AI agent to prevent the same mistake from happening again. It's like having an AI engineer who is constantly monitoring, evaluating and improving your AI agent.

Here are the results before and after processing:

Before Handit

Prompt:

`You are a financial data processor. Your task is to normalize all monetary values in the following JSON data to ${targetCurrency}.

IMPORTANT: Use the convert_currency tool whenever you encounter a monetary value that needs to be converted to ${targetCurrency}.

The data structure follows this pattern:
- Each field has a "value" property containing the actual data
- Monetary values may be in various currencies (USD, EUR, GBP, etc.)
- You need to identify monetary values and convert them to ${targetCurrency}
- Use the convert_currency tool for each conversion needed

JSON Data to process:
${JSON.stringify(data, null, 2)}

Instructions:
1. Analyze the JSON data to identify all monetary values
2. For each monetary value that is NOT in ${targetCurrency}, use the convert_currency tool
3. The tool will return conversion results that you should use to add a "normalizedValue" field
4. Preserve the original structure completely - only add the normalizedValue field
5. If a value is already in ${targetCurrency}, you can skip conversion but still add a normalizedValue field with the same values

Remember: Always use the convert_currency tool for conversions. Do not attempt to convert currencies manually.`

Input: Multi-currency Invoice with mixed currencies

Ouput:

{
   "extractedData": {
    "documentType": {
      "value": "invoice",
      "confidence": 0.98,
      "reason": "Large header text 'INVOICE' at top of document (clear OCR region)."
    },
    "headerCurrency": {
      "value": "SGD",
      "confidence": 0.62,
      "reason": "The printed final/Total line shows '5000 SGD' which indicates the invoice's presented total currency. Document contains multiple currencies, but the final Total is in SGD."
    },
    "invoiceDate": {
      "value": "16-04-2024",
      "confidence": 0.99,
      "reason": "Date field shows '04/16/2024' near the top; converted to DD-MM-YYYY as requested."

...

    "lineItems": [
      {
        "description": "Consulting",
        "quantity": null,
        "rate": null,
        "amount": "5,000.00",
        "currency": "SEK",
        "confidence": 0.82,
        "reason": "Row shows 'Consulting' with amount '5,000.00 SK'. Interpreted as 5,000.00 SEK (subtotal later also shows SEK), symbol truncated in row but subtotal confirms SEK."
      },
      {
        "description": "Software License",
        "quantity": null,
        "rate": null,
        "amount": "150,000",
        "currency": "JPY",
        "confidence": 0.95,
        "reason": "Row shows '¥150,000' next to 'Software License' — yen symbol indicates Japanese Yen."
      },
      {
        "description": "Maintenance",
        "quantity": null,
        "rate": null,
        "amount": "1,000.0",
        "currency": "CAD",
        "confidence": 0.94,
        "reason": "Row displays '1,000.0 CAD' beside 'Maintenance' (explicit CAD currency code)."
      },

...

    "_conversions": [
    {
      "originalArgs": {
        "amount": 150000,
        "fromCurrency": "JPY",
        "toCurrency": "SGD"
      },
      "result": {
        "convertedAmount": 1309.5,
        "originalAmount": 150000,
        "fromCurrency": "JPY",
        "toCurrency": "SGD",
        "exchangeRate": 0.00873,
        "conversionDate": "2025-09-01",
        "method": "api_conversion",
        "apiResponse": {
          "originalAmount": 150000,
          "originalCurrency": "JPY",
          "convertedAmount": 1309.5,
          "targetCurrency": "SGD",
          "exchangeRate": 0.00873,
          "date": "2025-09-01",
          "success": true,
          "apiVersion": "v4",
          "historical": false
        }
      },
      "timestamp": "2025-09-01T20:35:55.758Z",
      "status": "unmatched"
    },
    {
      "originalArgs": {
        "amount": 1000,
        "fromCurrency": "CAD",
        "toCurrency": "SGD"
      },
      "result": {
        "convertedAmount": 934,
        "originalAmount": 1000,
        "fromCurrency": "CAD",
        "toCurrency": "SGD",
        "exchangeRate": 0.934,
        "conversionDate": "2025-09-01",
        "method": "api_conversion",
        "apiResponse": {
          "originalAmount": 1000,
          "originalCurrency": "CAD",
          "convertedAmount": 934,
          "targetCurrency": "SGD",
          "exchangeRate": 0.934,
          "date": "2025-09-01",
          "success": true,
          "apiVersion": "v4",
          "historical": false
        }
      },
      "timestamp": "2025-09-01T20:35:55.856Z",
      "status": "unmatched"
    }
  ]
}

Whoa! It looks like the output isn’t what we expected

First issue ❌: The extracted output does not adhere to the system prompt's requirement to add a 'normalizedValue' field for each monetary value
Second issue ❌: There is an array at the end called _conversions, but it doesn’t contain all the conversions

Results:

Accuracy: 44%
Correctness Evaluation: 48%

Lack of clarity on output format for monetary values.

After Handit

First, let’s see what results Handit gives us

Here it is — it highlights several insights, including the fact that the extracted output does not adhere to the system prompt's requirement to add a 'normalizedValue' field.

HERE IS THE BEST PART!

It automatically sent us a PR with the fixes from our AI agent, ready to deploy, reaching an accuracy of 90%!

After merging the PR, here are the results:

Input: Same Multi-currency Invoice

Output: All amounts normalized to SGD with normalizedValue fields

{
  "extractedData": {
    "lineItems": [
      {
        "description": "Consulting",
        "quantity": null,
        "rate": null,
        "amount": "5,000.00",
        "currency": "SEK",
        "fromCurrency": "SEK",
        "toCurrency": "SGD",
        "confidence": 0.82,
        "reason": "Row shows 'Consulting' with amount '5,000.00 SEK'. Interpreted as 5,000.00 SEK (subtotal later also shows SEK).",
        "normalizedValue": 683.41
      },
      {
        "description": "Software License",
        "quantity": null,
        "rate": null,
        "amount": "150,000",
        "currency": "JPY",
        "fromCurrency": "JPY",
        "toCurrency": "SGD",
        "confidence": 0.95,
        "reason": "Row shows '¥150,000' next to 'Software License' — yen symbol indicates Japanese Yen.",
        "normalizedValue": 1309.5
      },
      {
        "description": "Maintenance",
        "quantity": null,
        "rate": null,
        "amount": "1,000.0",
        "currency": "CAD",
        "fromCurrency": "CAD",
        "toCurrency": "SGD",
        "confidence": 0.94,
        "reason": "Row displays '1,000.0 CAD' beside 'Maintenance' (explicit CAD currency code).",
        "normalizedValue": 934
      },

...

    ],
...
  }
}


}

The mistakes have been fixed automatically!

All monetary values now have normalizedValue fields with:

✅ Converted amounts in SGD
✅ Original amounts preserved

Results:

Accuracy: 90%
Correctness Evaluation: 90%
Overall Performance Boost: ↗️ +46%

2. Architecture Overview

So I built this thing called "Multi-Currency Invoice", and honestly, it's doing some pretty wild stuff. Let me break down what's actually happening under the hood.

Let's understand the architecture of our AI agent at a very high level:

This architecture separates concerns into distinct nodes:

Ingestion
- Purpose: Manages file uploads and session organization for invoice processing
- Input: Images, PDFs, documents (max 10 files, 50MB each)
- Output: Organized file storage with session directories
- Core capability: File ingestion pipeline with automatic session management
Extraction
- Purpose: AI analyzes uploaded documents and extracts all data from invoices and other financial documents
- Input: Images, PDFs, Office documents with base64 encoding support
- Output: Structured JSON containing all extracted fields and confidence scores
- AI capability: Multimodal analysis (vision + text), parallel processing, and VLLM support
CurrencyNormalization
- Purpose: AI normalizes monetary values across different currencies to a target currency using intelligent analysis
- Input: Structured JSON containing all fields extracted in the previous step
- Output: Normalized JSON containing all fields, converted values, and applied exchange rates
- API capability: Historical exchange rates based on the invoice issue date
- AI capability: Detects monetary value that needs to be converted

And... How does this AI agent gets better over time?

Here is the secret weapon: Handit.ai

Observability
- Every interaction with this AI agent is monitored by handit
Failure Detection
- Handit automatically identifies errors in any of our LLMs — like when a monetary value was not normalized correctly (Really important for this AI agent)
Automated Fix Generation
- If a failure is detected, Handit automatically fixes our prompts for us

Want to deep dive into the tools, prompts, and nodes? Check the repo here: https://github.com/Handit-AI/handit-examples/tree/main/examples/unstructured-to-structured

3. Stack

Tech Stack

Framework: Node.js 18+, Express 4 (MVC)
Middleware: helmet, cors, morgan, express.json/urlencoded, multer (uploads)
Config/Runtime typing: dotenv, zod
AI/LLM: LangChain (langchain, @langchain/openai); OpenAI models (OPENAI_MODEL) — default gpt-5-mini-2025-08-07 (extraction) and gpt-4o-mini (normalization)
Observability, Evaluation and Optimization: @handit.ai/node

External APIs

OpenAI API
- Purpose: LLM for document data extraction (multimodal) and currency normalization with tools
- Access: LangChain/OpenAI SDKs
- Auth: OPENAI_API_KEY
ExchangeRate-API
- Purpose: Latest and historical FX rates
- Endpoints:
- v4 (free, latest): https://api.exchangerate-api.com/v4/latest/{base}
- v6 (historical, with key): https://v6.exchangerate-api.com/v6/{API_KEY}/history/{base}/{year}/{month}/{day}[/{amount}]
- Auth: EXCHANGE_RATE_API_KEY (required for v6)
Handit.ai
- Purpose: Execution tracing/observability, Evaluation, Optimization
- Access: @handit.ai/node SDK
- Auth: HANDIT_API_KEY (required to start server)

4. Conclusions

Thanks for reading!

I hope this deep dive into building a self-improving AI agent that speaks all currencies has been valuable for you

The project is fully open source - feel free to:
🔧 Modify it for your specific needs
🏭 Adapt it to any industry you want
🚀 Use it as a foundation for your own AI agents
🤝 Contribute improvements back to the community

Full code open source at: https://github.com/Handit-AI/handit-examples/tree/main/examples/multi-currency-invoice

This project demonstrates how to:

Integrate LangChain with real APIs for intelligent document processing
Use LLM tools for automated currency conversion
Handle multi-currency documents intelligently
Preserve document structure while adding normalized values
Fetch real-time exchange rates for accurate conversions
Build a self-improvement AI agent gets smarter the more you use it

This project comes with Handit.ai configured. If you want to configure Handit.ai for your own projects, I suggest following the documentation: https://docs.handit.ai/quickstart

Build a self-improving AI agent that turns documents into structured data (with LangGraph)

Oliver S — Wed, 20 Aug 2025 01:46:23 +0000

Project: Unstructured to structured

What this AI agent actually does?

This self-improving AI agent takes messy documents (invoices, contracts, medical reports, whatever) and turns them into clean, structured data and CSV tables. But here's the kicker - it actually gets better at its job over time.

Full code open source at: https://github.com/Handit-AI/handit-examples/tree/main/examples/unstructured-to-structured

Let’s dive in!

What this AI agent actually does?
1. The self-improvement (Best Part)
- Before Handit
- After Handit
2. Architecture Overview
- Another Perspective of the Workflow 🧠
3. Inference Schema Node - the schema detective
4. Document Data Capture Node - the data extractor
5. Generate CSV Node - the table builder
6. Conclusions

1. The self-improvement (Best Part)

I’m going to start with the best part — the cherry on top

Here’s a really cool thing — this AI agent actually gets better over time. Here is the secret weapon Handit.ai

Every action, every response is fully observed and analyzed. The system can see:

If the schema inferences worked well
Which data extractions failed
How long processing takes
What document types cause issues
When the LLM makes mistakes
If the CSVs generated are filled with real data
And more...

And yes sir! When this powerful tool detects any mistakes it fixes automatically.

This means the AI agent can actually improve itself. If the LLM extracts the wrong field or generates incorrect schemas or CSVs, Handit.ai tracks that failure and automatically adjusts the AI agent to prevent the same mistake from happening again. It's like having an AI engineer who is constantly monitoring, evaluating and improving your AI agent.

Here are the results before and after implementing Handit.ai:

Before Handit

Input: Purchases Order

Output:

Whoa! It looks like our AI agent is making some mistakes.

First mistake ❌: In Table 2, it duplicated two columns — item_code and sku. We don’t need both because they represent the same thing.

Second mistake ❌: This is the most crucial one. The Epson unit price is NOT 549.99, and the total is also NOT 549.99. This is very critical when dealing with important data.

After Handit

First, let’s see what results Handit gives us

Here it is — it highlights several insights, including that there are inconsistencies in the data extraction. Handit automatically detected the issue for us.

HERE IS THE BEST PART!

It automatically sent us a PR! with the fixes from our AI agent, ready to deploy, boosting accuracy by 43%.

After merging the PR, here are the results:

Input: Same Purchases Order

Output:

The mistakes have been fixed automatically!

Table 2 now contains only one code column — item_code — and groups everything correctly. ✅

The Epson unit price is now correct at 549.99, and the total is correct as well. ✅

2. Architecture Overview

So I built this thing called “Unstructured to Structured”, and honestly, it’s doing some pretty wild stuff. Let me break down what’s actually happening under the hood.

Let’s understand the architecture of our AI agent at a very high level:

This architecture separates concerns into distinct nodes:

inference_schema
- Purpose: AI analyzes uploaded documents to create a unified JSON schema
- Input: Images, PDFs, text files
- Output: Structured schema defining data fields and relationships
- AI capability: Multimodal analysis (vision + text)
document_data_capture
- Purpose: Maps document content to the inferred schema using AI extraction
- Input: Documents + inferred schema
- Output: Structured JSON with field mappings
- AI capability: Field extraction with confidence scores
generate_csv
- Purpose: Convert structured JSON into clean CSV tables
- Input: Structured JSON from the previous node
- Output: CSVs files ready for analysis
- AI capability: Intelligent table structure planning

And... How does this AI agent gets better over time?

Here is the secret weapon: Handit.ai

Observability
- Every interaction with our AI agent is monitored by handit
Failure Detection
- Handit automatically identifies errors in any of our LLMs — like when a CSV file doesn’t contain the right content (Really important for this AI agent)
Automated Fix Generation
- If a failure is detected, Handit automatically fixes our prompts for us

Another Perspective of the Workflow 🧠

Think of it as a smart pipeline that processes documents step by step. Here's what happens:

You upload documents - like invoices, contracts, medical reports (any type)
The agent analyzes everything - it looks at all your documents and figures out the best structure (schema)
It creates a unified schema - one JSON schema that can represent all your documents
Then extracts the data - maps each document to the schema with AI
Finally builds tables - creates CSV files and structured data you can actually use
Self-improvement - Handit observes every interaction with our agents, and if a failure is detected, it fixes it for us

3. Inference Schema Node - the schema detective

This is where the magic starts:

Reads images, PDFs, and text
Proposes a unified JSON schema that fits everything
Works across any document type
Adds useful field types and reasoning


class SchemaField(BaseModel):
    """
    Represents a single field in the inferred schema.

    Each field defines the structure, validation rules, and metadata for a piece
    of data extracted from documents. Fields can be simple (string, number) or
    complex (objects, arrays) depending on the document structure.
    """

    name: str = Field(description="Field name")
    types: List[str] = Field(description="Allowed JSON types, e.g., ['string'], ['number'], ['string','null'], ['object'], ['array']")
    description: str = Field(description="What the field represents and how to interpret it")
    required: bool = Field(description="Whether this field is commonly present across the provided documents")
    examples: Optional[List[str]] = Field(default=None, description="Representative example values if known")
    enum: Optional[List[str]] = Field(default=None, description="Enumerated set of possible values when applicable")
    format: Optional[str] = Field(default=None, description="Special format hint like 'date', 'email', 'phone', 'currency', 'lang' etc.")
    reason: str = Field(description="Brief rationale for inferring this field (signals, patterns, layout cues)")


class SchemaSection(BaseModel):
    """
    Logical grouping of fields to organize the schema structure.

    Sections help organize related fields into meaningful groups rather than
    having all fields in a flat list. This improves schema readability and
    makes it easier to understand the document structure.
    """

    name: str = Field(description="Section name (generic), e.g., 'core', 'entities', 'dates', 'financial', 'items', 'metadata'")
    fields: List[SchemaField] = Field(description="Fields within this section")


class InferredSchema(BaseModel):
    """
    Top-level inferred schema for a heterogeneous set of documents.

    This schema represents the unified structure that can accommodate various
    document types and formats. It combines common patterns found across
    multiple documents into a single, flexible schema definition.
    """

    title: str = Field(description="Human-readable title of the inferred schema")
    version: str = Field(description="Schema semantic version, e.g., '0.1.0'")
    description: str = Field(description="High-level description of the schema and how it was inferred")

    common_sections: List[SchemaSection] = Field(description="Sections that apply broadly across the provided documents")
    specialized_sections: Optional[Dict[str, List[SchemaSection]]] = Field(
        default=None,
        description="Optional mapping of document_type -> sections specific to that type",
    )

    rationale: str = Field(description="Concise explanation of the main signals used to infer this schema")

system_prompt = """
You are a senior information architect. Given multiple heterogeneous documents (any type, any language), infer the most appropriate, general JSON schema that can represent them.

Guidance:
- Infer structure purely from the supplied documents; avoid biasing toward any specific document type.
- Use lower_snake_case for field names.
- Use JSON types: string, number, boolean, object, array, null. When a field may be missing, include null in its types.
- Allow nested objects and arrays where the documents imply hierarchical structure.
- Include brief, useful descriptions for fields when possible without inventing content.
- Return ONLY JSON that matches the provided Pydantic model for an inferred schema.

 Per-field requirements:
 - For each field, add a short 'reason' explaining the signals used to infer the field (keywords, repeated labels, table headers, layout proximity, visual grouping, etc.).
"""

4. Document Data Capture Node - the data extractor

Maps every uploaded document’s content to the inferred schema using AI extraction


system_prompt = """
You are a robust multimodal (vision + text) document-to-schema mapping system. Given an inferred schema and a document (image/pdf/text), analyze layout and visual structure first, then map fields strictly to the provided schema.

Requirements:
- Use the provided schema as the contract for output structure (keep sections/fields as-is).
- For each field, search labels/headers/aliases using the 'synonyms' provided by the schema and semantic similarity (including multilingual variants).
- Prioritize visual layout cues (titles, headers, table columns, proximity, group boxes) before plain text.
- Do NOT invent values. If a value isn't found, set it to null and add a short reason.
- For every field, include a short 'reason' explaining the mapping (signals used) and a 'normalized_value' when applicable (e.g., date to ISO, amounts to numeric, emails lowercased, trimmed strings).
- Return ONLY a JSON object that mirrors the schema sections/fields. Each field should be an object: {{"value": <any|null>, "normalized_value": <any|null>, "reason": <string>, "confidence": <number optional>}}.
"""

5. Generate CSV Node - the table builder

Finally, it creates structured tables from all your data:

system_prompt = """You are a data shaping assistant.

You are given a set of JSON documents with the same schema (same keys & depth).

Your job is to analyze the documents and create 1..N CSV tables that include ALL the data from the files, but omit any 'reason' or 'confidence' values.

IMPORTANT: You must analyze the actual structure of the documents provided and create tables based on what you find, not on assumptions.

CRITICAL EXTRACTION RULES:
- ALWAYS check for "normalized_value" first, then "value" if normalized_value is null/empty
- If a field has both "normalized_value" and "value", prefer "normalized_value"
- If "normalized_value" is null/empty, use "value"
- Double check the data, if the data is correct
- Extract the actual string/number values, not the field objects

Rules:
- Analyze the actual JSON structure provided in the documents
- Create as many tables as needed to organize the data clearly
- Include ALL data fields from the documents (except reason/confidence)
- Skip 'reason' and 'confidence' fields completely
- Prefer 'normalized_value' over 'value' when both exist
- Make table and column names clear and descriptive
- Use lower_snake_case for naming
- If you see arrays, consider if they should be separate tables
- If you see nested objects, consider if they should be flattened or separate tables
- Be smart about data organization - group related fields together...

Want to deep dive into the tools, prompts, and nodes? Check the repo here: https://github.com/Handit-AI/handit-examples/tree/main/examples/unstructured-to-structured

6. Conclusions

Thanks for reading!

I hope this deep dive into building a self-improving AI agent has been useful for your own projects.

The project is fully open source - feel free to:
🔧 Modify it for your specific needs
🏭 Adapt it to any industry (healthcare, finance, retail, etc.)
🚀 Use it as a foundation for your own AI agents
🤝 Contribute improvements back to the community

Full code open source at: https://github.com/Handit-AI/handit-examples/tree/main/examples/unstructured-to-structured

This project comes with Handit.ai configured. If you want to configure Handit.ai for your own projects, I suggest following the documentation: https://docs.handit.ai/quickstart

How to build a self-improving agent that updates your UI in real time

Oliver S — Thu, 07 Aug 2025 18:44:49 +0000

What my AI agent actually does (and why it's pretty cool)

Invoice Copilot: Talk to your invoices. This self-improving AI agent answers questions and creates visual reports in real-time

So I built this thing called Invoice Copilot, and honestly, it's doing some pretty wild stuff. Let me break down what's actually happening under the hood.

The problem I was trying to solve

You know how most AI agents are pretty static? They follow some instructions, give you an output, and if an error occurs nothing happens until the engineers modify something manually. My agent has super powers - it actually improves itself! Literally, it has autonomous fixes.

Like, imagine you have a bunch of invoices, you upload them and you ask the AI: "Hey, show me a bar chart of my monthly expenses." Most AI tools would just give you the chart and if there is an error about the sum for example, nothing happens. Mine actually creates the React component with the chart, updates your UI in real-time, and if the LLM drifts, for example the sum was wrong 😱, it actually detects the issue and fixes itself so it never happens again.

Full code open source at: https://github.com/Handit-AI/invoice-copilot

Let's dive in!

What my AI agent actually does (and why it's pretty cool)
The problem I was trying to solve
1. Architecture Overview
2. Setting Up Your Environment
- Backend
- Frontend
3. The Core: MainDecisionAgent 🧠 (the boss)
4. Action Classes: Specialized Tools for Every Task 🎯
- EditFileAction - the UI builder
- SimpleReportAction - the data analyst
- OtherRequestAction - the polite redirector
5. The self-improvement (Best Part)
- 1. Let's setup Handit.ai observability
- 2. Set up evaluations
- 3. Set up self-improvement (very interesting part)
6. Results
7. Conclusions

1. Architecture Overview

Let's understand the architecture of our Invoice Copilot:

This architecture separates concerns into distinct nodes:

MainDecisionAgent: This node is responsible for analyzing user requests and decide which actions to take based on the request type, current context, and execution history.
Update File Agent: This is the most sophisticated action class, responsible for generating complete React components with professional charts and visualizations using the Recharts library.
Load Invoice Data: This function scans the processed/ directory for JSON files containing invoice data that has been processed by Chunkr AI.
Simple Report Action: This action is used when users ask for specific information about the data that doesn't require charts or complex visualizations. It provides direct answers to questions.
Other Request Action: This action is used when users make requests that are outside the scope of business reporting or data visualization.
Finish Action - Exit Loop: If the MainDecisionAgent believes no more actions are needed, then proceeds to return a finish
Format Response: This method takes the complete execution history and generates a professional response that explains what was accomplished, addresses the user's original request, and provides helpful next steps.

2. Setting Up Your Environment

Backend

1. Clone the Repository

git clone https://github.com/Handit-AI/invoice-copilot.git
cd invoice-copilot/backend

2. Create Virtual Environment

# Create virtual environment
python -m venv .venv

# Activate virtual environment
# On macOS/Linux:
source .venv/bin/activate

# On Windows:
.venv\Scripts\activate

3. Install Dependencies

# Install dependencies
pip install -r requirements.txt

4. Environment Configuration

# Copy environment example
cp env.example .env

5. Configure API Keys

Edit the .env file and add your API keys:

# Required API Keys
# Get your API key from: https://platform.openai.com/api-keys
OPENAI_API_KEY=your_openai_api_key_here
# Get your API key from: https://www.handit.ai/
HANDIT_API_KEY=your_handit_api_key_here
# Get your API key from: https://chunkr.ai/
CHUNKR_API_KEY=your_chunkr_api_key_here

# Optional Configuration
OPENAI_MODEL=gpt-4o-mini-2024-07-18

6. Run the Application 🚀

Development Mode

# Make sure virtual environment is activated
source .venv/bin/activate  # macOS/Linux
# or
.venv\Scripts\activate     # Windows

# Start the FastAPI server
python main.py

The server will start on http://localhost:8000

Frontend

1. Clone the Repository

#If you have cloned before, then skip this
git clone https://github.com/Handit-AI/invoice-copilot.git
#Go to the frontend folder
cd invoice-copilot/frontend

2. Install Dependencies

# Using npm
npm install

# Using yarn
yarn install

3. Environment Configuration

# Copy environment example
cp .env.example .env

4. Run the Application 🚀

Development Mode

# Start development server
npm run dev

# Or with yarn
yarn dev

The application will start on http://localhost:3000

3. The Core: MainDecisionAgent 🧠 (the boss)

Think of it as the brain that decides what to do next. Here's what happens:

You ask for something - like "create a pie chart of expenses by category"
The agent thinks about it - it looks at your request and decides what tool to use
It picks the right action - maybe it needs to edit a file, or maybe just give you a simple answer
Then assigns the task to the correct node - For example, if it is something that requires coding a graph then, it actually creates the React component of the graph you asked for
It tells you what it did - gives you a nice summary

Here is a summary of the prompt and the main function:

def analyze_and_decide(self, user_query: str, execution_id: str, 
                      history: List[Dict[str, Any]], working_dir: str = "") -> Dict[str, Any]:
    # It looks at what you asked for
    # It checks what it's done before (so it doesn't repeat itself)
    # It decides what tool to use next

    system_prompt = f"""You are a professional report and data visualization specialist. 
    Given the following request, decide which tool to use from the available options.

    Available tools/actions:

     1. edit_file: Create or edit professional reports with data                          visualizations, if graphs are needed to complete the user request
        - Parameters: target_file, instructions, chart_description
        - Example:
          tool: edit_file
          reason: I need to create a professional report...
     [...more info...]

    2. simple_report: ONLY If the user wants to know especific information about the data, the user resquest is simple and graphs are not needed to complete the user request
     [...more info...]

    3. other_request: If the user request is not related to the report, graphs, statistics data, you can use this tool to do other requests.
     [...more info...]

    4. finish: Complete the task and provide final response
        - No parameters required
        - Example:
          tool: finish
          reason: I have successfully completed all the user request.
        params: {{}}

    Respond with a YAML object

    If you believe no more actions are needed, use "finish" as the tool and explain why in the reason.
    """

    # It asks the LLM what to do
    response = call_llm(system_prompt, user_query)

    # It gets back something like:
    # tool: edit_file
    # reason: User wants a visualization, so I need to create a chart
    # params: target_file: DynamicWorkspace.tsx, instructions: "create pie chart"

4. Action Classes: Specialized Tools for Every Task 🎯

EditFileAction - the UI builder

This is where the magic happens. When you ask for a chart, this thing:

Loads your real invoice data - it reads the JSON files you've uploaded
Figures out what charts to make - it analyzes your data and decides on the best visualizations
Generates React code - it actually writes the React component with Recharts
Updates your UI - it replaces the file and your dashboard updates instantly

def execute(self, params: Dict[str, Any], working_dir: str = "", 
            execution_id: str = None) -> Dict[str, Any]:
    # Load real invoice data
    invoice_data = load_invoice_data()

    # Tell the LLM to create a React component
    system_prompt = f"""
    Create a COMPLETE professional business report React component.

    MANDATORY REQUIREMENTS:
    - Use ONLY Recharts library for ALL charts
    - Extract REAL values from the provided invoice JSON data
    - Calculate actual metrics from the invoice data
    - NO sample/fake data whatsoever
    """

    # The LLM generates the React code
    response = call_llm(system_prompt, user_prompt)

    # We replace the entire file with the new component
    success, message = overwrite_entire_file(full_path, new_react_code)

SimpleReportAction - the data analyst

For simple questions like "What's my total revenue?", it just analyzes the data and gives you a straight answer:

def execute(self, params: Dict[str, Any], working_dir: str = "", 
            execution_id: str = None) -> Dict[str, Any]:
    # Load the invoice data
    invoice_data = load_invoice_data()

    # Ask the LLM to analyze it
    system_prompt = f"""
    Answer the user's request based on the provided data.

    GUIDELINES:
    - For math operations, be super accurate and precise
    - Focus on providing specific information requested
    - Use only the real data processed
    """

    response = call_llm(system_prompt, user_prompt)

    return {
        "success": True,
        "response": response,
        "request_type": "simple_report"
    }

OtherRequestAction - the polite redirector

If you ask it something completely unrelated like "Can you write me a poem?", it politely redirects you back to what it's good at:

def execute(self, params: Dict[str, Any], working_dir: str = "", 
            execution_id: str = None) -> Dict[str, Any]:
    system_prompt = f"""
    The user has made a request that's not related to reports or data.

    Please respond politely and explain that you specialize in:
    - Creating reports and data visualizations
    - Analyzing invoice and financial data
    - Generating charts and graphs with business insights
    """

    response = call_llm(system_prompt, user_request)

    return {
        "success": True,
        "response": response,
        "request_type": "other_request"
    }

[...Other Tools...]

Want to dive deep into the tools/actions and prompts? Check out the full open-source code at: https://github.com/Handit-AI/invoice-copilot

5. The self-improvement (Best Part)

Here's the really cool thing - this AI agent actually gets better over time. Here is the secret weapon Handit.ai

Every action, every response is fully observed and analyzed. The system can see:

Which decisions worked well
Which ones failed
How long things take
What users actually want
When the LLM makes mistakes
And more...

And yes sir! When this powerful tool detects any mistakes it fixes automatically.

This means the AI agent can actually improve itself. If the LLM calculates the wrong sum or generates incorrect answers, Handit.ai tracks that failure and automatically adjusts the AI agent to prevent the same mistake from happening again. It's like having an AI engineer who is constantly monitoring, evaluating and improving your AI agent.

To get self-improvement we need to accomplish these steps:

1. Let's setup Handit.ai observability

This will give us full tracing to see inside our llm's and tools to understand what they’re doing.

Notice this project comes configured with Handit.ai Observability, you'll only need to get your own API token. Follow these steps:

1. Create an account here: Handit.ai

2. After creating your account, get your token here: Handit.ai token

3. Copy your token and add it to your .env file:

HANDIT_API_KEY=your_handit_token_here

Once you have accomplish this step, every time you interact with the conversational agent, you will get full observability in the Handit.ai Tracing Dashboard

Like this:

2. Set up evaluations

1. Add your AI provider (OpenAI, GoogleAI, etc.) token here: Token for evaluation

2. Assign evaluators to llm's nodes here: (Handit.ai Evaluation) [https://dashboard.handit.ai/evaluation-hub]

For this project specially assign Correctness Evaluation to simple_report_action, this is important because this will evaluate the accuracy of the text generation of our llm node (simple_report_action).

Like this:

Of course you can assign different evaluators to more llm nodes

3. Set up self-improvement (very interesting part)

1. Run this on your terminal:

npm install -g @handit.ai/cli

2. Run this command and follow the terminal instructions - this connects your repository to Handit for automatic PR creation!:

handit-cli github

✨ What happens next: Every time Handit detects that your AI agent failed, it will automatically send you a PR to your repo with the fixes!

This is like having an AI engineer who never sleeps, constantly monitoring your agent and fixing issues before you even notice them! 🤖👨‍

6. Results

To test the project, first you need to drag and drop some invoices (jpg, png, pdf, jpeg) in the chat interface

First input: create a report based on my invoices, with some charts

Result:

Second input: what is the total sum of all my invoices?

Result:

Whoa! Seems like the LLM made a mistake — the sum of $634.73 + $440.00 + $685.35 + $248.98 is not $1,998.06

Let’s check Observability to figure out what’s going on:

That's right! Handit has detected that the llm made a mistake: The total sum reported in the output is incorrect; it does not accurately reflect the sum of the individual invoices provided. Additionally, the breakdown of the invoice amounts contains errors in the total sum calculation.

But now what? 🤔

What happens when your AI agent is in production and fails silently? You could lose users without even knowing why - that's where the real magic happens.

Handit will automatically fix your AI agent and send you a PR to your repo! ✅

Like this example:

It automatically improves your AI agent, fixing issues so they never happen again:

Now you have an AI agent with self-improvement capabilities.

7. Conclusions

Thanks for reading!

I hope this deep dive into building a self-improving AI agent has been useful for your own projects.

Full code open source at: https://github.com/Handit-AI/invoice-copilot.

This project comes with Handit.ai configured. If you want to configure Handit.ai for your own projects, I suggest following the documentation: https://docs.handit.ai/quickstart

What new feature should have this project? Let me know in the comments! 💬

DEV Community: Oliver S

From $USD to €EUR to MXN$: A Self-Improving AI Agent That Speaks All Currencies

Project: Multi-Currency Invoice

What this AI agent actually does?

Table of Contents

1. It gets smarter the more you use it (Best Part)

Before Handit

After Handit

2. Architecture Overview

3. Stack

Tech Stack

External APIs

4. Conclusions

Build a self-improving AI agent that turns documents into structured data (with LangGraph)

Project: Unstructured to structured

What this AI agent actually does?

Table of Contents

1. The self-improvement (Best Part)

Before Handit

After Handit

2. Architecture Overview

Another Perspective of the Workflow 🧠

3. Inference Schema Node - the schema detective

4. Document Data Capture Node - the data extractor

5. Generate CSV Node - the table builder

6. Conclusions

How to build a self-improving agent that updates your UI in real time

What my AI agent actually does (and why it's pretty cool)

The problem I was trying to solve

Table of Contents

1. Architecture Overview

2. Setting Up Your Environment

1. Clone the Repository

2. Create Virtual Environment

3. Install Dependencies

4. Environment Configuration

5. Configure API Keys

6. Run the Application 🚀

1. Clone the Repository

2. Install Dependencies

3. Environment Configuration

4. Run the Application 🚀

3. The Core: MainDecisionAgent 🧠 (the boss)

4. Action Classes: Specialized Tools for Every Task 🎯

EditFileAction - the UI builder

SimpleReportAction - the data analyst

OtherRequestAction - the polite redirector

5. The self-improvement (Best Part)

1. Let's setup Handit.ai observability

2. Set up evaluations

3. Set up self-improvement (very interesting part)

6. Results

7. Conclusions