DEV Community: Chris Yaowen Zhang

The Limits of Spec-Driven Development in AI coding

Chris Yaowen Zhang — Tue, 02 Dec 2025 21:07:04 +0000

In the 1990s, developers wrote long functional specifications before coding. By 2010, agile replaced the idea that you should define everything upfront. Today, as AI coding struggle with quality, the old playbook is returning: writing detailed specs in hopes of getting reliable outcomes.

On paper, spec-driven development (SDD) feels like the perfect solution: write a detailed spec first, then let the model generate “correct” code from it.

But reality hits hard.

Just like the pattern we have seen before: when we try to “solve unpredictability” by writing more things down upfront, the development fails, and always for the same reason — Reality changes faster than specs do.

What Is Spec-Driven Development?

Spec-driven development (SDD) is the practice of writing detailed upfront specifications first, and then using AI to generate code from them. These specs aim to define a system’s behavior, requirements, constraints, and interfaces precise enough for an AI model to produce code reliably.

But it overlooks the fact that static artifacts can't contain all the context regardless how precise your specs are.

Let’s break this down.

Where Spec-Driven Development Fails

SDD are failing for four reasons that no amount of prompting or AI models have fixed yet:

1. Specs Are Expensive to Maintain
Writing comprehensive specs takes a significant amount of time. In addition, software development is an interactive process. With so many variables in play (requirements changing, constraints shifting, and new insights emerging during implementation), keeping specs in sync with the code creates a maintenance tax that grows with system complexity. Instead of reducing overhead, SDD often doubles it.

Suppose you’re building a subscription invoices system. You write a spec describing billing cycles, proration rules, tax conditions, and grace periods. But a week later, finance says, “We need European VAT handling”.

Updating the code is much easier than updating the spec first. But this leads to a situation where the code, the spec, and the team’s mental model no longer match.

As a result, every update becomes documentation debt disguised as engineering discipline.

2. Specs Don't Reflect All Context
Specs are used to describe what a system should do, but they can't explain why it works that way. And the “why” carries the real context:

Why certain assumptions were made
Why specific tradeoffs were chosen
What the team learned while iterating

What real-world constraints shaped the solution. But these things never make it into the spec. And the missing context is where the real problems show up:

Edge cases only appear when the system is used.
Performance issues only appear under load.
User behavior only appears after launch.

So LLMs don’t struggle because the spec is “wrong.” They struggle because the spec can never capture all the context they need.

3. Over-specification creates the illusion of completeness
A detailed spec feels like control. It gives teams a sense that all cases are covered. But this confidence is often false.

Software development is exploratory. The most important insights come after you begin building. Being too fixed to a static spec leads to less iteration, creativity, and emergent solutions. It makes development into a brittle, waterfall-like process, just with AI in the loop.

4. The wrong level of abstraction
SDD tools today are optimized for parsing specs, not interpreting intent.

Most SDD approaches focus on implementation detail - The hows:

Field definitions
Enums
Request/response schemas
Function signatures

But what matters more is the whys behind:

Intent
Constraints
Context

Most current SDD tools (including systems like Kiro) generate code directly from these low-level specs. They can produce accurate scaffolding, but missing context for resilient behavior. The result is code that is structurally correct but misaligned with the actual intent of the system.

What Actually Matters — Context Engineering

The missing piece in AI coding isn't more detailed specs, but better preserved context. This means AI-native development should:

1. Start with intent
Instead of jumping into writing specs, the workflow should begin by defining the core context. For instance, the problem you’re solving and why, the non-negotiable constraints, and the assumptions you have in the context.

2. Keep context up to date
AI-led development should be just as iterative as traditional software development. When requirements change or new insights come up, the context the model uses needs to be refreshed so the team, and the AI, stays aligned.

3. Specs should follow the codebase
Specs should be living artifacts and aligned with the actual implementation.

4. Preserve the whys, and not just requirements
Code shouldn’t just be about what it does, but also explain why it was built that way.

The Path Forward

For stable contracts and well-understood domains, spec-driven approaches can work great. But for exploratory development that comes with evolving requirements, context-driven approaches adapt better.

Most real-world projects have both: stable contracts at system boundaries, adaptive iteration within them. This is the principle that shaped Yansu, our SDLC-driven AI coding platform originally built for internal use to serve PE firms and mid-market engineering teams. The philosophy translated as a dynamic software development lifecycle (SDLC) in Yansu that:

Captures intent and constraints from discussions, examples, and tribal knowledge
Updates context and specs as understanding evolves
Simulates scenarios that reflect real system behavior before writing any code
Embeds explicitly the "whys" in the code, so the team can trace back to the reason behind each line

Click here if you’d like to explore our work.

Announcing Enthusiast 1.4: AI Agents Meet E-commerce Workflows

Chris Yaowen Zhang — Thu, 23 Oct 2025 15:09:04 +0000

Introducing Enthusiast v1.4

We tend to think of AI as operating on clean, structured data. E-commerce teams know better: most of their information comes as PDFs from vendors, spreadsheets with inconsistent formatting, or scanned purchase notes from clients. Embedding these unstructured sources into automated workflows has long been a challenge. With the latest release of Enthusiast 1.4, we’re closing that gap.

Enthusiast is our open-source, agentic AI toolkit for e-commerce — enabling engineering teams to build custom AI agents grounded in product data and workflows, with full control over infrastructure, integrations, and models.

In version 1.4, agents can now interact directly with uploaded files — from invoices and product manuals to catalogs and datasets — enabling richer, more contextual automation across e-commerce systems.

This release also introduces two new pre-built agents that utilize the new file capabilities in tangible e-commerce scenarios.

New in Enthusiast v1.4

1. File Uploads in Agent Conversations
Enthusiast v1.4 introduces file upload support, allowing agents to receive and process files directly within a conversation. Supported file types include images, PDFs and CSVs. Developers can also build their own extensions to make other file types accessible to the AI.

Uploaded content becomes part of the agent’s reasoning context. Agents can parse, extract, and cross-reference information across multiple files — without leaving the chat interface or building custom ingestion scripts.

This feature enables engineering teams to build workflows that require contextual interaction with files.

2. New Pre-Built Agent: Product Catalog Enrichment
Building on the new file upload capability, Enthusiast 1.4 introduces a pre-built agent for catalog enrichment — designed to extract structured product data from unstructured sources.

Many e-commerce teams know the problem well: receiving product sheets from multiple vendors, each with different layouts and inconsistent structure. Transforming this data into a customer-friendly catalog is a time-consuming manual process, prone to error.

The Catalog Enrichment Agent automates that process. You define which product attributes your catalog should contain — for example, name, SKU, dimensions, material, or price. Then, you upload supplier PDFs, spreadsheets, or photos. The agent processes each file, extracts the relevant information, and maps it to your catalog structure automatically.

This allows engineering teams to standardize data onboarding across vendors without writing custom parsers or cleaning scripts.

Example: Upload a vendor’s 200-page product catalog PDF. The agent identifies each product, extracts all defined attributes, and returns a structured, enriched dataset ready to merge with your existing system.

3. New Pre-Built Agent: Purchase Order OCR
Also built on the new file upload functionality, Enthusiast 1.4 introduces an agent for automated order creation — designed to translate unstructured order inputs into structured draft orders.

In some industries, order processing still relies on manual workflows. Customers send purchase requests as paper letters, handwritten lists, or email attachments in inconsistent formats. Each one requires manual interpretation and data entry before it reaches the Order Management System.

The Order Creation Agent automates this process end-to-end. You forward or upload the scan, photo, or document then the agent reads the content, identifies product names or SKUs, matches them to your existing catalog, and builds an order in your e-commerce system (such as Medusa.js).

This allows teams to integrate legacy order channels into automated workflows. Unlike traditional OCR tools, using an AI Agent that also internalizes the full product catalog, makes it possible to automatically process a wide variety of inputs with a high confidence rate.

Example: A wholesale client sends a photo of a handwritten order list. The agent parses each item, finds the corresponding SKUs in your Medusa catalog, and generates a ready-to-review draft order.

Native Medusa Integration
Enthusiast v1.4 ships with a native Medusa connector, so your agents can:

→ Sync and index catalogs directly from Medusa
→ Access and enrich product data using AI-driven workflows
→ Automate customer-facing or back-office operations

This connector makes it possible for Medusa developers to easily extend existing e-commerce logic with AI capabilities instantly, using the same familiar infrastructure and conventions.

Built for Engineering Workflows
As with every Enthusiast release, v1.4 prioritizes transparency, flexibility, and full control across the stack:

→ Run locally with Docker, deploy to the cloud with Kubernetes, or host on your own servers.
→ Model-agnostic architecture - works with OpenAI, Gemini, Mistral, or Ollama
→ E-commerce connectors for Medusa, Shopify, Shopware, and Solidus
→ LangSmith integration for execution tracing and debugging

You stay in control of your data, deployment, and models – while leveraging an open, extensible AI stack.

Get Started

Deploy your first file-aware, workflow-connected agent in minutes.

GitHub: upsidelab/enthusiast
Docs: upsidelab.io/tools/enthusiast

Contributions and ideas are always welcome. Let’s keep building open, extensible AI systems solving e-commerce challenges!

How to use GPT for natural language querying

Chris Yaowen Zhang — Thu, 04 Sep 2025 03:35:56 +0000

As developers, we spend a lot of time designing ways for users to provide structured input: forms, dropdowns, validation rules, error handling. It works, but it also creates friction. Users don’t think in schemas, they think in plain language.

That’s where GPT comes in. Instead of forcing users into rigid formats, we can let them type what they want in natural language and then parse it directly into clean, structured data.

In this article you will learn how to use GPT for data processing and allowing users to provide natural input to an application. Let’s take a look into some interesting use cases.

It turns out that GPT does a pretty good job of parsing natural language input into data structures defined by a schema. We can use it for transforming manuals into a JSON graph, that can be then used for, e.g. presenting step-by-step instructions or visualizing dependencies.

Example: Let’s cook!

Example: Let’s cook!
We can use GPT to transform a recipe into a graph, defined by a JSON structure. It can then be used to, for example, present step-by-step instructions or visualizing dependencies between them.

The Upside team found a pretty good spaghetti recipe available here.
We can try transforming it into a JSON.

First, let’s define a JSON schema of the JSON that we’d like to get from GPT:

The schema can look like this:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://upsidelab.io/recipe.schema.json",
  "title": "Recipe",
  "description": "A recipe definition",
  "type": "object",
  "properties": {
    "ingredients": {
      "description": "The list of ingredients required to prepare the recipe",
      "type": "array",
      "items": {
        "$ref": "#Ingredient"
      }
    },
    "steps": {
      "description": "The list of steps required to prepare the recipe",
      "type": "array",
      "items": {
        "$ref": "#Step"
      }
    }
  },
  "$defs": {
    "ingredient": {
      "$anchor": "Ingredient",
      "type": "object",
      "properties": {
        "id": { "type": "number" },
        "name": { "type": "string" },
        "quantity": { "type": "string" }
      }
    },
    "step": {
      "$anchor": "Step",
      "type": "object",
      "properties": {
        "id": { "type": "number" },
        "name": { "type": "string" },
        "description": { "type": "string" },
        "dependsOnSteps": {
          "type": "array",
          "items": { "type": "number" }
        },
        "dependsOnIngredients": {
          "type": "array",
          "items": { "type": "number" }
        }
      }
    }
  }
}

Then a prompt for GPT. We’ll split it into three parts:

Schema definition
The recipe to be parsed
A summary of what we expect it to return (just the JSON, conforming to the schema)

Summarize the following recipe into a json that conforms to the following json schema delimited by three backticks

\`
${schemaDefinition}
\`
Use the following recipe, which is delimited by three backticks
\`
${recipe}
\`
Respond with the JSON only. Your output should be parsable by a JSON parser and should conform to the JSON schema definied before.

To get stable results, you may need to tweak the prompt. We’ve noticed that more complex domains GPT will often require a more complex prompt that more explicitly specifies our requirements. For a spaghetti recipe it’s enough though.
Sample output:

{
    "ingredients": [
        { "id": 1, "name": "olive oil", "quantity": "1 tbsp" },
        { "id": 2, "name": "smoked streaky bacon", "quantity": "4 rashers" },
        { "id": 3, "name": "medium onions", "quantity": "2" },
        { "id": 4, "name": "carrots", "quantity": "2" },
        { "id": 5, "name": "celery sticks", "quantity": "2" },
        { "id": 6, "name": "garlic cloves", "quantity": "2" },
        { "id": 7, "name": "rosemary sprigs", "quantity": "2-3" },
        { "id": 8, "name": "beef mince", "quantity": "500g" },
        { "id": 9, "name": "plum tomatoes", "quantity": "2 x 400g tins" },
        { "id": 10, "name": "basil", "quantity": "small pack" },
        { "id": 11, "name": "dried oregano", "quantity": "1 tsp" },
        { "id": 12, "name": "bay leaves", "quantity": "2" },
        { "id": 13, "name": "tomato purée", "quantity": "2 tbsp" },
        { "id": 14, "name": "beef stock cube", "quantity": "1" },
        { "id": 15, "name": "red chilli", "quantity": "1 (optional)" },
        { "id": 16, "name": "red wine", "quantity": "125ml" },
        { "id": 17, "name": "cherry tomatoes", "quantity": "6" },
        { "id": 18, "name": "parmesan", "quantity": "75g" },
        { "id": 19, "name": "spaghetti", "quantity": "400g" },
        { "id": 20, "name": "crusty bread", "quantity": "to serve (optional)" }
    ],
    "steps": [
        { 
            "id": 1, 
            "name": "Prepare the saucepan", 
            "description": "Put a large saucepan on a medium heat and add 1 tbsp olive oil.", 
            "dependsOnSteps": [],
            "dependsOnIngredients": [1] 
        },
        { 
            "id": 2, 
            "name": "Cook bacon", 
            "description": "Add 4 finely chopped bacon rashers and fry for 10 mins until golden and crisp.", 
            "dependsOnSteps": [1],
            "dependsOnIngredients": [2] 
        },
        { 
            "id": 3, 
            "name": "Add vegetables and herbs", 
            "description": "Reduce the heat and add the 2 onions, 2 carrots, 2 celery sticks, 2 garlic cloves and the leaves from 2-3 sprigs rosemary, all finely chopped, then fry for 10 mins.", 
            "dependsOnSteps": [2],
            "dependsOnIngredients": [3, 4, 5, 6, 7] 
        },
        { 
            "id": 4, 
            "name": "Brown the mince", 
            "description": "Increase the heat to medium-high, add 500g beef mince and cook stirring for 3-4 mins until the meat is browned all over.", 
            "dependsOnSteps": [3],
            "dependsOnIngredients": [8] 
        },
        { 
            "id": 5, 
            "name": "Add bolognese sauce ingredients", 
            "description": "Add 2 tins plum tomatoes, the finely chopped leaves from ¾ small pack basil, 1 tsp dried oregano, 2 bay leaves, 2 tbsp tomato purée, 1 beef stock cube, 1 deseeded and finely chopped red chilli (if using), 125ml red wine and 6 halved cherry tomatoes. Stir well.", 
            "dependsOnSteps": [4],
            "dependsOnIngredients": [9, 10, 11, 12, 13, 14, 15, 16, 17] 
        },
        { 
            "id": 6, 
            "name": "Simmer the sauce", 
            "description": "Bring to the boil, reduce to a gentle simmer and cover with a lid. Cook for 1 hr 15 mins stirring occasionally, until you have a rich, thick sauce.", 
            "dependsOnSteps": [5],
            "dependsOnIngredients": [] 
        },
        { 
            "id": 7, 
            "name": "Add parmesan and season", 
            "description": "Add the 75g grated parmesan, check the seasoning and stir.", 
            "dependsOnSteps": [6],
            "dependsOnIngredients": [18] 
        },
        { 
            "id": 8, 
            "name": "Cook spaghetti", 
            "description": "When the bolognese is nearly finished, cook 400g spaghetti following the pack instructions.", 
            "dependsOnSteps": [7],
            "dependsOnIngredients": [19] 
        },
        { 
            "id": 9, 
            "name": "Serve", 
            "description": "Drain the spaghetti and either stir into the bolognese sauce, or serve the sauce on top. Serve with more grated parmesan, the remaining basil leaves and crusty bread, if you like.", 
            "dependsOnSteps": [8],
            "dependsOnIngredients": [10, 18, 20] 
        }
    ]
}

This is something we can easily display on a graph:

If we aren’t satisfied with the outcome, we can also use a follow-up prompt to mutate the structure. If you look at the recipe, step 8 (cook spaghetti) is independent of the previous steps.

Let’s ask GPT to change that:

Step 8 doesn't depend on any of the previous steps and can be done in parallel. Change the resulting JSON to reflect that
{
    "ingredients": [
        { "id": 1, "name": "olive oil", "quantity": "1 tbsp" },
        { "id": 2, "name": "smoked streaky bacon", "quantity": "4 rashers" },
        { "id": 3, "name": "medium onions", "quantity": "2" },
        { "id": 4, "name": "carrots", "quantity": "2" },
        { "id": 5, "name": "celery sticks", "quantity": "2" },
        { "id": 6, "name": "garlic cloves", "quantity": "2" },
        { "id": 7, "name": "rosemary sprigs", "quantity": "2-3" },
        { "id": 8, "name": "beef mince", "quantity": "500g" },
        { "id": 9, "name": "plum tomatoes", "quantity": "2 x 400g tins" },
        { "id": 10, "name": "basil", "quantity": "small pack" },
        { "id": 11, "name": "dried oregano", "quantity": "1 tsp" },
        { "id": 12, "name": "bay leaves", "quantity": "2" },
        { "id": 13, "name": "tomato purée", "quantity": "2 tbsp" },
        { "id": 14, "name": "beef stock cube", "quantity": "1" },
        { "id": 15, "name": "red chilli", "quantity": "1 (optional)" },
        { "id": 16, "name": "red wine", "quantity": "125ml" },
        { "id": 17, "name": "cherry tomatoes", "quantity": "6" },
        { "id": 18, "name": "parmesan", "quantity": "75g" },
        { "id": 19, "name": "spaghetti", "quantity": "400g" },
        { "id": 20, "name": "crusty bread", "quantity": "to serve (optional)" }
    ],
    "steps": [
        { 
            "id": 1, 
            "name": "Prepare the saucepan", 
            "description": "Put a large saucepan on a medium heat and add 1 tbsp olive oil.", 
            "dependsOnSteps": [],
            "dependsOnIngredients": [1] 
        },
        { 
            "id": 2, 
            "name": "Cook bacon", 
            "description": "Add 4 finely chopped bacon rashers and fry for 10 mins until golden and crisp.", 
            "dependsOnSteps": [1],
            "dependsOnIngredients": [2] 
        },
        { 
            "id": 3, 
            "name": "Add vegetables and herbs", 
            "description": "Reduce the heat and add the 2 onions, 2 carrots, 2 celery sticks, 2 garlic cloves and the leaves from 2-3 sprigs rosemary, all finely chopped, then fry for 10 mins.", 
            "dependsOnSteps": [2],
            "dependsOnIngredients": [3, 4, 5, 6, 7] 
        },
        { 
            "id": 4, 
            "name": "Brown the mince", 
            "description": "Increase the heat to medium-high, add 500g beef mince and cook stirring for 3-4 mins until the meat is browned all over.", 
            "dependsOnSteps": [3],
            "dependsOnIngredients": [8] 
        },
        { 
            "id": 5, 
            "name": "Add bolognese sauce ingredients", 
            "description": "Add 2 tins plum tomatoes, the finely chopped leaves from ¾ small pack basil, 1 tsp dried oregano, 2 bay leaves, 2 tbsp tomato purée, 1 beef stock cube, 1 deseeded and finely chopped red chilli (if using), 125ml red wine and 6 halved cherry tomatoes. Stir well.", 
            "dependsOnSteps": [4],
            "dependsOnIngredients": [9, 10, 11, 12, 13, 14, 15, 16, 17] 
        },
        { 
            "id": 6, 
            "name": "Simmer the sauce", 
            "description": "Bring to the boil, reduce to a gentle simmer and cover with a lid. Cook for 1 hr 15 mins stirring occasionally, until you have a rich, thick sauce.", 
            "dependsOnSteps": [5],
            "dependsOnIngredients": [] 
        },
        { 
            "id": 7, 
            "name": "Add parmesan and season", 
            "description": "Add the 75g grated parmesan, check the seasoning and stir.", 
            "dependsOnSteps": [6],
            "dependsOnIngredients": [18] 
        },
        { 
            "id": 8, 
            "name": "Cook spaghetti", 
            "description": "When the bolognese is nearly finished, cook 400g spaghetti following the pack instructions.", 
            "dependsOnSteps": [],
            "dependsOnIngredients": [19] 
        },
        { 
            "id": 9, 
            "name": "Serve", 
            "description": "Drain the spaghetti and either stir into the bolognese sauce, or serve the sauce on top. Serve with more grated parmesan, the remaining basil leaves and crusty bread, if you like.", 
            "dependsOnSteps": [7, 8],
            "dependsOnIngredients": [10, 18, 20] 
        }
    ]
}

As the previous response is a part of the follow-up prompt, GPT does a pretty good job at keeping the unchanged elements stable.

Use case - build a BPMN specification of a business process

Let’s expand this further - this approach allows us to generate a BPMN definition of a business process, which can subsequently be used by a tool like Camunda.

Let’s try the following prompt:

Generate camunda xml file that describes a process delimited by three backticks 

\`
${processDescription}
\`

Respond with the XML only. The XML should contain process definition and a diagram of the process. The XML should be a valid camunda xml file and should be parsable by its modeller tool. There should be nothing except the xml in the response.

Where process description could be for example:

An invoice is received by a system, parsed by an OCR system, then reviewed by an accountant. After the initial review, it's sent to a manager for an approval. When the approval is made, it's sent to account payable. If the approval is not given by the manager, it is sent to accounting for further clarification.

After sending that to GPT, we’ll get the following output:

<?xml version="1.0" encoding="UTF-8"?>
<definitions xmlns="http://www.omg.org/spec/BPMN/20100524/MODEL"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xmlns:camunda="http://camunda.org/schema/1.0/bpmn"
             xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI"
             xmlns:dc="http://www.omg.org/spec/DD/20100524/DC"
             xmlns:di="http://www.omg.org/spec/DD/20100524/DI"
             xsi:schemaLocation="http://www.omg.org/spec/BPMN/20100524/MODEL BPMN20.xsd"
             id="Definitions_1"
             targetNamespace="http://bpmn.io/schema/bpmn">

  <process id="invoiceProcess" name="Invoice Processing" isExecutable="true">
    <startEvent id="startEvent" name="Invoice Received">
      <outgoing>Flow_1</outgoing>
    </startEvent>

    <sequenceFlow id="Flow_1" sourceRef="startEvent" targetRef="parseInvoice" />

    <serviceTask id="parseInvoice" name="Parse Invoice with OCR" camunda:type="external" camunda:topic="parseInvoice">
      <incoming>Flow_1</incoming>
      <outgoing>Flow_2</outgoing>
    </serviceTask>

    <sequenceFlow id="Flow_2" sourceRef="parseInvoice" targetRef="reviewByAccountant" />

    <userTask id="reviewByAccountant" name="Review by Accountant">
      <incoming>Flow_2</incoming>
      <outgoing>Flow_3</outgoing>
    </userTask>

    <sequenceFlow id="Flow_3" sourceRef="reviewByAccountant" targetRef="managerApproval" />

    <exclusiveGateway id="managerApproval" name="Manager Approval">
      <incoming>Flow_3</incoming>
      <outgoing>Flow_4</outgoing>
      <outgoing>Flow_5</outgoing>
    </exclusiveGateway>

    <sequenceFlow id="Flow_4" sourceRef="managerApproval" targetRef="sendToAccountPayable" name="Approved">
      <conditionExpression xsi:type="tFormalExpression">\${approved}</conditionExpression>
    </sequenceFlow>

    <sequenceFlow id="Flow_5" sourceRef="managerApproval" targetRef="sendToAccounting" name="Not Approved">
      <conditionExpression xsi:type="tFormalExpression">not(\${approved})</conditionExpression>
    </sequenceFlow>

    <userTask id="sendToAccountPayable" name="Send to Account Payable">
      <incoming>Flow_4</incoming>
      <outgoing>Flow_6</outgoing>
    </userTask>

    <userTask id="sendToAccounting" name="Send to Accounting for Clarification">
      <incoming>Flow_5</incoming>
      <outgoing>Flow_7</outgoing>
    </userTask>

    <endEvent id="endEvent" name="Process End">
      <incoming>Flow_6</incoming>
      <incoming>Flow_7</incoming>
    </endEvent>

    <sequenceFlow id="Flow_6" sourceRef="sendToAccountPayable" targetRef="endEvent" />
    <sequenceFlow id="Flow_7" sourceRef="sendToAccounting" targetRef="endEvent" />

  </process>

  <bpmndi:BPMNDiagram id="BPMNDiagram_1">
    <bpmndi:BPMNPlane id="BPMNPlane_1" bpmnElement="invoiceProcess">
      <bpmndi:BPMNShape id="StartEvent_1_di" bpmnElement="startEvent">
        <dc:Bounds x="173" y="102" width="36" height="36" />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNEdge id="Flow_1_di" bpmnElement="Flow_1">
        <di:waypoint x="209" y="120" />
        <di:waypoint x="261" y="120" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNShape id="ServiceTask_1_di" bpmnElement="parseInvoice">
        <dc:Bounds x="261" y="80" width="100" height="80" />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNEdge id="Flow_2_di" bpmnElement="Flow_2">
        <di:waypoint x="361" y="120" />
        <di:waypoint x="411" y="120" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNShape id="UserTask_1_di" bpmnElement="reviewByAccountant">
        <dc:Bounds x="411" y="80" width="100" height="80" />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNEdge id="Flow_3_di" bpmnElement="Flow_3">
        <di:waypoint x="511" y="120" />
        <di:waypoint x="561" y="120" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNShape id="ExclusiveGateway_1_di" bpmnElement="managerApproval" isMarkerVisible="true">
        <dc:Bounds x="561" y="95" width="50" height="50" />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNEdge id="Flow_4_di" bpmnElement="Flow_4">
        <di:waypoint x="611" y="120" />
        <di:waypoint x="661" y="120" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNEdge id="Flow_5_di" bpmnElement="Flow_5">
        <di:waypoint x="586" y="145" />
        <di:waypoint x="586" y="195" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNShape id="UserTask_2_di" bpmnElement="sendToAccountPayable">
        <dc:Bounds x="661" y="80" width="100" height="80" />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNShape id="UserTask_3_di" bpmnElement="sendToAccounting">
        <dc:Bounds x="536" y="195" width="100" height="80" />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNEdge id="Flow_6_di" bpmnElement="Flow_6">
        <di:waypoint x="761" y="120" />
        <di:waypoint x="811" y="120" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNEdge id="Flow_7_di" bpmnElement="Flow_7">
        <di:waypoint x="586" y="275" />
        <di:waypoint x="586" y="325" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNShape id="EndEvent_1_di" bpmnElement="endEvent">
        <dc:Bounds x="811" y="102" width="36" height="36" />
      </bpmndi:BPMNShape>
    </bpmndi:BPMNPlane>
  </bpmndi:BPMNDiagram>
</definitions>

Looking good!

Remarkable insights:

GPT seems to omit some of the XMLNS references quite often. This can be easily fixed in post-processing, which may be way more cost-effective than regenerating the response.
I once got a response where the diagram definition looked like the following:

<bpmndi:BPMNPlane id="BPMNPlane_1" bpmnElement="invoiceProcess">
    <!-- Diagram layout omitted for brevity -->
</bpmndi:BPMNPlane>

Even though it’s quite amusing, this is an edge case that we’d likely need to support if aiming for production use. The most common approaches are to add more validations and retries to our application’s code, improve the prompt to direct GPT, or try different temperatures of the model.

Use case - build dynamic forms in React

Another common use case that we can try to automate is building custom forms dynamically. Many applications, such as surveys, custom forms etc., rely on this feature to gain feedback from its candidates, customers or potential leads.

GPT makes it incredibly easy to build a tool that allows a regular user to create such a form by providing natural language input. Let’s use a library called react-jsonschema-form. It’s a React-based library that generates forms based on a JSON definition. It also contains an extensive schema definition that such JSONs can be validated against.

We can leverage the fact that GPT was trained on a large dataset that also includes definitions of commonly used JSON schemas. That means we don’t have to provide the exact schema, and we can refer to it in our prompt by name.

Let’s create a prompt that:

Specifies that we would like to use the schema of react-jsonschema-form
Include the user's definition of the ideal form
Will specify some additional criteria for the output (i.e. make sure that we only get a JSON as a response)

Generate a JSON definition that follows the schema of react-jsonschema-form. Use the following form that is delimited by three backticks

\`
${userInput}
\`

Respond with the JSON only and don't include any additional characters. Make sure that the output is a valid JSON and that it passes validation with the @rjsf/validator-ajv8 validator

Where userInput could be:

The form should allow a customer to provide an order number (starting with letter R followed by 9 digits), customer's phone number (limited only to country code and digits) and a customer support request. There should also be a field for selecting the type of the issue from a list of the following options: order missing, change in order, return request. At the end of the form, there should be a checkbox labeled "I agree to the privacy policy of company X"

In this case GPT gets back to us with

{
  "title": "Customer Support Form",
  "type": "object",
  "required": [
    "orderNumber",
    "phoneNumber",
    "supportRequest",
    "issueType",
    "privacyPolicyAgreement"
  ],
  "properties": {
    "orderNumber": {
      "type": "string",
      "title": "Order Number",
      "pattern": "R[0-9]{9}"
    },
    "phoneNumber": {
      "type": "string",
      "title": "Phone Number",
      "pattern": "\\+[0-9]+"
    },
    "supportRequest": {
      "type": "string",
      "title": "Customer Support Request"
    },
    "issueType": {
      "type": "string",
      "title": "Type of Issue",
      "enum": [
        "order missing",
        "change in order",
        "return request"
      ]
    },
    "privacyPolicyAgreement": {
      "type": "boolean",
      "title": "I agree to the privacy policy of company X",
      "const": true
    }
  }
}

We can quickly verify it in react-jsonschema-form’s playground: https://rjsf-team.github.io/react-jsonschema-form/

As you can see, it correctly generated a form, together with relevant validations. As with previous examples, we can make modifications by follow-up prompts.

Let’s try making the privacy policy field selectable by default. Sending the following prompt:

The privacy policy agreement should be checked by default

Will result in a new JSON definition of the form:

{
  "title": "Customer Support Form",
  "type": "object",
  "required": ["orderNumber", "phoneNumber", "supportRequest", "issueType"],
  "properties": {
    "orderNumber": {
      "type": "string",
      "title": "Order Number",
      "pattern": "R[0-9]{9}"
    },
    "phoneNumber": {
      "type": "string",
      "title": "Phone Number",
      "pattern": "\\+[0-9]+"
    },
    "supportRequest": {
      "type": "string",
      "title": "Customer Support Request"
    },
    "issueType": {
      "type": "string",
      "title": "Type of Issue",
      "enum": ["order missing", "change in order", "return request"]
    },
    "privacyPolicyAgreement": {
      "type": "boolean",
      "title": "I agree to the privacy policy of company X",
      "default": true
    }
  }
}

Which looks like that:

As you can see, the checkbox is now selected by default. Similar to that, we can make further tweaks to the existing form.

Use case - build dynamic UIs

In the previous example, we were bound to forms, which are useful but limited in terms of what we can do. To build more general UIs dynamically, we can use a tool called react-json-schema, combine it with GPT, and build fully dynamic complex UIs.

To start with, let’s define some components. We’ll use react-bootstrap to simplify the UI work needed.

Let’s make a wrapper component for the whole app:

function AppRoot({ children }) {
  return (
    <>
      {children}
    </>
  )
}

export default AppRoot

Then a basic layout component:

import { Container } from 'react-bootstrap';

function AppMainLayout({ children }) {
  return (
    <Container>
      {children}
    </Container>
  )
}

export default AppMainLayout

A customizable navbar:

import Container from 'react-bootstrap/Container';
import Nav from 'react-bootstrap/Nav';
import Navbar from 'react-bootstrap/Navbar';

function AppNavbar({ title, links }) {
  return (
    <Navbar bg="primary" data-bs-theme="dark">
    <Container>
      <Navbar.Brand href="/">{title}</Navbar.Brand>
      <Nav className="me-auto">
        {links && links.map((link) => {
          return <Nav.Link href={link.url} key={link.title}>{link.title}</Nav.Link>
        })}
      </Nav>
    </Container>
  </Navbar>
  )
}

export default AppNavbar;

A toolbar with some action buttons (that we can wire together later):

import Button from 'react-bootstrap/Button';

function AppToolbar() {
  return (
    <div className="py-2">
      <Button variant="primary" className="mx-1">New user</Button>
      <Button variant="secondary" className="mx-1">Refresh</Button>
      <Button variant="secondary" className="mx-1">Deactivate</Button>
      <Button variant="secondary" className="mx-1">Reset password</Button>
    </div>
  )
}

export default AppToolbar

And a table for displaying a list of users:

import Table from 'react-bootstrap/Table'

function AppUserList() {
  return (
    <Table striped bordered hover>
      <thead>
        <tr>
          <th>#</th>
          <th>Email</th>
          <th>First Name</th>
          <th>Last Name</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <td>1</td>
          <td>test@example.com</td>
          <td>John</td>
          <td>Doe</td>
        </tr>
        <tr>
          <td>2</td>
          <td>jack@example.com</td>
          <td>Jack</td>
          <td>Dummy</td>
        </tr>
        <tr>
          <td>3</td>
          <td>thomas@example.com</td>
          <td>Tomas</td>
          <td>Jonas</td>
        </tr>
        <tr>
          <td>4</td>
          <td>paul@example.com</td>
          <td>Paul</td>
          <td>Test</td>
        </tr>
      </tbody>
    </Table>
  )
}

export default AppUserList
view raw

Then connect it all together in our web application:

import ReactDOM from 'react-dom/client';
import reportWebVitals from './reportWebVitals';
import ReactJsonSchema from 'react-json-schema';
import AppNavbar from './components/AppNavbar'
import AppRoot from './components/AppRoot'
import AppMainLayout from './components/AppMainLayout'
import AppToolbar from './components/AppToolbar'
import AppUserList from './components/AppUserList'

const uiRenderer = new ReactJsonSchema();
uiRenderer.setComponentMap({ AppRoot, AppNavbar, AppMainLayout, AppToolbar, AppUserList });

const uiSchema = await fetchGPTResponse(prompt);

const root = ReactDOM.createRoot(document.getElementById('root'));
root.render(
  uiRenderer.parseSchema(uiSchema)
);

It’s easy:

Initialize react-json-schema parser
Configure it to recognize our custom components
Prompt GPT for a schema based on user’s description
Render the schema that GPT responded with

Now it’s time to define the actual prompt to be used:

Generate a JSON that follows the react-json-schema format for the following set of components and their props delimited by three backticks:
\`
AppRoot (children: Node[]) , AppNavbar (title: string, links: Link[]), AppMainLayout (children: Node[]) , AppToolbar (), AppUserList (), h1 (text: string), h2 (text: string)
\`
The following custom types are defined in the application
\`
type Link = {
title: string;
url: string;
}
\`

The JSON should build a UI following a specification delimited by three backticks
\`
The UI should start with a navbar, a title specifying that this is a dashboard for managing users. Below the title there should be a more detailed description of the dashboard. The main element of the UI should be a table with a list of users. Above the table there should be a toolbar allowing to modify the list of users.
\`
The UI must be wrapped in the AppRoot component. Specify the props directly in the same object as the component, without an additional props object. Only use the props that are defined above. The output should be a valid react-json-schema specification. Don't output any additional characters.

Remarkable insights: As you can see, we started with a list of available components, followed by an additional list of props available for each of them. My tests have shown that it’s also helpful to explicitly state how to provide props (without a nested props object). Without that remark, GPT put it inside a nested object, which didn’t work with react-json-schema library. My assumption is that react-json-schema is not a very popular library and lacks extensive examples online - hence the need to point GPT in the right direction.

A sample GPT response will look like:

{
  "component": "AppRoot",
  "children": [
    {
      "component": "AppNavbar",
      "title": "User Management Dashboard",
      "links": [
        {
          "title": "Home",
          "url": "/"
        },
        {
          "title": "Settings",
          "url": "/settings"
        }
      ]
    },
    {
      "component": "AppMainLayout",
      "children": [
        {
          "component": "h1",
          "text": "User Management Dashboard"
        },
        {
          "component": "h2",
          "text": "Manage and oversee user accounts and permissions"
        },
        {
          "component": "AppToolbar",
          "children": []
        },
        {
          "component": "AppUserList",
          "children": []
        }
      ]
    }
  ]
}

PICS Which renders a quite decent dashboard! GPT was able to infer that the majority of the layout should go inside a component called AppMainLayout, but didn’t put the AppNavbar there. It also filled the navbar with some example links.

Use case - querying an ElasticSearch index

ElasticSearch is commonly used for building indexes of products or application logs. We can use GPT to translate user’s queries to an ElasticSearch query.

The approach would be:

Specify that we’d like to get an ElasticSearch query as a result
Provide information about the structure of the index
Provide information about the user's query
Specify additional criteria for the output

Example: We have an ElasticSearch index with the following definition:

{
  "products_production": {
    "aliases": {},
    "mappings":{
      "spree_product": {
        "properties": {
          "available_on": {
            "type":"date",
            "format":"dateOptionalTime"
          },
          "brand_id": {
            "type":"keyword"
          },
          "created_at": {
            "type":"date",
            "format":"dateOptionalTime"
          },
          "description": {
            "type":"text",
            "analyzer":"snowball"
          },
          "discontinue_on": {
            "type":"date",
            "format":"dateOptionalTime"
          },
          "featured": {
            "type":"boolean"
          },
          "name": {
            "type":"text",
            "boost":100.0,
            "fields": {
              "untouched": {
                "type":"keyword"
              }
            },
            "analyzer":"snowball"
          },
          "price": {
            "type":"double"
          },
          "sku": {
            "type":"keyword"
          },
          "suggestion_terms": {
            "type":"text",
            "analyzer":"simple"
          },
          "taxon_ids": {
            "type":"keyword"
          }
        }
      }
    },
    "settings": {
      "index": {
        "number_of_shards":"1",
        "provided_name":"products_production",
        "creation_date":"1684177003621",
        "analysis": {
          "filter": {
            "nGram_filter": {
              "token_chars": [
                "letter",
                "digit",
                "punctuation",
                "symbol"
              ],
              "min_gram": "3",
              "type": "nGram",
              "max_gram": "4"
            }
          },
          "analyzer": {
            "nGram_analyzer": {
              "filter":[
                "lowercase",
                "asciifolding",
                "nGram_filter"
              ],
              "type": "custom",
              "tokenizer": "whitespace"
            },
            "whitespace_analyzer": {
              "filter": [
                "lowercase",
                "asciifolding"
              ],
              "type": "custom",
              "tokenizer": "whitespace"
            }
          }
        },
        "number_of_replicas": "1",
        "uuid": "WV8HWbS_QsudGKL223hwrA",
        "version": {
          "created": "6080091"
        }
      }
    }
  }
}
Create an elasticsearch query json that operates on an index with the following definition delimited by three backticks
\`
${indexDefinition}
\`

The query should return the results described here delimited by three backticks:
\`
Return the list of the most expensive products that are currently available, aren't featured and don't have any taxons associated with them.
\`

Return only a JSON and make sure it can be interpreted by an elasticsearch 6.8 instance

It’s worth mentioning that this approach should never be used on an ElasticSearch that’s not meant to be fully accessible by the user due to the potential risk of making arbitrary queries to the cluster.
It comes back with the desired query:

{
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "available_on": {
              "lte": "now"
            }
          }
        },
        {
          "term": {
            "featured": false
          }
        },
        {
          "bool": {
            "must_not": {
              "exists": {
                "field": "taxon_ids"
              }
            }
          }
        }
      ]
    }
  },
  "sort": [
    {
      "price": {
        "order": "desc"
      }
    }
  ]
}

We can now pass it to ElasticSearch and get the results:

{
    "took": 51,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": 435,
        "max_score": null,
        "hits": [
            {
                "_index": "products_production",
                "_type": "product",
                "_id": "1694",
                "_score": null,
                "_source": {
                    "name": "Asus ROG Zephyrus S Ultra Slim Gaming Laptop, 15.6\u0026quot; 144Hz IPS Type FHD, GeForce RTX 2070, Intel Core i7-9750H, 16GB DDR4, 512GB PCIe Nvme SSD, Aura Sync RGB, Windows 10, GX531GW-AB76",
                    "description": "\u003cp\u003eNvidia GeForce RTX 2070 8GB DDR6 (base: 1080 MHz, Boost: 1305 MHz; TDP: 90W). 230W Power Adaptor. Support Type-c PD 3.0 up to 65W Latest Intel Core i7-9750h Hexa-Core processor; Bluetooth 5.0 15.6\u0026rdquo; 144Hz 3ms IPS-Type full HD (1920x1080) Display 16GB 2666Hz DDR4 | 512GB PCIe NVMe SSD | Windows 10 Home 0.62\u0026rdquo; thin, 4.6 lbs ultraportable military-grade magnesium alloy body with premium metal cover Rog active Aerodynamic System (AAs) with upgraded 12V fans and anti-dust tunnels to preserve cooling performance and system stability Gigabit wave 2 Wi-Fi 5 (802.11AC) | customizable 4-zone ASUS Aura RGB Gaming Keyboard The all new ROG Zephyrus s gaming Laptop is redefining ultra-slim gaming laptops yet again with innovative engineering to cool Its 9th Gen Intel Core i7 processor and GeForce RTX Max-Q graphics, so you can immerse yourself in its no-compromise 144Hz/3ms Display. *The actual transfer speed of USB 3.0, 3.1 (Gen 1 and 2), and/or Type-C will vary depending on many factors including the processing speed of the host device, file attributes and other factors related to system configuration and your operating environment.\u003c/p\u003e",
                    "taxon_ids": [],
                    "brand_id": null,
                    "available_on": "2019-12-02T00:00:00.000Z",
                    "discontinue_on": null,
                    "featured": false,
                    "price": "1595.73",
                    "created_at": "2019-12-02T20:07:26.594Z",
                    "suggestion_terms": ["Asus ROG Zephyrus S Ultra Slim Gaming Laptop, 15.6\u0026quot; 144Hz IPS Type FHD, GeForce RTX 2070, Intel Core i7-9750H, 16GB DDR4, 512GB PCIe Nvme SSD, Aura Sync RGB, Windows 10, GX531GW-AB76"]
                },
                "sort": [1595.73]
            },  
            {
                "_index": "products_production",
                "_type": "product",
                "_id": "1470",
                "_score": null,
                "_source": {
                    "name": "Weber - Genesis II S-435 62006001 Stainless Steel",
                    "description": "\u003cp\u003eA grill that has it all. A powerful grilling engine, impressive features, and an outstanding 10 year warranty on all parts of the grill to back it up. The immense space and sear station will execute as many steaks as you need to perfection, while the side burner saut\u0026eacute;s veggies. Premium stainless steel cooking grates retain even heat and are easy to clean. The Tuck-Away warming rack keeps food warm while the main dish grills underneath. And between cookouts, store your grilling tools and accessories in the enclosed grill cabinet. Those are just a few features that set the Genesis II S-435 apart from your typical grill.\u003c/p\u003e\r\n\u003cul\u003e\r\n\u003cli\u003ePropane Grill fueled by propane\u003c/li\u003e\r\n\u003cli\u003eElectric Ignition Electronic ignition makes it easy to start\u003c/li\u003e\r\n\u003cli\u003eSide Burner Side burner for cooking versatility.\u003c/li\u003e\r\n\u003cli\u003e9,000 BTU-per-hour input Sear Station burner\u003c/li\u003e\r\n\u003cli\u003eStainless steel heat deflectors\u003c/li\u003e\r\n\u003cli\u003eStainless steel lid with thermometer\u003c/li\u003e\r\n\u003cli\u003eEnclosed cabinet with stainless steel doors\u003c/li\u003e\r\n\u003cli\u003eSix tool hooks\u003c/li\u003e\r\n\u003cli\u003eTwo heavy-duty front locking swivel casters and two large wheels\u003c/li\u003e\r\n\u003c/ul\u003e",
                    "sku": "62006001",
                    "taxon_ids": [],
                    "brand_id": null,
                    "available_on": "2019-11-03T00:00:00.000Z",
                    "discontinue_on" :"2019-11-04T00:00:00.000Z", 
                    "featured": false,
                    "price": "1315.5",
                    "created_at": "2019-11-03T19:02:48.694Z",
                    "suggestion_terms": ["Weber - Genesis II S-435 62006001 Stainless Steel"]
                },
                "sort" :[1315.5]
            },
            // more results here
        ]
    }
}

Follow-up use case - building a query in a query language that the user may not know

This feature is useful when a user needs to query a tool with its own specific query language or with a structure that the user doesn’t know. AWS seems to be running an A/B test of a feature where you can generate a CloudWatch search query based on a natural language input.

Use case - data visualization

Interesting use case - you allow users to pull report data (by e.g. querying an ElasticSearch index or a database). This is often combined with visualizing the data. Could we allow the user to also define how they’d like the data presented? Let’s say a manager needs a pie chart for their PowerPoint presentation.

Fortunately, there’s a tool called Vega which is a visualization grammar that allows you to define graphs using a JSON schema. It also comes with Vega-Lite, which allows us to easily render these charts using JavaScript. There’s also an interactive playground that we can use for testing our diagrams.

Let’s take a sample CSV file from Vega-Lite’s demo that contains the stock prices history of a few tech companies. It will be easy for us to reference it on the playground.

The file is a regular CSV that contains a few columns:

symbol,date,price
MSFT,Jan 1 2000,39.81
MSFT,Feb 1 2000,36.35
MSFT,Mar 1 2000,43.22

Let’s try to generate some charts that utilize this structure.

Generate a JSON using vega-lite schema that takes input data from file data/stocks.csv. The visualization should work on a dataset with the following columns delimited by three backticks
\`
symbol date price
\`
The visualization should result in the following result delimited by three backticks
\`
I would like to get a line chart of each of the stocks over time. There should also be a second line chart that shows only MSFT stock. Finally, I would like to see a pie chart of the average stock prices for each symbol
\`

Return only the resulting JSON and don't include any additional characters. Make sure that the visualization conforms to the vega-lite schema.

After trying it out with GPT we get the following definition

{
  "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
  "data": {"url": "data/stocks.csv"},
  "vconcat": [
    {
      "title": "Stock Prices Over Time",
      "mark": "line",
      "encoding": {
        "x": {"field": "date", "type": "temporal", "title": "Date"},
        "y": {"field": "price", "type": "quantitative", "title": "Price"},
        "color": {"field": "symbol", "type": "nominal", "title": "Symbol"}
      }
    },
    {
      "title": "MSFT Stock Price Over Time",
      "transform": [{"filter": "datum.symbol === 'MSFT'"}],
      "mark": "line",
      "encoding": {
        "x": {"field": "date", "type": "temporal", "title": "Date"},
        "y": {"field": "price", "type": "quantitative", "title": "Price"}
      }
    },
    {
      "title": "Average Stock Prices",
      "transform": [{"aggregate": [{"op": "mean", "field": "price", "as": "average_price"}], "groupby": ["symbol"]}],
      "mark": "arc",
      "encoding": {
        "theta": {"field": "average_price", "type": "quantitative", "title": "Average Price"},
        "color": {"field": "symbol", "type": "nominal", "title": "Symbol"}
      }
    }
  ]
}

Which gives us the desired set of charts

There’s also an interesting project on Github that showcases a similar concept. Their base prompt looks slightly differently:

You are a great assistant at vega-lite visualization creation. No matter what the user asks, you should always respond with a valid vega-lite specification in JSON.

You should create the vega-lite specification based on user's query.

Besides, Here are some requirements:
1. Do not contain the key called 'data' in vega-lite specification.
2. If the user ask many times, you should generate the specification based on the previous context.
3. You should consider to aggregate the field if it is quantitative and the chart has a mark type of react, bar, line, area or arc.
4. Consider to use bin for field if it is a chart like heatmap or histogram.
5. The available fields in the dataset and their types are:
${metas
    .map((field) => `${field.name} (${field.semanticType})`)
    .join(", ")}

Good practices - learn from the Upside team's experiences

Once the request was submitted to ChatGPT, we should run a JSON schema (or XML) validator on the response it provides. Despite its reliability, it may act differently than expected. Sometimes a retry will be enough to get a result that’s consistent with the defined structure. On the user experience side, there are several approaches, but you should tailor them to your specific needs. A good example of JSON schema validation is using Vega-Lite’s JSON Schema, available here.

You can apply the same method with any JSON schema you create, like for a spaghetti recipe.

Complexity - try not to bite more than GPT can chew‍

The prompts can’t be too long, as GPT may lose information and context with complicated inputs. You can use clever prompting engineering to break a task into smaller ones that gradually reduces the cognitive load and follows a logical reasoning process. For example, instead of asking GPT to format a recipe in a specific JSON schema, at first break it down into steps “Break this recipe down into steps". Once that’s achieved - "Structure this as the following JSON schema". It’s about breaking it down into steps, where each step builds on the understanding of the previous one and aligns it with the defined structure.

Performance - being patient is the key‍

Generating outputs with GPT is quite time-consuming, especially if we need to process multiple prompts in order to achieve the desired use case. We can expect improvements when it comes to hardware, so this won’t always be an issue, but for now, we need to focus on the UX level to cover up these issues. Improving UX in such cases is challenging, and often the best solution is to stream the response in smaller parts. However, if the model needs to return a full result (e.g. a JSON schema), this approach doesn’t work.

Cost - manage your expenses

However, using e.g. GPT-4 can be quite costly at scale. I wouldn’t be too worried about that in the long term, as like any technology, it will get cheaper over time, but for now it’s good to keep an eye on how much money is spent. The most sensible approach is to perform as much preprocessing as possible before pushing input to GPT, in order to limit the cost of the actual GPT usage.

Reliability - fine-tuning for diverse inputs

Playing around with these examples makes it quite easy to get it to a point where it fairly consistently delivers a great result. However, once you launch a GPT-based feature to a wider audience that will use it with different input data, you may need plenty of fine-tuning in order to ensure the solution's reliability. You never really know what the user’s will type into the chat box once you deploy your app. As developers of the feature, we are usually biased. User testing those solutions is important, especially since using a chat interface to solve actual problems (rather than just getting answers) is still a pretty novel idea and people have different expectations.

Prompt injection risks - always double check

It goes without saying that you need to be careful about the responses it may generate // (Optional) Screenshot of DPD example (the one where a customer prompted it to write a song about a shitty delivery company) ![[dpd_chatbot_fail-1.png]]

Summary

GPT does a pretty good job in parsing data. Basing on the experiments described in the post, we can define a simple framework that can handle diverse input and output formats:

Specify the desired schema
Specify the details of available fields (when needed)
Specify the input data
Reinforce important details (e.g. that the response should be a JSON and only a JSON)

Even now, there are many applications that can utilize GPT, without being AI-first. It doesn’t need to be a whole GPT-based experience, it can supplement the existing experiences and improve user productivity.

How to Build a Self-Correcting AI Agent for Product Search in E-Commerce

Chris Yaowen Zhang — Thu, 21 Aug 2025 20:26:11 +0000

Shopify just launched AI agents that let shoppers search, explore, and purchase using natural language.

If you’ve tried retrieval-augmented generation (RAG) pipelines for product search, you’ve probably hit the usual walls: vague results, brittle prompts, and silent failures when the data isn’t structured just right. When your catalog involves complex product descriptions, categorizations and multiple supporting documents, a basic retrieval or prompt-based approach just doesn’t cut it.

In the age of agentic commerce, how can we enable users to say things like “I have a small family of four. We live in Munich. What’s the best internet plan for us?” and have the system identify relevant products, draft an initial proposal, review and refine it based on available data, and engage in a meaningful conversation?

In this post, you’ll learn how to build a practical AI agent for searching product catalogs using Enthusiast, an AI toolkit designed for e-commerce and knowledge-intensive tasks. We will cover setting up the environment, customizing the agent, and quickly testing it on sample data.

But first, let’s look at how agentic workflows differ from traditional pipelines and why that matters.

Non-Agentic Workflow vs. Agentic Workflow

In a traditional (non-agentic) workflow, product search is driven by fixed queries or rigid filter logic. It’s simple and fast, but struggles with nuanced language or evolving user intent. The system can’t adapt on the fly. It just follows predefined instructions.
On the other hand, an agentic workflow introduces flexibility and adaptability. AI agents dynamically interpret user inputs, construct queries intelligently, and adjust their approach based on the context of the interaction and feedback received. This allows them to handle more complex, ambiguous requests while improving reliability and user experience.

What Makes Up an AI Agent

To build an effective AI agent for product catalog search, the following components are essential:

Input Handling: Accepts and interprets user requests.
Feedback Handling and Memory: Incorporates user and system feedback to improve future interactions and maintains memory of past interactions.
Tools: Interfaces with external tools or databases to execute tasks.
Reasoning: Analyzes input and feedback to make informed decisions.

To build such an agent, we need an execution environment. Let’s explore how Enthusiast can serve as an effective option.

Introducing Enthusiast

Most LangChain tutorials stop at toy examples or require heavy customization to support real-world workflows. Enthusiast changes that. It’s built from the ground up to support:

Tool-based agents with LangChain and ReAct
SQL-backed querying with Django or external sources
Structured memory and retry logic out of the box
Open-source, customizable behavior
Self-hosting with cloud/local model support

Whether you're debugging search in a product catalog or surfacing relevant documents across internal departments, Enthusiast gives you a working foundation in minutes with real production logic, not just playground demos.

Build in action

Alright, now let’s bring that to life. We’ll walk through a real case: spinning up a local environment, loading data, and creating a self-correcting LangChain agent that actually understands and interacts with your product catalog.

Setting Up the Development Environment

To get started, you need to set up your development environment by cloning the Enthusiast starter repository and using its Docker configuration.

Clone the repository:
git clone https://github.com/upsidelab/enthusiast-starter
Navigate into the repository directory:
cd enthusiast-starter
Copy default configuration file and add your own OpenAI API key:
cp config/env.sample config/env
echo OPENAI_API_KEY=xxxx >> config/env
Build and run the Docker containers:
docker compose up

Once it’s running, open your browser and go to: http://localhost:10001 Log in with the default credentials:****

Email: admin@example.com
Password: changeme

You’ll be prompted to create your first dataset. Give it a name, for example, “My Dataset”.

Import a Sample Product Dataset
Enthusiast comes with a sample set of products that can be useful if you want to get started quickly. In this case, we have a set of products that represent different phone and mobile plans - with varying internet speeds, data limits, landline access, cable TV options, and more. They make a great test case for experimenting with different approaches to agentic product recommendations.

Let’s import this into our dataset:

Click on “Add Source” in the top-right corner of the screen.
From the dropdown, select “Product source”.
A popup will appear for configuring the source.
Select “Sample Product Source” from the list and click “Add”.
You should now see it listed under configured sources.
Repeat the same process for documents by selecting “Document source” from the dropdown.
This time, choose “Sample Document Source” as the type and add it as well.

Enthusiast will automatically index the dataset so it’s searchable right away.

Once the data is loaded, you can go to the Products tab to verify that the sample data was successfully imported and indexed. This ensures that your dataset is ready for querying by the agent.

Create a Custom Agent Structure

Now that your product catalog is loaded, it’s time to build an agent that can operate on it. Enthusiast supports extending and modifying agent behavior through the enthusiast_custom directory in the project.

Inside the enthusiast-starter repository, locate the src/enthusiast_custom directory. This is the package that contains your custom agents and plugins. This code will be bundled by the Dockerfile and automatically installed into your Enthusiast instance.
Let’s also install a plugin that provides a reusable base implementation for a ReAct-style agent. Run the following command inside the src/ directory to add the plugin:
poetry add enthusiast-agent-re-act
Then, create a new directory inside enthusiast_custom, calling it for example product_search. Inside this directory, add an empty init.py file to make it a Python package. This is where you’ll define your agent’s implementation.
Add your new agent to the config/settings_override.py file so that Enthusiast can recognize it. Update the AVAILABLE_AGENTS dictionary to include your custom module:

AVAILABLE_AGENTS = {
    "Product Search": "enthusiast_custom.product_search"
}

You can now rebuild and restart your Docker Compose setup to apply these changes: docker compose up --build

Once the application is restarted, you’ll see your new agent listed in the UI on the left. Time to give it some logic.

Step 1 – Generate an SQL Query

We’ll start with a basic implementation that generates an SQL query and executes it on the product catalog indexed in Enthusiast. The agent will reason through user queries and interact with the catalog to retrieve relevant results.

To do this, we’ll use the enthusiast-agent-re-act plugin that we added earlier. It provides a BaseReActAgent class, which defines the core structure of a ReAct-style agent, including how it connects prompts, tools, memory, and output processing.

Here’s how we’ll structure the product_search agent module:
product_search/agent.py

Start by defining the agent class. In a basic scenario, no overrides are required - agent’s default implementation will respond to user’s queries by creating an agent executor configured with tools and memory, and will pass the user’s request there.
Here’s what the simplest implementation looks like:

from enthusiast_agent_re_act import BaseReActAgent

class ProductSearchAgent(BaseReActAgent):
    pass

product_search/product_search_tool.py

Next, implement a tool the agent can use to run SQL queries against your product catalog.

Let’s first declare the expected input schema using a Pydantic model. This schema will be provided to the agent together with the tool definition, to let the agent determine what’s needed to call this tool. Since we specify that the tool requires an SQL query, the agent will try to produce one based on everything it knows so far in order to invoke it.

from pydantic import BaseModel, Field

class ProductSearchToolInput(BaseModel):
    sql_query: str = Field(description="sql select query to execute on the catalog_product table. include both where and order clauses")

This tool receives an SQL string from the agent, executes it using Django’s ORM, serializes the resulting product objects, and returns a message with the result. The NAME and DESCRIPTION fields in the tool definition help the agent determine when this tool is relevant to the current task.

Here’s a basic version of the tool implementation:

from enthusiast_common.tools import BaseLLMTool
from catalog.models import Product
from django.core import serializers

class ProductSearchTool(BaseLLMTool):
    NAME = "products_search_tool"
    DESCRIPTION = "use it to search for products"
    ARGS_SCHEMA = ProductSearchToolInput
    RETURN_DIRECT = False

    def run(self, sql_query: str):
        products = Product.objects.raw(self._sanitize_sql(sql_query))
        serialized = serializers.serialize("json", list(products))
        return f"Found the following products: {serialized}. Notify the user."
    def _sanitize_sql(self, sql_query: str):
               # This is where you should add some extra safety checks to the SQL query before you execute it on a live database
        return sql_query

product_search/prompt.py

Then, create the system prompt that will guide how the agent reasons and interacts with tools. Add the following:

PRODUCT_FINDER_AGENT_PROMPT = """
I want you to help finding products using the ReACT (Reasoning and Acting) approach.
Always verify your answer
Use a json blob to specify a tool by providing an action key (tool name) and an action_input key (tool input).

Valid "action" values: {tool_names}

Provide only ONE action per $JSON_BLOB, as shown:


{{
  "action": $TOOL_NAME,
  "action_input": $INPUT
}}

For each step, follow the format:
User query: the user's question or request
Thought: what you should do next
Action: 
{{
  "action": "<tool>",
  "action_input": {{"<tool_argument_name>": "<tool_argument_value>", ...}}
}}
Observation: the result returned by the tool
... (repeat Thought/Action/Action Input/Observation as needed)
Thought: I now have the necessary information
Final Answer: the response to the user

Here are the tools you can use:
{tools}

Do not come up with any other types of JSON than specified above.
Your output to user should always begin with '''Final Answer: <output>'''

Begin!
Chat history: {chat_history}
User query: {input}
{agent_scratchpad}"""

product_search/config.py

Finally, wire everything together in the config file. This tells Enthusiast which components make up your agent:

def get_config(conversation_id: int, streaming: bool) -> AgentConfigWithDefaults:
    return AgentConfigWithDefaults(
        conversation_id=conversation_id,
        agent_class=ProductSearchAgent,
        llm_tools=[
            LLMToolConfig(
                tool_class=AgentProductSearchTool,
            ),
        ],
        prompt_template=ChatPromptTemplate.from_messages(
            [
                (
                    "system",
                    PRODUCT_FINDER_AGENT_PROMPT,
                ),
            ]
        ),
        llm=LLMConfig(
            callbacks=[ReactAgentWebsocketCallbackHandler(conversation_id)],
            streaming=streaming,
        ),
        agent_callback_handler=AgentCallbackHandlerConfig(
            handler_class=AgentActionWebsocketCallbackHandler, args={"conversation_id": conversation_id}
        ),
    )

Once these components are in place and the Docker container is rebuilt, try executing a sample query:
What’s the best plan for a small family?
The agent will reason about the input, construct an SQL query, and invoke the search tool, likely failing due to invalid schema or search criteria. Let’s see what we can do with that.

Step 2 – Let the Agent Handle Its Own Errors

In the initial version, if the SQL query generated by the agent was incorrect, the tool would simply fail without giving the agent any indication of what went wrong. We can improve this by modifying the tool to catch SQL errors and return the error message as part of the response.

This way, the agent can treat the error as feedback and make another attempt, refining the query on its own.

To do this, update the run method in ProductSearchTool as follows:

from django.core import serializers
import logging

logger = logging.getLogger(__name__)

class ProductSearchTool(BaseLLMTool):
    NAME = "products_search_tool"
    DESCRIPTION = "use it to search for products"
    ARGS_SCHEMA = ProductSearchToolInput
    RETURN_DIRECT = False

    def run(self, sql_query: str):
        try:
            products = Product.objects.raw(self._sanitize_sql(sql_query))
            serialized = serializers.serialize("json", list(products))
            return f"Found the following products: {serialized}. Notify the user."
        except Exception as e:
            logger.info(e)
            return f"The query you've generated is incorrect. Error {type(e).__name__} - {e}. Please fix it and try again. Make sure it's a valid SQL."

With this change, when the SQL query fails, the agent gets the error message and can use it to revise its approach. Since the agent maintains memory of previous steps, it can iterate on its output to try and produce a valid query.

Try running the same query again:
What’s the best plan for a small family?

If the first attempt fails, the agent will receive the error, analyze it, and try to generate a better query.

Step 3 – Help the Agent Understand the Data

Letting the agent correct its own mistakes is helpful, but trial and error can be inefficient. Instead of waiting for the agent to fail and recover, we can give it a clearer understanding of the data structure up front.

One simple way to do this is by including a few sample rows from the product catalog directly in the prompt. This helps the agent understand both the schema and the shape of the data, which improves its chances of generating valid queries from the start.

To add this context, let’s override the get_answer method in your agent like this:

class ProductSearchAgent(BaseReActAgent):
    def get_answer(self, input_text: str) -> str:
        sample_products = self._injector.product_retriever._get_sample_products_json()
        agent_executor = self._build_agent_executor()
        agent_output = agent_executor.invoke(
            {"input": input_text, "sample_products": sample_products},
            config=self._build_invoke_config()
        )
        return agent_output["output"]

This method will use functionality provided by the base class to build a LangChain-based agent executor, pass the input to it, and return the response to the user. One important change here is that besides user’s input (passed as input_text ), it will also pull a few sample products from the database and will inject them into the agent’s system prompt as sample_products.

In your prompt template (prompt.py), add this placeholder at the end:
Here are some sample products in the database: {sample_products}

This additional context will be included with every call to the agent. It initializes the agent with a basic understanding of the structure and shape of the data, which makes it easier for the agent to construct accurate queries from the start.
Let’s give it a try.

You should notice that the agent now constructs queries that better match how the data is shaped. For example, it may use the category column to search for plans labeled as “Home,” or rely on the properties column to filter for plans with specific internet speeds.

Step 4 – Retry When No Results Are Found

Even if the agent is capable of generating valid SQL queries and has seen sample data, there’s still a chance it will produce a query that technically runs but returns no results.

In the current implementation, when that happens, the tool simply returns an empty list, and the agent assumes there are no relevant options. But in reality, the issue may be with how the agent built the query, not with a lack of products.

To address this, we can update the tool to return a clear message when no products are found—encouraging the agent to try a different approach. Here’s how the updated run method might look:

class ProductSearchTool(BaseLLMTool):
    NAME = "products_search_tool"
    DESCRIPTION = "use it to search for products"
    ARGS_SCHEMA = ProductSearchToolInput
    RETURN_DIRECT = False

    def run(self, sql_query: str):
        try:
            products = Product.objects.raw(self._sanitize_sql(sql_query))
            products = list(products)
        except Exception as e:
            logger.info(e)
            return f"The query you've generated is incorrect. Error {type(e).__name__} - {e}. Please fix it and try again. Make sure it's a valid SQL."

        if len(products) > 0:
            serialized = serializers.serialize("json", products)
            return f"Found the following products: {serialized}. Notify the user."
        else:
            return "No products found using this search criteria. Try using a different query and try again."

With this change, the agent receives explicit feedback when a query returns no matches. It can then choose to revise the query and try again with broader or alternative criteria.

This gives the agent an opportunity to step back and reconsider its assumptions, leading to better resilience and more accurate results when dealing with uncertain or ambiguous user requests.

Step 5 – Respect the Expected Number of Results

In some cases, a user might indicate how many products they want to see—perhaps just one recommendation or the top three matches. Right now, the agent doesn’t take that into account. It may return a long list of results, even if the user only wanted a few.

We can improve this by passing the expected number of results as part of the tool input. The tool will then check whether the number of matches exceeds this limit. If it does, it will prompt the agent to follow up and narrow the criteria.

First, update the input schema to include this new parameter:

class AgentProductSearchToolInput(BaseModel):
    sql_query: str = Field(description="sql select query to execute on the catalog_product table. include both where and order clauses")
    result_number: int = Field(description="the number of results that the user wants to get")
Then, update the run method in the tool to handle this input:
class ProductSearchTool(BaseLLMTool):
    NAME = "products_search_tool"
    DESCRIPTION = "use it to search for products"
    ARGS_SCHEMA = AgentProductSearchToolInput
    RETURN_DIRECT = False

    def run(self, sql_query: str, result_number: int):
        try:
            products = Product.objects.raw(self._sanitize_sql(sql_query))
            products = list(products)
        except Exception as e:
            logger.info(e)
            return f"The query you've generated is incorrect. Error {type(e).__name__} - {e}. Please fix it and try again. Make sure it's a valid SQL."

        if len(products) > 0:
            serialized_products = serializers.serialize("json", products)
            if len(products) > result_number:
                return f"Found the following products: {serialized_products}. However, the user would like to get at most {result_number}. Ask a follow up question that will make it possible to limit the amount of results further. Don't return these products to the user yet."
            else:
                return f"Found the following products: {serialized_products}. Notify the user."
        else:
            return "No products found using this search criteria. Try using a different query and try again."

This addition helps turn the agent into a more effective product search assistant. Instead of assuming that the initial results are appropriate, the agent now reflects on the quantity of data returned, checks it against user expectations, and adjusts accordingly. This creates a more collaborative flow where the agent and user refine the query together to land on a relevant result.

Step 6 – Enable the Agent to Finalize a Purchase

Once the user finds a plan that matches their needs, the next logical step is to help them act on it. Right now, our agent can recommend products but doesn’t support any kind of checkout process.

To make this possible, we’ll give the agent the ability to generate a contract URL the user can follow to finalize their purchase. This effectively allows the agent to transition from discovery to action.

Start by creating a new tool, PurchaseTool, which accepts a plan_sku and returns a contract finalization link:

from enthusiast_common.tools import BaseLLMTool
from pydantic import BaseModel, Field

class PurchaseToolInput(BaseModel):
    plan_sku: str = Field("SKU of the plan that the customer wants to purchase")

class PurchaseTool(BaseLLMTool):
    NAME = "purchase_tool"
    DESCRIPTION = "use when the user wants to sign a contract for a plan"
    ARGS_SCHEMA = PurchaseToolInput
    RETURN_DIRECT = False

    def run(self, plan_sku: str):
        url = f"https://fiberup.upsidelab.net/finalize_contract?{plan_sku}"
        return f"Show the following url to the user, and ask them to use it to finalize their signature {url}"
Then, register this tool in your agent’s config:
from .purchase_tool import PurchaseTool
...
llm_tools=[
    LLMToolConfig(tool_class=AgentProductSearchTool),
    LLMToolConfig(tool_class=PurchaseTool),
],

Lastly, modify the search tool’s return message slightly to encourage the agent to propose a contract. The agent will likely figure it out even without this hint, but there’s no harm in pushing it more explicitly:

return f"Found the following products: {serialized_products}. Notify the user and propose signing a contract for this plan."

With this addition, your agent becomes a guided assistant that helps the user discover a suitable plan and smoothly transition into completing the purchase.

Step 7 – Ask for Additional Customer Details

Before the agent pushes the user to sign a contract, it can also ensure that it collects any additional information needed to complete the process—such as the customer’s name and location.
To support this, update the PurchaseToolInput schema with two new fields:

class PurchaseToolInput(BaseModel):
    plan_sku: str = Field("SKU of the plan that the customer wants to purchase")
    customer_name: str = Field("Name of the customer")
    zipcode: str = Field("Zipcode of the customer")
Then modify the run method to include this information in the contract URL:
import urllib.parse

class PurchaseTool(BaseLLMTool):
    NAME = "purchase_tool"
    DESCRIPTION = "use when the user wants to sign a contract for a plan"
    ARGS_SCHEMA = PurchaseToolInput
    RETURN_DIRECT = False

    def run(self, plan_sku: str, customer_name: str, zipcode: str):
        url = f"https://fiberup.upsidelab.net/finalize_contract?{plan_sku}&name={urllib.parse.quote_plus(customer_name)}&zipcode={zipcode}"
        return f"Show the following url to the user, and ask them to use it to finalize their signature {url}"

Thanks to the structured schema and tool description, the agent will know that it must collect these inputs from the user before invoking the tool. If the information isn’t provided initially, the agent can follow up with questions like:

Could you tell me your name and zip code so I can finalize the contract?

This closes the loop and ensures that the agent not only helps discover the right plan but can also guide the user through to a complete and personalized purchase process.

Summary

In this walkthrough, we explored how to build a practical AI agent for product catalog search using Enthusiast. Starting from a basic ReAct-style agent capable of generating SQL queries, we incrementally introduced more sophisticated behaviors:

Error recovery through exception feedback
Schema-aware reasoning via sample data
Retry logic when no results are found
Adapting results to match user expectations
Finalizing user purchases with structured follow-up
Collecting required customer details before contract generation

Each step was designed to bring the agent closer to an experience that feels like a helpful, iterative assistant.

To try this yourself, check out the Enthusiast starter repository: https://github.com/upsidelab/enthusiast

For full documentation and examples, visit: https://github.com/upsidelab/enthusiast

🌍 Scaling Globally with Spree? Here's How to Automate Translations (Using DeepL + 1 Line of Code)

Chris Yaowen Zhang — Wed, 23 Jul 2025 13:11:47 +0000

Manually translating product catalogs? Not only is it tedious — it slows down your entire international expansion.

The good news? With Spree 4.6 and a new gem called spree_automation_interfaces, you can plug in translation tools like DeepL and have your products auto-translated with the click of a button.

Let me show you how we got it working in under 20 minutes.

🛒 Problem: Spree supports translations, but the process is still manual

Spree’s backend already allows you to manage product translations across multiple locales. But historically, companies relied on exporting CSVs → emailing translators → re-importing the files — a messy, slow, error-prone loop.

Even with tools like Google Translate or DeepL, manual copy-paste just doesn't scale.

So we decided to automate the process, while still keeping room for human review.

🔧 Solution: Spree Automation Interfaces + DeepL

As a part of supporting Spree users, we at Upside published spree_automation_interfaces that lets you connect automated translation services directly to your Spree store.

In this guide, we’ll show you how to:

Set up your store for multiple languages (French & German)
Plug in DeepL using a custom provider
Add an “Auto-Translate” button inside your admin panel

Step 1: Enable Multiple Languages in Spree
(You can skip this if your store already supports multiple locales)

In the Spree Admin Panel, go to Settings → Store in the sidebar to the left.
You will see a list of "Supported Locales". For the purpose of this guide, we'll add set up additional locales: French and German.

Click Update.
To confirm that the setup works correctly, navigate to a product in the admin panel, and go to its "Translations" tab. You should see a table for filling additional translations - this means that the setup is correct.

🎉 At this point, you can provide the translations manually. It's time to automate the process.

Step 2: Add the Right Gems
We need to add the following gems:

spree_automation_interfaces - that brings the capability of automated translations into Spree
deface - to allow spree_automation_interfaces to add new features to the admin panel
deepl-rb - which we will use as a translations provider

gem 'spree_automation_interfaces', github: 'spree-contrib/spree_automation_interfaces'
gem 'deface'
gem 'deepl-rb', require: 'deepl'

Then run:

bundle install

And restart your development server.

Step 3: Implement the DeepL Translations Provider

Now, we need to provide an implementation for the Automation Interfaces, that will be called by Spree when the user requests translations to be created.

The required interface is defined as follows:

module Spree
  module Products
    module Translations
      interface _AutomatedTranslationsProvider
        def call: (source_attributes: Hash[String, String], source_locale: String, target_locale: String) -> Hash[Symbol | String, String]
      end
    end
  end
end

Let's create an implementation in app/services/spree/translations/deepl_translations_provider.rb

module Spree
  module Translations
    class DeeplTranslationsProvider
      def call(source_attributes:, source_locale:, target_locale:)
        name = source_attributes['name']
        description = source_attributes['description']
        translated_name, translated_description = DeepL.translate([name, description], nil, target_locale)
        {
          'name' => translated_name,
          'description' => translated_description
        }
      end
    end
  end
end

Step 4: Initialize DeepL Client
We also need to initialize DeepL's client. Let's create the following initializer config/initalizers/deepl.rb

DeepL.configure do |config|
  config.host = ENV.fetch('DEEPL_HOST', 'https://api-free.deepl.com')
  config.auth_key = ENV.fetch('DEEPL_API_KEY')
end

Step 5: Plug Your Provider into Spree

Finally, configure Spree to use the provider we've just created

In config/initializers/spree_automation_interfaces.rb

Rails.application.config.after_initialize do  
  SpreeAutomationInterfaces::Dependencies.products_automated_translations_provider = 'Spree::Translations::DeeplTranslationsProvider'
end

Step 6: Try It!
Head to any product’s Translations tab in the admin panel.

You’ll see a new button: “Translate automatically”

Once you click it, it will fetch translations using the provider we've just implemented and add them to the product.

Bonus: Go Further with Sidekiq or API Triggers

This interface also works programmatically, so you can:

Trigger translations from code
Run a background job to translate your entire catalog
Schedule daily translation syncs

For example, you could build a Sidekiq worker that batches and translates 10,000 SKUs overnight.

🧱 Building Open Source Projects with Friends - Do and Donts?

Chris Yaowen Zhang — Tue, 15 Jul 2025 13:31:49 +0000

There’s something undeniably rewarding about building with people you admire — not just as coworkers, but as friends.

We recently launched Enthusiast, an open-source AI agent designed to help teams access their internal knowledge in plain language — think product info, support docs, or internal processes, all searchable and actionable.

But instead of telling you about the tech (we’ll link that below), I want to talk about something you don’t often hear in dev blogs: what it’s like to build open source with friends — and why that turned out to be our biggest strength.

What It’s Like to Build with Friends

There’s a classic warning in tech: “Don’t work with friends.”

And after having three friends as technical co-founders in the past… I can confirm: the fear is valid.

Yes, personal dynamics can blur feedback.
Yes, tough conversations might get avoided.
And yes, your project can strain... or worse, even breaking the friendship. (At least for a while.)

So why did I start to work with my friends on Enthusiast again? And how did it proved the opposite?

We were mindful of the risks.

I'm always a believer that past failures contribute to future success. Thanks to the past 3 "failed" friendship turned co-founders, I became much more aware how to navigate the situations. And here’s how I approached it this time:

📐 Evaluation phase
Before committing, I asked myself two questions:

Do we share the same values?
How do I feel after spending time with them?

My friend Kasia often says her life principle is "Love, Health, and Wealth". That made me thought of mine. Turned out to be Faith, Love, Health, Wealth, and Joy (yeah, I'm greedy). But what matters is that we have overlaps.

What's as important is that after every hangout, I felt recharged. Calmer. Softer. That was a green flag, for both emotionally and energetically.

🚩 Decision Phase
Once we agreed to start building together, I did the basics:

Clear expectations around roles, time, and compensation
A roadmap and even a strategy workshop

(🛼 Spoiler: the roadmap became irrelevant pretty fast. Priorities shifted. But the alignment exercise still built trust.)

🏃‍♀️ Working Together: A Daily Practice
Friendship doesn’t replace structure. If anything, it demands more of conscious planning. (Now I think of it. Any relationship should follow this rule.)

So I created a living “Do & Don’t” list for working with each friend. On top of that, the simple but foundational things are done even more intentionally:

Be prepared for meetings
Say lots of “yes, and…”
Follow through on what we promise
Stay transparent and give updates (over-communication is okay!)
Be playful, without crossing boundaries

(I like to plug fun quizzes into our weekly updates 😄)

*Did we hit conflict? *

Yes, of course.

But we operate on the shared belief that good work comes from iteration, not perfection, so we don't always have to be the one that is right.

So we compromise, experiment, and move forward forward together. Because progress is not done by one decision.

As long as we’re learning and acting, momentum builds. And with enough of that, speed follows.

Are we still the same friends outside of work?

I was really worried about this in the beginning, which was great. Because that kept me alert (while looking chill) enough to put more understanding and careful action into our relationship. (No, it's not exhausting. Because I love them.)

At the end of the day, relationships are built through time intentionally spent. Working together gave us that.

How about outside work? We don't really talk about work (that much). Instead, we continue to talk about life, faith, family, and the future. And we also plan fun activities to celebrate life together.

(One of those golden summer evenings by the lake, hanging out with Kasia and my dog)

If anything, working together has helped us grow together — not just on the project, but also on the journey of life.

And for that, I’m deeply grateful.

Want to Build With Us?
If you:

Believe in open source
Like the idea of tools built with care and speed
Or are simply curious about what a "Company GPT" might look like…

We’d love to have you join us.
👉 GitHub Repo

Improving INP: How to Make Your React App Feel Faster

Chris Yaowen Zhang — Wed, 09 Jul 2025 12:52:42 +0000

If your React app feels a little sluggish when users click, type, or scroll—it’s not just a UX issue anymore. Google’s new INP (Interaction to Next Paint) metric measures how quickly your app responds to interactions, and it now plays a role in SEO rankings.

What is the “Interaction to Next Paint” (INP) metric?

Google released a new Core Web Vital called Interaction to Next Paint (INP) in 2024. It measures the speed of interactions with the website.

In this article, we will walk you through INP from the developer’s perspective. This includes guiding you through testing for INP, verifying potential causes of low INP scores, and how you can leverage new React tools to optimize your website for recent changes.

Interaction to Next Paint (INP) is a scoring mechanism that analyzes latency from the time the user begins the interaction to the moment the next frame is presented with visual feedback. We’ll take a closer look at what mechanisms in modern applications cause low INP score later.

It measures for one of the following interactions.

Clicking with a mouse
Tapping on a device with a touchscreen
Pressing a key on either a physical or onscreen keyboard

For example, when the user types a character into the search input that filters a list of cats by their breeds, the INP value will be a duration from the moment of triggering a group of keyboard events (keydown, …, keyup) to the moment of re-rendering the updated list

The main goal of INP is to minimize this duration, ensuring swift visual feedback for user interactions and optimizing overall responsiveness on the page.

What INP isn't?

There is already a well-known metric called “First Input Delay” (FID) which measures the time from when a user first interacts with a page to the time when the main thread is next idle. It offered developers a new way to measure responsiveness as real users experience it.

FID vs INP - What’s the difference?

Comparing these two mechanisms reveals similar measurement goal - a page responsiveness. One of the essential differences is that the FID measures only initial interaction, while the INP takes the worst latency recorded during the entire session.

💡 There's a little nuance to how the INP is calculated depending on the total interactions per website. For websites with more than 50 interactions per session, it takes the 75th percentile of the slowest latency recorded. This eliminates potential hiccups for pages with large numbers of interactions.

Even though first impressions are important, the first interaction is not necessarily representative of all interactions throughout the life of a page. This is one of the reasons why INP replaces the FID.

Should you care about INP?

In short, yes, if you care about SEO or user experience. The INP will be another metric that not only directly influences your website's ranking on Google search results but also provides valuable insights for tracking the responsiveness of your product.

Receiving poor results from the INP might be a sign to profile your application for performance.

Measuring INP

For a released product, that runs in a production environment, accumulates user traffic, and qualifies for inclusion in the Chrome User Experience Report (CrUX) you can quickly measure the new INP metric using PageSpeed insights. It gives a snapshot of your page and origin's performance over the previous 28 days.

Manually diagnose slow interactions

Ideally, you should rely on the field data from the CrUX report, that suggests your website struggles with slow interactions. Sometimes, however, it’s not possible, and we need another way.

The main goal is to identify the problem with lowest amount of effort at first. For manual investigation, we’ll use the Web Vitals addon. Then, dive deeper into what causes the issue. For that, we can leverage the Profiler available in the browser’s DevTools tab.

Using Web Vitals addon (Chrome)
Requirements:

Chrome browser
Web Vitals addon
React Developer Tools addon (Useful for highlighting component updates)

Let’s start with a simple configuration of the Web Vitals addon. This will display HUD overlay as well as log information about each interaction with the page in the console.

We’ll use React Developer Tools to highlight component updates. This will help us with identifying what’s happening in our application.

💡 Once React Developer Tools is installed, open Chrome developer tools, then “Profiler” section. Enable “Highlight updates when components render”.

Let’s go back to previously introduced app which allows us to filter cats by their breeds. First issue we can observe is that the list itself is long. We’re displaying over 1900 results at once. That’s already something we could optimize, but let’s leave it for later. Let’s try to interact with the page by typing “B” into the input, and then clearing it.

Few things are happening here:

A controlled Input component updates immediately on each character we type.
A list updates on each input change.

The moment of “freeze” shown on the video is the main issue that causes INP score to suffer. We can verify that by observing the INP delta times logged in the console on the right. This translates into general poor result of the INP score for our website (344 ms).

What exactly causes poor INP results?

Figuring out what's causing poor INP is the most important, and the most difficult step on the road of enhancing the User Experience (UX).

As Google Chrome’s team suggest, interactions can be broken down into three phases:

The input delay: It starts when the user initiates an interaction with the page, and ends when the event callbacks for the interaction begin to run.
The processing time: It consist of the time it makes for event callbacks to run to completion.
The presentation delay: It’s the time it takes for the browser to present the next frame which contains the visual result of the interaction.

I found a bottleneck - now what? Optimizing your React app for performance.

As a React developer you’re probably aware of core concepts used to optimize the app.

I’d distinguish following:

Memoization memo, useMemo, useCallback
Debouncing/Throttling (using a hook or a higher-order-function that triggers the callback)
List virtualization
List pagination

Let’s go back to our simple app about cats. Take a look at the component that provides browsing functionality:

export function CatsBreedsBrowserTemplate() {
  // 1. Fetching a list of all cats from the API (no limitting the response).
  const { data, isLoading } = useCatsData();

  // 2. Storing a `SearchBar` value.
  const [filterByBreed, setFilterByBreed] = useState("");

  // 3. Filtering the `data` array based on the `filterByBreed`.
  const searchResults = useMemo(() => {
    const catsData = data ?? [];

    if (!filterByBreed || !catsData.length) {
      return catsData;
    }

    return catsData.filter((cat) => cat.breed.includes(filterByBreed));
  }, [data, filterByBreed]);

  // 4. Rendering the list of cats
  const renderCatsSearchResult = useCallback(() => {
    if (isLoading) {
      // 5. Early-return: show loading indicator
      return <CircularProgress />
      );
    }
    if (!searchResults.length) {
      // 6. Early-return: show empty placeholder
      return <p>No cats found :(</p>
    }
    // 7. Return actual list component.
    return <CatsList cats={searchResults} />
  }, [isLoading, searchResults]);

  return (
    <div>
      <SearchBar
        label="Filter by cat's breed"
                // 8. On each search input change, set `filterByBreed` state.
        onChange={(e) => setFilterByBreed(e.target.value)}
      />

      {renderCatsSearchResult()}
    </div>
  );
}

Let’s see how can we utilize already known concepts to optimize the INP time.

Issue #1 - List of cats re-renders on each change of the SearchBar component.

As we know from the React’s Reconciliation model, when state changes, React re-renders the components and all its descendants. By calling setFilterByBreed on each SearchBar change, we perform unnecessary re-renders of the expensive CatsList component.

Instead of triggering onChange immediately, let’s apply debouncing on the SearchBar component.

You can use a library, or implement a simple debounce utility by yourself. Here’s a simple higher-order-function that triggers callback on timeout:

const DEFAULT_DEBOUNCE_DELAY = 500;

export const debounce = <T,>(
  callback: (value: T) => void,
  delay = DEFAULT_DEBOUNCE_DELAY
) => {
  let timer: number;
  return (value: T) => {
    clearTimeout(timer);
    timer = setTimeout(() => callback(value), delay);
  };
};

A SearchBar component is a simple wrapper around mui’s TextField component:

export interface SearchBarProps extends Partial<TextFieldProps<'standard'>> {}

function SearchBar({ onChange, ...rest }: SearchBarProps) {
  return <TextField onChange={onChange} {...rest} />;
}

Using debounced onChange:

export interface SearchBarProps extends Partial<TextFieldProps<'standard'>> {}

function SearchBar({ onChange, ...rest }: SearchBarProps) {
  // Re-initializing debounced onChange on `onChange` change (no pun's intented).
  const debouncedOnChange = useMemo(
    () => onChange && debounce(onChange),
    [onChange]
  );

  return <TextField onChange={debouncedOnChange} {...rest} />;
}

This improves the INP, because we no longer try to re-render expensive component on each user interaction.

We still however, render a long list of elements in the DOM, synchronously.

Issue #2- List renders all elements at once causing low INP score by freezing the layout

This time we could:
a) Virtualize the list (by limiting amount of elements attached to the DOM tree, and displaying only such that are visible in the viewport + some offset).
b) Paginate the data (combine intersection-observer with appending next elements to the DOM on user scroll).

Both solutions allow us to keep initial list lightweight, and get rid of the layout “freeze” that blocks us from interacting with the website. There’s a plenty of resources for implementing both Virtualization, and Pagination in React. You can check one of them.

React ≥18, concurrency, and the new mental model of reactivity

Apart from what I already mentioned, there’s a relatively new approach to reactivity in React using its latest features. By introducing concurrency, we’re now able to transition (no pun intended) from well-known optimization techniques, take the responsibility from the developer, and introduce less intrusive optimizations managed by the tool itself (with a bit of our help).

Non-urgent updates

With React < 18, every update is urgent. From React 18, every update is urgent by default, but now we’re able to set a low priority on updates that could potentially block the UI.

Instead of thinking about re-renders, we should start thinking about the urgency of updates we perform.

Let’s once again go back to our unoptimized cats browser (Fig. 1).

export function CatsBreedsBrowserTemplate() {
  ...

  return (
      ...

      <SearchBar
        ...
    // 1. On each search input change, set `filterByBreed` state.
        onChange={(e) => setFilterByBreed(e.target.value)}
      />

      ...
  );
}

Remember the part (Fig. 1 [8], Fig. 5 [1]) where, we update the state of the filterByBreed on each SearchBar change? We’ll utilize the useTransition hook to mark this as non-urgent update.

// 1. Import the `useTransition` hook.
import { useTransition } from "react";

export function CatsBreedsBrowserTemplate() {
  // 2. Declare `useTransition` fields.
  const [isPending, startTransition] = useTransition();

  ...

  return (
      ...

      <SearchBar
        ...
        onChange={(e) =>
          // 3. Mark state update as non-urgent,
          //    by wrapping it with the `startTransition` function.
          startTransition(() => {
            setFilterByBreed(e.target.value);
          })
        }
      />

      ...
  );
}
view raw

(Fig 6)

As simple as that, we were able to eliminate the “freezing” UI issue.

Why this works?‍

Under the hood, react maintains a queue of updates ready to be performed. Before React 18, all updates were performed sequentially until there were no more updates left. When we update the component’s state, React will queue this update, and perform it until it completes. This however, due to the JavaScript event loop’s “Run-to-completion” strategy, blocks processing of other tasks (that’s why we can see the UI “freeze” with non-optimized example).

With the release of React 18, a significant change introduced a new mechanism of processing the updates queue. Instead of forcefully processing an update, React now schedules a unit of update’s work, checks if the browser allows to perform it, and then completes it. Otherwise, it gives the control back to the browser. This way, long-running non-urgent updates no longer blocks the browser’s work that needs to be performed due to the higher priority.

Transition is only one of the concurrent scheduling mechanisms. Make sure to check all already-existing, and incoming hooks such as useDeferredValue, or use.

Is concurrent scheduling a solution to all problems?

While the new concurrency model effectively addresses common challenges, it's not a silver bullet applicable to every aspect of your code.

Since useTransition splits the work in chunks, for expensive-to-render components, splitting the work is impossible. That’s where we need to combine the new concurrent scheduling with already known optimization strategies.

Avoid premature optimization

Understanding the optimization techniques mentioned and comprehending how the INP calculates the Web Vitals score is crucial for delivering a seamless and responsive user experience for your product.

While optimization enhances performance, it often comes with associated costs. While it's tempting to immediately implement efficiency improvements, many enhancements result in higher memory consumption and maintenance challenges.

It is essential to prioritize simplicity in our solutions. I advise vigilant monitoring of your application to identify areas causing issues and optimizing those selectively, rather than prematurely optimizing, which can introduce unnecessary complexity to your project.

Workshop: Build Your Agentic Knowledge Base Using Open-Source Tools

Chris Yaowen Zhang — Wed, 09 Jul 2025 09:19:57 +0000

Hi everyone!

If you're exploring how to build your own internal “Company GPT” that runs on your stack, connects to your private data, and doesn’t compromise on control or security, we’re hosting a live hands-on workshop just for that!

In this workshop, we will use open-source tools, so no black boxes, no vendor lock-in. You’ll set up custom AI agents that can handle repetitive workflows like content generation, product queries, and support requests, all in a few clicks.

Whether you're curious about LLM-based retrieval, self-hosted agents, or just want to see a clean architecture for internal AI, this is a practical deep dive to get you started fast.

We will also show you how to ensure reliable answers with layered validation, so your AI agents stay accurate and trustworthy

By the end of this session, you’ll walk away with:
✅ A fully functional, AI-powered knowledge base
✅ Customizable AI agents trained on your internal content
✅ Ready-to-use code and templates to expand or scale
✅ Live expert guidance (no prior AI experience required)

⏰ Time: 5:00 PM CET / 11:00 AM ET July 16th
📍 Location: Online
✋ Format: Live workshop with Q&A
🎁 Access to source code and practical templates (GitHub link here)
❗️❗️Note: This is an online event, register to get Zoom link here!

Looking forward to seeing you there!

Looking for Senior Frontend Engineer (React/Vue)

Chris Yaowen Zhang — Tue, 08 Jul 2025 08:11:08 +0000

Dear Dev community,

We are hiring a senior frontend engineer. If you are interested, please see below job description or share with people that might be relevant.

Thanks!

Job description

At Upside, we’re not just shipping features - we’re partnering with ambitious companies across the globe to solve complex, meaningful problems through technology. We work in small, focused teams, combining engineering, product thinking, and design, to deliver outcomes, not just output.

Open-source is a strategic part of our work. We build and maintain a number of open-source projects that support our clients and internal R&D. Our latest release is Enthusiast, an AI Agent framework tailored for e-commerce use cases.

Role Overview:

We are seeking a Senior Frontend Engineer to drive the design, development, and maintenance of high-quality platforms for global clients. In this role, you will be a key contributor to our client-facing solutions, ensuring performance, scalability, and reliability. You’ll collaborate within a dynamic team of 3 to 9 developers, working together to craft scalable and resilient systems.

Job requirements

Extensive commercial experience in building web-based applications.
In-depth knowledge of modern frontend stack and knowledge of at least one framework (React or Vue).
Experience with Next.js or other SSR frameworks.
Experience with GraphQL.
Solid FP/OOP skills.
Practical experience with Continuous Integration/Continuous Deployment (CI/CD) practices and modern cloud environments.
Strong consulting mindset - balancing technical excellence with client needs and communicating effectively with both technical and non-technical stakeholders.
Fluent English, with clear written and verbal communication.

Bonus Points if you have:
Experience in advanced web animations (e.g. framer-motion, GSAP, Rive/Lottie).

Experience in advanced web technologies such as WebGL, WebSockets, WebAssembly, web workers, and PWA.

Your responsibilities would be

Developing complex digital platforms - from architecture decisions to implementation.
Work in autonomous teams (3-9 people) delivering mission-critical systems.
Collaborate directly with technology leaders, founders, and domain experts to shape solutions that drive real business outcomes.

What We Offer

Transparent progression path.
Flexible work: office in Kraków, hybrid or remote from anywhere in Poland, with company on-sites.
Additional paid days off beyond national holidays.
A deeply collaborative, engineering-led company culture.
Support for ongoing learning.

Salary:

Senior Engineer: 20 800 - 25 900 PLN net + VAT
Lead Engineer: >26 500 PLN net + VAT
Including additional days off

Location:
Office in Kraków, hybrid, fully-remote.

Apply via THIS LINK!

Rendering HEIC on the Web: How to Make Your Web App Handle iPhone Photos

Chris Yaowen Zhang — Wed, 02 Jul 2025 15:50:06 +0000

App developers, your iPhone users are uploading photos, and it’s breaking your app!

That’s because Apple’s default image format, HEIC, isn’t supported by most browsers.

We've seen this issue firsthand with quite a few clients that have iOS-heavy user bases. You might not notice it during testing, but the moment a real user uploads a photo from their iPhone, things fall apart.

In this post, we’ll walk through why it breaks your app, and how to fix it—with practical options for frontend and backend conversion, plus third-party tools. Alright, let's dive in!

First, let’s render it on our site 🔥

<img src="/assets/dog.heic" alt="">

Yeahhh… You see HEIC, while being highy efficient, is not highly supported in the browser environments. As you can see here none of the browsers besides latest releases of Safari support this format.

Consequently the code snippet above won’t work in most browsers.

Why should we care?

Implementing HEIC support in your web app is a time consuming task that often requires revision to make sure that your solution is compatible with your target users’s browsers. If 99% of them use Android then you can probably prioritise other tickets that have been sitting in your backlog.

Let’s look at some data though. According to Statcounter around 28% of smartphones worldwide are running IOS. Those 28% might want to upload a HEIC image they took with their iPhone to your app.

However, most of the businesses are running locally and are limited to a specific region or a country. If your business is targeting the United States market, the share of iOS users on mobile reaches almost 60%. In that case you can be sure that sooner or later a HEIC image will land in your uploader and there has to be a piece of code that will handle it.

Possible solutions

As you can see the only way to be able to reliably render HEIC in a browser is to convert it into a format that is widely supported. Right now, no one knows if Apple is going to change its licensing and allow other browsers to use HEIC images, and even if it does, waiting for it to happen doesn’t seem like a quick solution.

Here are a few approaches that I’m going to go through so that you can compare and choose the one that suits your case best:

Converting on the front-end
Converting on the back-end
Using third party solutions

Front-end conversion

When searching for HEIC conversion tools you are most likely to stumble upon libheif which is a great open source library written in C. But how do we get to run it with JavaScript in the browser environment?

There is the libheif-js library which lets you directly interact with the precompiled JavaScript libheif or run it using WebAssembly. You can also use it in node.js but I’ll cover that later. However, this is a low level solution that requires you to decode the image yourself and then render it using canvas. You can then preview it with a canvas or convert it into a jpg blob and display it with an img tag.

Here is an example of how that can be done. As you can see it’s complicated and requires a deep understanding of each step which might not be ideal if you only want to get the job done without getting into details of how libheif works.

Let’s take a look at the library called heic2any which is using libheif under the hood and makes it super simple to get started with HEIC conversion in the browser. Here’s an example implementation:

We have a file picker that accepts an HEIC image and previews it using the img tag. As you can see it takes a few lines of code to convert the image and generate a URL which we can then preview like any other static image.

I think it’s worth elaborating on one thing here…

Good to know: What is a blob?

You might be asking yourself - what is this whole blob thing in JavaScript? You see it everywhere when dealing with files and sometimes it might not be obvious why it’s being used in the first place.

Blobs allow you to create file-like objects made out of raw data and a specified MIME type. They are stored in the browser’s memory and can be accessed using a generated url. It is a perfect solution for any image manipulation because Blobs allow for storing the image data and then easily referencing it - in the example above by passing it as a src attribute.

This video gives a great introduction to how to create a blob yourself and explains a few related concepts like ArrayBuffers:

And just like that we managed to handle an HEIC image and view it without using any backend. However, keep in mind that converting images on the client is not an ideal solution and has its drawbacks that shouldn’t be undermined.

First of all, it’s heavy for the browser to first load the heic2any (2.7 MB) and then to run the actual image conversion. You never know if someone is using your app on the latest MacBook Pro or on the iPhone SE from 2016.

Besides that, you should never trust the client and handing over such crucial task as converting files to the client might be dangerous.

In our opinion frontend conversion should be only used for previewing and if you want to upload a file to the server you should send it in the original HEIC format to continue processing on the backend.

Backend conversion

Depending on your stack there are many tools you can use for handling the HEIC conversion on the backend.

We mentioned libheif before which has dedicated libraries for various languages like Node.js, C#, Python, Rust etc. Let’s take a look at an example implementation in Node.js, specifically using the heic-convert package which is an official user friendly wrapper for libheif.

Using Express.js you can quickly create a server that will handle an HEIC file and return the converted jpeg file. Multer is a middleware that lets you easily handle form data and files - I’m using the memory storage here which doesn’t save any file to the disk but with Multer you can also make the files get saved and return the image url that can be accessed multiple times.

To take it one step further in a production environment you would probably want to save those files to S3 or other storage solution and host them using a CDN.

In this example you simply get the image data which on the client side can be easily parsed to a blob and rendered. To use our backend implementation in the frontend demo I showed above you simply need to replace the heic2any call with a POST request to the server:

const formData = new FormData();
formData.append("file", blob);

fetch("http://localhost:4000", {
  method: "POST",
  body: formData,
})
  .then((res) => res.blob())
  .then((res) => {
    setPreview(URL.createObjectURL(res));
    setLoading(false);
  });

Here you go!

We have moved the crucial process of converting an image to the backend making it more reliable. Compared to the frontend conversion the performance will be more consistent as it’s always your server that is running the conversion.

The biggest disadvantage here is the effort that you have to put into spinning up your own server and maintaining it (unless you already have it).

Overall, if you need to reliably handle HEIC conversion with a prospect of also handling storage and referencing those images in a database, implementing a backend conversion is probably the best way to go.

Third party solutions

While implementing the file conversion yourself is not that hard nowadays due to a lot of libraries that get the job done for you, it can still be tricky to choose the right one and be sure that your custom implementation will work well in all the cases.

For example if your output image seems to have a slightly different color palette, you either have to tweak the parameters to match your expected output (assuming the library you’re using implements such interface) or switch to another library that might or might not solve the issue.

Also, the solutions I showed in the previous sections do not handle the file metadata. You would need to implement it separately.

You will also need to perform updates of those libraries and test if the update didn’t break anything in your code.

If you don’t want to struggle with all those things and just want to get a solution that is easy to implement and always works, you should look for an API that will outsource this process for you. For example:

They provide a straightforward documentation and you can always reach out to their support if your business has more specific requirements.

There are also more profound, end-to-end solutions for handling images which go beyond just image conversion:

They allow you to upload images, convert them, process them in many different ways like cropping, resizing or applying filters and access using a CDN. You can also specify the storage solution so it doesn’t have to be their servers. Filestack also provides you with an embedded file picker which means you don’t even have to make any requests yourself.

Using GPT for natural language querying

Chris Yaowen Zhang — Fri, 27 Jun 2025 15:09:48 +0000

Year 2023 was supposed to be the year of AI. For sure we all see some changes, especially in social media walls. For example, my LinkedIn is flooded with AI generated texts or images.
However, I prefer to focus on the realms of Artificial Intelligence that are equally fascinating and beneficial. I tried using AI, specifically GPT, to transform natural language into structured, machine-readable forms. This approach is used to generate machine-readable forms that translate into other use cases as well, i.e. the insights in this blog post can inform other equally interesting and useful applications. In this article you will learn how to use GPT for data processing and allowing users to provide natural input to an application. Let’s take a look into some interesting use cases.

Example: Let’s cook!

The Upside team found a pretty good spaghetti recipe available here.
We can try transforming it into a JSON.

First, let’s define a JSON schema of the JSON that we’d like to get from GPT:

The schema can look like this:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://upsidelab.io/recipe.schema.json",
  "title": "Recipe",
  "description": "A recipe definition",
  "type": "object",
  "properties": {
    "ingredients": {
      "description": "The list of ingredients required to prepare the recipe",
      "type": "array",
      "items": {
        "$ref": "#Ingredient"
      }
    },
    "steps": {
      "description": "The list of steps required to prepare the recipe",
      "type": "array",
      "items": {
        "$ref": "#Step"
      }
    }
  },
  "$defs": {
    "ingredient": {
      "$anchor": "Ingredient",
      "type": "object",
      "properties": {
        "id": { "type": "number" },
        "name": { "type": "string" },
        "quantity": { "type": "string" }
      }
    },
    "step": {
      "$anchor": "Step",
      "type": "object",
      "properties": {
        "id": { "type": "number" },
        "name": { "type": "string" },
        "description": { "type": "string" },
        "dependsOnSteps": {
          "type": "array",
          "items": { "type": "number" }
        },
        "dependsOnIngredients": {
          "type": "array",
          "items": { "type": "number" }
        }
      }
    }
  }
}

Then a prompt for GPT. We’ll split it into three parts:

Schema definition
The recipe to be parsed
A summary of what we expect it to return (just the JSON, conforming to the schema)

Summarize the following recipe into a json that conforms to the following json schema delimited by three backticks

\`
${schemaDefinition}
\`
Use the following recipe, which is delimited by three backticks
\`
${recipe}
\`
Respond with the JSON only. Your output should be parsable by a JSON parser and should conform to the JSON schema definied before.

{
    "ingredients": [
        { "id": 1, "name": "olive oil", "quantity": "1 tbsp" },
        { "id": 2, "name": "smoked streaky bacon", "quantity": "4 rashers" },
        { "id": 3, "name": "medium onions", "quantity": "2" },
        { "id": 4, "name": "carrots", "quantity": "2" },
        { "id": 5, "name": "celery sticks", "quantity": "2" },
        { "id": 6, "name": "garlic cloves", "quantity": "2" },
        { "id": 7, "name": "rosemary sprigs", "quantity": "2-3" },
        { "id": 8, "name": "beef mince", "quantity": "500g" },
        { "id": 9, "name": "plum tomatoes", "quantity": "2 x 400g tins" },
        { "id": 10, "name": "basil", "quantity": "small pack" },
        { "id": 11, "name": "dried oregano", "quantity": "1 tsp" },
        { "id": 12, "name": "bay leaves", "quantity": "2" },
        { "id": 13, "name": "tomato purée", "quantity": "2 tbsp" },
        { "id": 14, "name": "beef stock cube", "quantity": "1" },
        { "id": 15, "name": "red chilli", "quantity": "1 (optional)" },
        { "id": 16, "name": "red wine", "quantity": "125ml" },
        { "id": 17, "name": "cherry tomatoes", "quantity": "6" },
        { "id": 18, "name": "parmesan", "quantity": "75g" },
        { "id": 19, "name": "spaghetti", "quantity": "400g" },
        { "id": 20, "name": "crusty bread", "quantity": "to serve (optional)" }
    ],
    "steps": [
        { 
            "id": 1, 
            "name": "Prepare the saucepan", 
            "description": "Put a large saucepan on a medium heat and add 1 tbsp olive oil.", 
            "dependsOnSteps": [],
            "dependsOnIngredients": [1] 
        },
        { 
            "id": 2, 
            "name": "Cook bacon", 
            "description": "Add 4 finely chopped bacon rashers and fry for 10 mins until golden and crisp.", 
            "dependsOnSteps": [1],
            "dependsOnIngredients": [2] 
        },
        { 
            "id": 3, 
            "name": "Add vegetables and herbs", 
            "description": "Reduce the heat and add the 2 onions, 2 carrots, 2 celery sticks, 2 garlic cloves and the leaves from 2-3 sprigs rosemary, all finely chopped, then fry for 10 mins.", 
            "dependsOnSteps": [2],
            "dependsOnIngredients": [3, 4, 5, 6, 7] 
        },
        { 
            "id": 4, 
            "name": "Brown the mince", 
            "description": "Increase the heat to medium-high, add 500g beef mince and cook stirring for 3-4 mins until the meat is browned all over.", 
            "dependsOnSteps": [3],
            "dependsOnIngredients": [8] 
        },
        { 
            "id": 5, 
            "name": "Add bolognese sauce ingredients", 
            "description": "Add 2 tins plum tomatoes, the finely chopped leaves from ¾ small pack basil, 1 tsp dried oregano, 2 bay leaves, 2 tbsp tomato purée, 1 beef stock cube, 1 deseeded and finely chopped red chilli (if using), 125ml red wine and 6 halved cherry tomatoes. Stir well.", 
            "dependsOnSteps": [4],
            "dependsOnIngredients": [9, 10, 11, 12, 13, 14, 15, 16, 17] 
        },
        { 
            "id": 6, 
            "name": "Simmer the sauce", 
            "description": "Bring to the boil, reduce to a gentle simmer and cover with a lid. Cook for 1 hr 15 mins stirring occasionally, until you have a rich, thick sauce.", 
            "dependsOnSteps": [5],
            "dependsOnIngredients": [] 
        },
        { 
            "id": 7, 
            "name": "Add parmesan and season", 
            "description": "Add the 75g grated parmesan, check the seasoning and stir.", 
            "dependsOnSteps": [6],
            "dependsOnIngredients": [18] 
        },
        { 
            "id": 8, 
            "name": "Cook spaghetti", 
            "description": "When the bolognese is nearly finished, cook 400g spaghetti following the pack instructions.", 
            "dependsOnSteps": [7],
            "dependsOnIngredients": [19] 
        },
        { 
            "id": 9, 
            "name": "Serve", 
            "description": "Drain the spaghetti and either stir into the bolognese sauce, or serve the sauce on top. Serve with more grated parmesan, the remaining basil leaves and crusty bread, if you like.", 
            "dependsOnSteps": [8],
            "dependsOnIngredients": [10, 18, 20] 
        }
    ]
}

This is something we can easily display on a graph:

If we aren’t satisfied with the outcome, we can also use a follow-up prompt to mutate the structure. If you look at the recipe, step 8 (cook spaghetti) is independent of the previous steps.

Let’s ask GPT to change that:

Step 8 doesn't depend on any of the previous steps and can be done in parallel. Change the resulting JSON to reflect that
{
    "ingredients": [
        { "id": 1, "name": "olive oil", "quantity": "1 tbsp" },
        { "id": 2, "name": "smoked streaky bacon", "quantity": "4 rashers" },
        { "id": 3, "name": "medium onions", "quantity": "2" },
        { "id": 4, "name": "carrots", "quantity": "2" },
        { "id": 5, "name": "celery sticks", "quantity": "2" },
        { "id": 6, "name": "garlic cloves", "quantity": "2" },
        { "id": 7, "name": "rosemary sprigs", "quantity": "2-3" },
        { "id": 8, "name": "beef mince", "quantity": "500g" },
        { "id": 9, "name": "plum tomatoes", "quantity": "2 x 400g tins" },
        { "id": 10, "name": "basil", "quantity": "small pack" },
        { "id": 11, "name": "dried oregano", "quantity": "1 tsp" },
        { "id": 12, "name": "bay leaves", "quantity": "2" },
        { "id": 13, "name": "tomato purée", "quantity": "2 tbsp" },
        { "id": 14, "name": "beef stock cube", "quantity": "1" },
        { "id": 15, "name": "red chilli", "quantity": "1 (optional)" },
        { "id": 16, "name": "red wine", "quantity": "125ml" },
        { "id": 17, "name": "cherry tomatoes", "quantity": "6" },
        { "id": 18, "name": "parmesan", "quantity": "75g" },
        { "id": 19, "name": "spaghetti", "quantity": "400g" },
        { "id": 20, "name": "crusty bread", "quantity": "to serve (optional)" }
    ],
    "steps": [
        { 
            "id": 1, 
            "name": "Prepare the saucepan", 
            "description": "Put a large saucepan on a medium heat and add 1 tbsp olive oil.", 
            "dependsOnSteps": [],
            "dependsOnIngredients": [1] 
        },
        { 
            "id": 2, 
            "name": "Cook bacon", 
            "description": "Add 4 finely chopped bacon rashers and fry for 10 mins until golden and crisp.", 
            "dependsOnSteps": [1],
            "dependsOnIngredients": [2] 
        },
        { 
            "id": 3, 
            "name": "Add vegetables and herbs", 
            "description": "Reduce the heat and add the 2 onions, 2 carrots, 2 celery sticks, 2 garlic cloves and the leaves from 2-3 sprigs rosemary, all finely chopped, then fry for 10 mins.", 
            "dependsOnSteps": [2],
            "dependsOnIngredients": [3, 4, 5, 6, 7] 
        },
        { 
            "id": 4, 
            "name": "Brown the mince", 
            "description": "Increase the heat to medium-high, add 500g beef mince and cook stirring for 3-4 mins until the meat is browned all over.", 
            "dependsOnSteps": [3],
            "dependsOnIngredients": [8] 
        },
        { 
            "id": 5, 
            "name": "Add bolognese sauce ingredients", 
            "description": "Add 2 tins plum tomatoes, the finely chopped leaves from ¾ small pack basil, 1 tsp dried oregano, 2 bay leaves, 2 tbsp tomato purée, 1 beef stock cube, 1 deseeded and finely chopped red chilli (if using), 125ml red wine and 6 halved cherry tomatoes. Stir well.", 
            "dependsOnSteps": [4],
            "dependsOnIngredients": [9, 10, 11, 12, 13, 14, 15, 16, 17] 
        },
        { 
            "id": 6, 
            "name": "Simmer the sauce", 
            "description": "Bring to the boil, reduce to a gentle simmer and cover with a lid. Cook for 1 hr 15 mins stirring occasionally, until you have a rich, thick sauce.", 
            "dependsOnSteps": [5],
            "dependsOnIngredients": [] 
        },
        { 
            "id": 7, 
            "name": "Add parmesan and season", 
            "description": "Add the 75g grated parmesan, check the seasoning and stir.", 
            "dependsOnSteps": [6],
            "dependsOnIngredients": [18] 
        },
        { 
            "id": 8, 
            "name": "Cook spaghetti", 
            "description": "When the bolognese is nearly finished, cook 400g spaghetti following the pack instructions.", 
            "dependsOnSteps": [],
            "dependsOnIngredients": [19] 
        },
        { 
            "id": 9, 
            "name": "Serve", 
            "description": "Drain the spaghetti and either stir into the bolognese sauce, or serve the sauce on top. Serve with more grated parmesan, the remaining basil leaves and crusty bread, if you like.", 
            "dependsOnSteps": [7, 8],
            "dependsOnIngredients": [10, 18, 20] 
        }
    ]
}

As the previous response is a part of the follow-up prompt, GPT does a pretty good job at keeping the unchanged elements stable.

Use case - build a BPMN specification of a business process

Let’s expand this further - this approach allows us to generate a BPMN definition of a business process, which can subsequently be used by a tool like Camunda.

Let’s try the following prompt:

Generate camunda xml file that describes a process delimited by three backticks 

\`
${processDescription}
\`

Respond with the XML only. The XML should contain process definition and a diagram of the process. The XML should be a valid camunda xml file and should be parsable by its modeller tool. There should be nothing except the xml in the response.

Where process description could be for example:

After sending that to GPT, we’ll get the following output:

<?xml version="1.0" encoding="UTF-8"?>
<definitions xmlns="http://www.omg.org/spec/BPMN/20100524/MODEL"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xmlns:camunda="http://camunda.org/schema/1.0/bpmn"
             xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI"
             xmlns:dc="http://www.omg.org/spec/DD/20100524/DC"
             xmlns:di="http://www.omg.org/spec/DD/20100524/DI"
             xsi:schemaLocation="http://www.omg.org/spec/BPMN/20100524/MODEL BPMN20.xsd"
             id="Definitions_1"
             targetNamespace="http://bpmn.io/schema/bpmn">

  <process id="invoiceProcess" name="Invoice Processing" isExecutable="true">
    <startEvent id="startEvent" name="Invoice Received">
      <outgoing>Flow_1</outgoing>
    </startEvent>

    <sequenceFlow id="Flow_1" sourceRef="startEvent" targetRef="parseInvoice" />

    <serviceTask id="parseInvoice" name="Parse Invoice with OCR" camunda:type="external" camunda:topic="parseInvoice">
      <incoming>Flow_1</incoming>
      <outgoing>Flow_2</outgoing>
    </serviceTask>

    <sequenceFlow id="Flow_2" sourceRef="parseInvoice" targetRef="reviewByAccountant" />

    <userTask id="reviewByAccountant" name="Review by Accountant">
      <incoming>Flow_2</incoming>
      <outgoing>Flow_3</outgoing>
    </userTask>

    <sequenceFlow id="Flow_3" sourceRef="reviewByAccountant" targetRef="managerApproval" />

    <exclusiveGateway id="managerApproval" name="Manager Approval">
      <incoming>Flow_3</incoming>
      <outgoing>Flow_4</outgoing>
      <outgoing>Flow_5</outgoing>
    </exclusiveGateway>

    <sequenceFlow id="Flow_4" sourceRef="managerApproval" targetRef="sendToAccountPayable" name="Approved">
      <conditionExpression xsi:type="tFormalExpression">\${approved}</conditionExpression>
    </sequenceFlow>

    <sequenceFlow id="Flow_5" sourceRef="managerApproval" targetRef="sendToAccounting" name="Not Approved">
      <conditionExpression xsi:type="tFormalExpression">not(\${approved})</conditionExpression>
    </sequenceFlow>

    <userTask id="sendToAccountPayable" name="Send to Account Payable">
      <incoming>Flow_4</incoming>
      <outgoing>Flow_6</outgoing>
    </userTask>

    <userTask id="sendToAccounting" name="Send to Accounting for Clarification">
      <incoming>Flow_5</incoming>
      <outgoing>Flow_7</outgoing>
    </userTask>

    <endEvent id="endEvent" name="Process End">
      <incoming>Flow_6</incoming>
      <incoming>Flow_7</incoming>
    </endEvent>

    <sequenceFlow id="Flow_6" sourceRef="sendToAccountPayable" targetRef="endEvent" />
    <sequenceFlow id="Flow_7" sourceRef="sendToAccounting" targetRef="endEvent" />

  </process>

  <bpmndi:BPMNDiagram id="BPMNDiagram_1">
    <bpmndi:BPMNPlane id="BPMNPlane_1" bpmnElement="invoiceProcess">
      <bpmndi:BPMNShape id="StartEvent_1_di" bpmnElement="startEvent">
        <dc:Bounds x="173" y="102" width="36" height="36" />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNEdge id="Flow_1_di" bpmnElement="Flow_1">
        <di:waypoint x="209" y="120" />
        <di:waypoint x="261" y="120" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNShape id="ServiceTask_1_di" bpmnElement="parseInvoice">
        <dc:Bounds x="261" y="80" width="100" height="80" />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNEdge id="Flow_2_di" bpmnElement="Flow_2">
        <di:waypoint x="361" y="120" />
        <di:waypoint x="411" y="120" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNShape id="UserTask_1_di" bpmnElement="reviewByAccountant">
        <dc:Bounds x="411" y="80" width="100" height="80" />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNEdge id="Flow_3_di" bpmnElement="Flow_3">
        <di:waypoint x="511" y="120" />
        <di:waypoint x="561" y="120" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNShape id="ExclusiveGateway_1_di" bpmnElement="managerApproval" isMarkerVisible="true">
        <dc:Bounds x="561" y="95" width="50" height="50" />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNEdge id="Flow_4_di" bpmnElement="Flow_4">
        <di:waypoint x="611" y="120" />
        <di:waypoint x="661" y="120" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNEdge id="Flow_5_di" bpmnElement="Flow_5">
        <di:waypoint x="586" y="145" />
        <di:waypoint x="586" y="195" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNShape id="UserTask_2_di" bpmnElement="sendToAccountPayable">
        <dc:Bounds x="661" y="80" width="100" height="80" />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNShape id="UserTask_3_di" bpmnElement="sendToAccounting">
        <dc:Bounds x="536" y="195" width="100" height="80" />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNEdge id="Flow_6_di" bpmnElement="Flow_6">
        <di:waypoint x="761" y="120" />
        <di:waypoint x="811" y="120" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNEdge id="Flow_7_di" bpmnElement="Flow_7">
        <di:waypoint x="586" y="275" />
        <di:waypoint x="586" y="325" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNShape id="EndEvent_1_di" bpmnElement="endEvent">
        <dc:Bounds x="811" y="102" width="36" height="36" />
      </bpmndi:BPMNShape>
    </bpmndi:BPMNPlane>
  </bpmndi:BPMNDiagram>
</definitions>

Looking good!

Remarkable insights:

GPT seems to omit some of the XMLNS references quite often. This can be easily fixed in post-processing, which may be way more cost-effective than regenerating the response.
I once got a response where the diagram definition looked like the following:

<bpmndi:BPMNPlane id="BPMNPlane_1" bpmnElement="invoiceProcess">
    <!-- Diagram layout omitted for brevity -->
</bpmndi:BPMNPlane>

Use case - build dynamic forms in React

Let’s create a prompt that:

Specifies that we would like to use the schema of react-jsonschema-form
Include the user's definition of the ideal form
Will specify some additional criteria for the output (i.e. make sure that we only get a JSON as a response)

Generate a JSON definition that follows the schema of react-jsonschema-form. Use the following form that is delimited by three backticks

\`
${userInput}
\`

Respond with the JSON only and don't include any additional characters. Make sure that the output is a valid JSON and that it passes validation with the @rjsf/validator-ajv8 validator

Where userInput could be:

The form should allow a customer to provide an order number (starting with letter R followed by 9 digits), customer's phone number (limited only to country code and digits) and a customer support request. There should also be a field for selecting the type of the issue from a list of the following options: order missing, change in order, return request. At the end of the form, there should be a checkbox labeled "I agree to the privacy policy of company X"

In this case GPT gets back to us with

{
  "title": "Customer Support Form",
  "type": "object",
  "required": [
    "orderNumber",
    "phoneNumber",
    "supportRequest",
    "issueType",
    "privacyPolicyAgreement"
  ],
  "properties": {
    "orderNumber": {
      "type": "string",
      "title": "Order Number",
      "pattern": "R[0-9]{9}"
    },
    "phoneNumber": {
      "type": "string",
      "title": "Phone Number",
      "pattern": "\\+[0-9]+"
    },
    "supportRequest": {
      "type": "string",
      "title": "Customer Support Request"
    },
    "issueType": {
      "type": "string",
      "title": "Type of Issue",
      "enum": [
        "order missing",
        "change in order",
        "return request"
      ]
    },
    "privacyPolicyAgreement": {
      "type": "boolean",
      "title": "I agree to the privacy policy of company X",
      "const": true
    }
  }
}

We can quickly verify it in react-jsonschema-form’s playground: https://rjsf-team.github.io/react-jsonschema-form/

As you can see, it correctly generated a form, together with relevant validations. As with previous examples, we can make modifications by follow-up prompts.

Let’s try making the privacy policy field selectable by default. Sending the following prompt:

The privacy policy agreement should be checked by default

Will result in a new JSON definition of the form:

{
  "title": "Customer Support Form",
  "type": "object",
  "required": ["orderNumber", "phoneNumber", "supportRequest", "issueType"],
  "properties": {
    "orderNumber": {
      "type": "string",
      "title": "Order Number",
      "pattern": "R[0-9]{9}"
    },
    "phoneNumber": {
      "type": "string",
      "title": "Phone Number",
      "pattern": "\\+[0-9]+"
    },
    "supportRequest": {
      "type": "string",
      "title": "Customer Support Request"
    },
    "issueType": {
      "type": "string",
      "title": "Type of Issue",
      "enum": ["order missing", "change in order", "return request"]
    },
    "privacyPolicyAgreement": {
      "type": "boolean",
      "title": "I agree to the privacy policy of company X",
      "default": true
    }
  }
}

Which looks like that:

As you can see, the checkbox is now selected by default. Similar to that, we can make further tweaks to the existing form.

Use case - build dynamic UIs

To start with, let’s define some components. We’ll use react-bootstrap to simplify the UI work needed.

Let’s make a wrapper component for the whole app:

function AppRoot({ children }) {
  return (
    <>
      {children}
    </>
  )
}

export default AppRoot

Then a basic layout component:

import { Container } from 'react-bootstrap';

function AppMainLayout({ children }) {
  return (
    <Container>
      {children}
    </Container>
  )
}

export default AppMainLayout

A customizable navbar:

import Container from 'react-bootstrap/Container';
import Nav from 'react-bootstrap/Nav';
import Navbar from 'react-bootstrap/Navbar';

function AppNavbar({ title, links }) {
  return (
    <Navbar bg="primary" data-bs-theme="dark">
    <Container>
      <Navbar.Brand href="/">{title}</Navbar.Brand>
      <Nav className="me-auto">
        {links && links.map((link) => {
          return <Nav.Link href={link.url} key={link.title}>{link.title}</Nav.Link>
        })}
      </Nav>
    </Container>
  </Navbar>
  )
}

export default AppNavbar;

A toolbar with some action buttons (that we can wire together later):

import Button from 'react-bootstrap/Button';

function AppToolbar() {
  return (
    <div className="py-2">
      <Button variant="primary" className="mx-1">New user</Button>
      <Button variant="secondary" className="mx-1">Refresh</Button>
      <Button variant="secondary" className="mx-1">Deactivate</Button>
      <Button variant="secondary" className="mx-1">Reset password</Button>
    </div>
  )
}

export default AppToolbar

And a table for displaying a list of users:

import Table from 'react-bootstrap/Table'

function AppUserList() {
  return (
    <Table striped bordered hover>
      <thead>
        <tr>
          <th>#</th>
          <th>Email</th>
          <th>First Name</th>
          <th>Last Name</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <td>1</td>
          <td>test@example.com</td>
          <td>John</td>
          <td>Doe</td>
        </tr>
        <tr>
          <td>2</td>
          <td>jack@example.com</td>
          <td>Jack</td>
          <td>Dummy</td>
        </tr>
        <tr>
          <td>3</td>
          <td>thomas@example.com</td>
          <td>Tomas</td>
          <td>Jonas</td>
        </tr>
        <tr>
          <td>4</td>
          <td>paul@example.com</td>
          <td>Paul</td>
          <td>Test</td>
        </tr>
      </tbody>
    </Table>
  )
}

export default AppUserList
view raw

Then connect it all together in our web application:

import ReactDOM from 'react-dom/client';
import reportWebVitals from './reportWebVitals';
import ReactJsonSchema from 'react-json-schema';
import AppNavbar from './components/AppNavbar'
import AppRoot from './components/AppRoot'
import AppMainLayout from './components/AppMainLayout'
import AppToolbar from './components/AppToolbar'
import AppUserList from './components/AppUserList'

const uiRenderer = new ReactJsonSchema();
uiRenderer.setComponentMap({ AppRoot, AppNavbar, AppMainLayout, AppToolbar, AppUserList });

const uiSchema = await fetchGPTResponse(prompt);

const root = ReactDOM.createRoot(document.getElementById('root'));
root.render(
  uiRenderer.parseSchema(uiSchema)
);

It’s easy:

Initialize react-json-schema parser
Configure it to recognize our custom components
Prompt GPT for a schema based on user’s description
Render the schema that GPT responded with

Now it’s time to define the actual prompt to be used:

Generate a JSON that follows the react-json-schema format for the following set of components and their props delimited by three backticks:
\`
AppRoot (children: Node[]) , AppNavbar (title: string, links: Link[]), AppMainLayout (children: Node[]) , AppToolbar (), AppUserList (), h1 (text: string), h2 (text: string)
\`
The following custom types are defined in the application
\`
type Link = {
title: string;
url: string;
}
\`

The JSON should build a UI following a specification delimited by three backticks
\`
The UI should start with a navbar, a title specifying that this is a dashboard for managing users. Below the title there should be a more detailed description of the dashboard. The main element of the UI should be a table with a list of users. Above the table there should be a toolbar allowing to modify the list of users.
\`
The UI must be wrapped in the AppRoot component. Specify the props directly in the same object as the component, without an additional props object. Only use the props that are defined above. The output should be a valid react-json-schema specification. Don't output any additional characters.

A sample GPT response will look like:

{
  "component": "AppRoot",
  "children": [
    {
      "component": "AppNavbar",
      "title": "User Management Dashboard",
      "links": [
        {
          "title": "Home",
          "url": "/"
        },
        {
          "title": "Settings",
          "url": "/settings"
        }
      ]
    },
    {
      "component": "AppMainLayout",
      "children": [
        {
          "component": "h1",
          "text": "User Management Dashboard"
        },
        {
          "component": "h2",
          "text": "Manage and oversee user accounts and permissions"
        },
        {
          "component": "AppToolbar",
          "children": []
        },
        {
          "component": "AppUserList",
          "children": []
        }
      ]
    }
  ]
}

Use case - querying an ElasticSearch index

ElasticSearch is commonly used for building indexes of products or application logs. We can use GPT to translate user’s queries to an ElasticSearch query.

The approach would be:

Specify that we’d like to get an ElasticSearch query as a result
Provide information about the structure of the index
Provide information about the user's query
Specify additional criteria for the output

Example: We have an ElasticSearch index with the following definition:

{
  "products_production": {
    "aliases": {},
    "mappings":{
      "spree_product": {
        "properties": {
          "available_on": {
            "type":"date",
            "format":"dateOptionalTime"
          },
          "brand_id": {
            "type":"keyword"
          },
          "created_at": {
            "type":"date",
            "format":"dateOptionalTime"
          },
          "description": {
            "type":"text",
            "analyzer":"snowball"
          },
          "discontinue_on": {
            "type":"date",
            "format":"dateOptionalTime"
          },
          "featured": {
            "type":"boolean"
          },
          "name": {
            "type":"text",
            "boost":100.0,
            "fields": {
              "untouched": {
                "type":"keyword"
              }
            },
            "analyzer":"snowball"
          },
          "price": {
            "type":"double"
          },
          "sku": {
            "type":"keyword"
          },
          "suggestion_terms": {
            "type":"text",
            "analyzer":"simple"
          },
          "taxon_ids": {
            "type":"keyword"
          }
        }
      }
    },
    "settings": {
      "index": {
        "number_of_shards":"1",
        "provided_name":"products_production",
        "creation_date":"1684177003621",
        "analysis": {
          "filter": {
            "nGram_filter": {
              "token_chars": [
                "letter",
                "digit",
                "punctuation",
                "symbol"
              ],
              "min_gram": "3",
              "type": "nGram",
              "max_gram": "4"
            }
          },
          "analyzer": {
            "nGram_analyzer": {
              "filter":[
                "lowercase",
                "asciifolding",
                "nGram_filter"
              ],
              "type": "custom",
              "tokenizer": "whitespace"
            },
            "whitespace_analyzer": {
              "filter": [
                "lowercase",
                "asciifolding"
              ],
              "type": "custom",
              "tokenizer": "whitespace"
            }
          }
        },
        "number_of_replicas": "1",
        "uuid": "WV8HWbS_QsudGKL223hwrA",
        "version": {
          "created": "6080091"
        }
      }
    }
  }
}
Create an elasticsearch query json that operates on an index with the following definition delimited by three backticks
\`
${indexDefinition}
\`

The query should return the results described here delimited by three backticks:
\`
Return the list of the most expensive products that are currently available, aren't featured and don't have any taxons associated with them.
\`

Return only a JSON and make sure it can be interpreted by an elasticsearch 6.8 instance

{
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "available_on": {
              "lte": "now"
            }
          }
        },
        {
          "term": {
            "featured": false
          }
        },
        {
          "bool": {
            "must_not": {
              "exists": {
                "field": "taxon_ids"
              }
            }
          }
        }
      ]
    }
  },
  "sort": [
    {
      "price": {
        "order": "desc"
      }
    }
  ]
}

We can now pass it to ElasticSearch and get the results:

{
    "took": 51,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": 435,
        "max_score": null,
        "hits": [
            {
                "_index": "products_production",
                "_type": "product",
                "_id": "1694",
                "_score": null,
                "_source": {
                    "name": "Asus ROG Zephyrus S Ultra Slim Gaming Laptop, 15.6\u0026quot; 144Hz IPS Type FHD, GeForce RTX 2070, Intel Core i7-9750H, 16GB DDR4, 512GB PCIe Nvme SSD, Aura Sync RGB, Windows 10, GX531GW-AB76",
                    "description": "\u003cp\u003eNvidia GeForce RTX 2070 8GB DDR6 (base: 1080 MHz, Boost: 1305 MHz; TDP: 90W). 230W Power Adaptor. Support Type-c PD 3.0 up to 65W Latest Intel Core i7-9750h Hexa-Core processor; Bluetooth 5.0 15.6\u0026rdquo; 144Hz 3ms IPS-Type full HD (1920x1080) Display 16GB 2666Hz DDR4 | 512GB PCIe NVMe SSD | Windows 10 Home 0.62\u0026rdquo; thin, 4.6 lbs ultraportable military-grade magnesium alloy body with premium metal cover Rog active Aerodynamic System (AAs) with upgraded 12V fans and anti-dust tunnels to preserve cooling performance and system stability Gigabit wave 2 Wi-Fi 5 (802.11AC) | customizable 4-zone ASUS Aura RGB Gaming Keyboard The all new ROG Zephyrus s gaming Laptop is redefining ultra-slim gaming laptops yet again with innovative engineering to cool Its 9th Gen Intel Core i7 processor and GeForce RTX Max-Q graphics, so you can immerse yourself in its no-compromise 144Hz/3ms Display. *The actual transfer speed of USB 3.0, 3.1 (Gen 1 and 2), and/or Type-C will vary depending on many factors including the processing speed of the host device, file attributes and other factors related to system configuration and your operating environment.\u003c/p\u003e",
                    "taxon_ids": [],
                    "brand_id": null,
                    "available_on": "2019-12-02T00:00:00.000Z",
                    "discontinue_on": null,
                    "featured": false,
                    "price": "1595.73",
                    "created_at": "2019-12-02T20:07:26.594Z",
                    "suggestion_terms": ["Asus ROG Zephyrus S Ultra Slim Gaming Laptop, 15.6\u0026quot; 144Hz IPS Type FHD, GeForce RTX 2070, Intel Core i7-9750H, 16GB DDR4, 512GB PCIe Nvme SSD, Aura Sync RGB, Windows 10, GX531GW-AB76"]
                },
                "sort": [1595.73]
            },  
            {
                "_index": "products_production",
                "_type": "product",
                "_id": "1470",
                "_score": null,
                "_source": {
                    "name": "Weber - Genesis II S-435 62006001 Stainless Steel",
                    "description": "\u003cp\u003eA grill that has it all. A powerful grilling engine, impressive features, and an outstanding 10 year warranty on all parts of the grill to back it up. The immense space and sear station will execute as many steaks as you need to perfection, while the side burner saut\u0026eacute;s veggies. Premium stainless steel cooking grates retain even heat and are easy to clean. The Tuck-Away warming rack keeps food warm while the main dish grills underneath. And between cookouts, store your grilling tools and accessories in the enclosed grill cabinet. Those are just a few features that set the Genesis II S-435 apart from your typical grill.\u003c/p\u003e\r\n\u003cul\u003e\r\n\u003cli\u003ePropane Grill fueled by propane\u003c/li\u003e\r\n\u003cli\u003eElectric Ignition Electronic ignition makes it easy to start\u003c/li\u003e\r\n\u003cli\u003eSide Burner Side burner for cooking versatility.\u003c/li\u003e\r\n\u003cli\u003e9,000 BTU-per-hour input Sear Station burner\u003c/li\u003e\r\n\u003cli\u003eStainless steel heat deflectors\u003c/li\u003e\r\n\u003cli\u003eStainless steel lid with thermometer\u003c/li\u003e\r\n\u003cli\u003eEnclosed cabinet with stainless steel doors\u003c/li\u003e\r\n\u003cli\u003eSix tool hooks\u003c/li\u003e\r\n\u003cli\u003eTwo heavy-duty front locking swivel casters and two large wheels\u003c/li\u003e\r\n\u003c/ul\u003e",
                    "sku": "62006001",
                    "taxon_ids": [],
                    "brand_id": null,
                    "available_on": "2019-11-03T00:00:00.000Z",
                    "discontinue_on" :"2019-11-04T00:00:00.000Z", 
                    "featured": false,
                    "price": "1315.5",
                    "created_at": "2019-11-03T19:02:48.694Z",
                    "suggestion_terms": ["Weber - Genesis II S-435 62006001 Stainless Steel"]
                },
                "sort" :[1315.5]
            },
            // more results here
        ]
    }
}

Follow-up use case - building a query in a query language that the user may not know

Use case - data visualization

Let’s take a sample CSV file from Vega-Lite’s demo that contains the stock prices history of a few tech companies. It will be easy for us to reference it on the playground.

The file is a regular CSV that contains a few columns:

symbol,date,price
MSFT,Jan 1 2000,39.81
MSFT,Feb 1 2000,36.35
MSFT,Mar 1 2000,43.22

Let’s try to generate some charts that utilize this structure.

Generate a JSON using vega-lite schema that takes input data from file data/stocks.csv. The visualization should work on a dataset with the following columns delimited by three backticks
\`
symbol date price
\`
The visualization should result in the following result delimited by three backticks
\`
I would like to get a line chart of each of the stocks over time. There should also be a second line chart that shows only MSFT stock. Finally, I would like to see a pie chart of the average stock prices for each symbol
\`

Return only the resulting JSON and don't include any additional characters. Make sure that the visualization conforms to the vega-lite schema.

After trying it out with GPT we get the following definition

{
  "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
  "data": {"url": "data/stocks.csv"},
  "vconcat": [
    {
      "title": "Stock Prices Over Time",
      "mark": "line",
      "encoding": {
        "x": {"field": "date", "type": "temporal", "title": "Date"},
        "y": {"field": "price", "type": "quantitative", "title": "Price"},
        "color": {"field": "symbol", "type": "nominal", "title": "Symbol"}
      }
    },
    {
      "title": "MSFT Stock Price Over Time",
      "transform": [{"filter": "datum.symbol === 'MSFT'"}],
      "mark": "line",
      "encoding": {
        "x": {"field": "date", "type": "temporal", "title": "Date"},
        "y": {"field": "price", "type": "quantitative", "title": "Price"}
      }
    },
    {
      "title": "Average Stock Prices",
      "transform": [{"aggregate": [{"op": "mean", "field": "price", "as": "average_price"}], "groupby": ["symbol"]}],
      "mark": "arc",
      "encoding": {
        "theta": {"field": "average_price", "type": "quantitative", "title": "Average Price"},
        "color": {"field": "symbol", "type": "nominal", "title": "Symbol"}
      }
    }
  ]
}

Which gives us the desired set of charts

There’s also an interesting project on Github that showcases a similar concept. Their base prompt looks slightly differently:

You are a great assistant at vega-lite visualization creation. No matter what the user asks, you should always respond with a valid vega-lite specification in JSON.

You should create the vega-lite specification based on user's query.

Besides, Here are some requirements:
1. Do not contain the key called 'data' in vega-lite specification.
2. If the user ask many times, you should generate the specification based on the previous context.
3. You should consider to aggregate the field if it is quantitative and the chart has a mark type of react, bar, line, area or arc.
4. Consider to use bin for field if it is a chart like heatmap or histogram.
5. The available fields in the dataset and their types are:
${metas
    .map((field) => `${field.name} (${field.semanticType})`)
    .join(", ")}

Good practices - learn from the Upside team's experiences

You can apply the same method with any JSON schema you create, like for a spaghetti recipe.

Complexity - try not to bite more than GPT can chew‍

Performance - being patient is the key‍

Cost - manage your expenses

Reliability - fine-tuning for diverse inputs

Prompt injection risks - always double check

Summary

GPT does a pretty good job in parsing data. Basing on the experiments described in the post, we can define a simple framework that can handle diverse input and output formats:

Specify the desired schema
Specify the details of available fields (when needed)
Specify the input data
Reinforce important details (e.g. that the response should be a JSON and only a JSON)

Building an agency site to showcase services and publish blogs. Is it better to code it from scratch for full control, or use a customizable template for a faster launch? Looking for perspectives from others who’ve faced this decision.

Chris Yaowen Zhang — Tue, 24 Jun 2025 15:16:14 +0000