DEV Community: Upside

🌍 Scaling Globally with Spree? Here's How to Automate Translations (Using DeepL + 1 Line of Code)

Chris Yaowen Zhang — Wed, 23 Jul 2025 13:11:47 +0000

Manually translating product catalogs? Not only is it tedious — it slows down your entire international expansion.

The good news? With Spree 4.6 and a new gem called spree_automation_interfaces, you can plug in translation tools like DeepL and have your products auto-translated with the click of a button.

Let me show you how we got it working in under 20 minutes.

🛒 Problem: Spree supports translations, but the process is still manual

Spree’s backend already allows you to manage product translations across multiple locales. But historically, companies relied on exporting CSVs → emailing translators → re-importing the files — a messy, slow, error-prone loop.

Even with tools like Google Translate or DeepL, manual copy-paste just doesn't scale.

So we decided to automate the process, while still keeping room for human review.

🔧 Solution: Spree Automation Interfaces + DeepL

As a part of supporting Spree users, we at Upside published spree_automation_interfaces that lets you connect automated translation services directly to your Spree store.

In this guide, we’ll show you how to:

Set up your store for multiple languages (French & German)
Plug in DeepL using a custom provider
Add an “Auto-Translate” button inside your admin panel

Step 1: Enable Multiple Languages in Spree
(You can skip this if your store already supports multiple locales)

In the Spree Admin Panel, go to Settings → Store in the sidebar to the left.
You will see a list of "Supported Locales". For the purpose of this guide, we'll add set up additional locales: French and German.

Click Update.
To confirm that the setup works correctly, navigate to a product in the admin panel, and go to its "Translations" tab. You should see a table for filling additional translations - this means that the setup is correct.

🎉 At this point, you can provide the translations manually. It's time to automate the process.

Step 2: Add the Right Gems
We need to add the following gems:

spree_automation_interfaces - that brings the capability of automated translations into Spree
deface - to allow spree_automation_interfaces to add new features to the admin panel
deepl-rb - which we will use as a translations provider

gem 'spree_automation_interfaces', github: 'spree-contrib/spree_automation_interfaces'
gem 'deface'
gem 'deepl-rb', require: 'deepl'

Then run:

bundle install

And restart your development server.

Step 3: Implement the DeepL Translations Provider

Now, we need to provide an implementation for the Automation Interfaces, that will be called by Spree when the user requests translations to be created.

The required interface is defined as follows:

module Spree
  module Products
    module Translations
      interface _AutomatedTranslationsProvider
        def call: (source_attributes: Hash[String, String], source_locale: String, target_locale: String) -> Hash[Symbol | String, String]
      end
    end
  end
end

Let's create an implementation in app/services/spree/translations/deepl_translations_provider.rb

module Spree
  module Translations
    class DeeplTranslationsProvider
      def call(source_attributes:, source_locale:, target_locale:)
        name = source_attributes['name']
        description = source_attributes['description']
        translated_name, translated_description = DeepL.translate([name, description], nil, target_locale)
        {
          'name' => translated_name,
          'description' => translated_description
        }
      end
    end
  end
end

Step 4: Initialize DeepL Client
We also need to initialize DeepL's client. Let's create the following initializer config/initalizers/deepl.rb

DeepL.configure do |config|
  config.host = ENV.fetch('DEEPL_HOST', 'https://api-free.deepl.com')
  config.auth_key = ENV.fetch('DEEPL_API_KEY')
end

Step 5: Plug Your Provider into Spree

Finally, configure Spree to use the provider we've just created

In config/initializers/spree_automation_interfaces.rb

Rails.application.config.after_initialize do  
  SpreeAutomationInterfaces::Dependencies.products_automated_translations_provider = 'Spree::Translations::DeeplTranslationsProvider'
end

Step 6: Try It!
Head to any product’s Translations tab in the admin panel.

You’ll see a new button: “Translate automatically”

Once you click it, it will fetch translations using the provider we've just implemented and add them to the product.

Bonus: Go Further with Sidekiq or API Triggers

This interface also works programmatically, so you can:

Trigger translations from code
Run a background job to translate your entire catalog
Schedule daily translation syncs

For example, you could build a Sidekiq worker that batches and translates 10,000 SKUs overnight.

Improving INP: How to Make Your React App Feel Faster

Chris Yaowen Zhang — Wed, 09 Jul 2025 12:52:42 +0000

If your React app feels a little sluggish when users click, type, or scroll—it’s not just a UX issue anymore. Google’s new INP (Interaction to Next Paint) metric measures how quickly your app responds to interactions, and it now plays a role in SEO rankings.

What is the “Interaction to Next Paint” (INP) metric?

Google released a new Core Web Vital called Interaction to Next Paint (INP) in 2024. It measures the speed of interactions with the website.

In this article, we will walk you through INP from the developer’s perspective. This includes guiding you through testing for INP, verifying potential causes of low INP scores, and how you can leverage new React tools to optimize your website for recent changes.

Interaction to Next Paint (INP) is a scoring mechanism that analyzes latency from the time the user begins the interaction to the moment the next frame is presented with visual feedback. We’ll take a closer look at what mechanisms in modern applications cause low INP score later.

It measures for one of the following interactions.

Clicking with a mouse
Tapping on a device with a touchscreen
Pressing a key on either a physical or onscreen keyboard

For example, when the user types a character into the search input that filters a list of cats by their breeds, the INP value will be a duration from the moment of triggering a group of keyboard events (keydown, …, keyup) to the moment of re-rendering the updated list

The main goal of INP is to minimize this duration, ensuring swift visual feedback for user interactions and optimizing overall responsiveness on the page.

What INP isn't?

There is already a well-known metric called “First Input Delay” (FID) which measures the time from when a user first interacts with a page to the time when the main thread is next idle. It offered developers a new way to measure responsiveness as real users experience it.

FID vs INP - What’s the difference?

Comparing these two mechanisms reveals similar measurement goal - a page responsiveness. One of the essential differences is that the FID measures only initial interaction, while the INP takes the worst latency recorded during the entire session.

💡 There's a little nuance to how the INP is calculated depending on the total interactions per website. For websites with more than 50 interactions per session, it takes the 75th percentile of the slowest latency recorded. This eliminates potential hiccups for pages with large numbers of interactions.

Even though first impressions are important, the first interaction is not necessarily representative of all interactions throughout the life of a page. This is one of the reasons why INP replaces the FID.

Should you care about INP?

In short, yes, if you care about SEO or user experience. The INP will be another metric that not only directly influences your website's ranking on Google search results but also provides valuable insights for tracking the responsiveness of your product.

Receiving poor results from the INP might be a sign to profile your application for performance.

Measuring INP

For a released product, that runs in a production environment, accumulates user traffic, and qualifies for inclusion in the Chrome User Experience Report (CrUX) you can quickly measure the new INP metric using PageSpeed insights. It gives a snapshot of your page and origin's performance over the previous 28 days.

Manually diagnose slow interactions

Ideally, you should rely on the field data from the CrUX report, that suggests your website struggles with slow interactions. Sometimes, however, it’s not possible, and we need another way.

The main goal is to identify the problem with lowest amount of effort at first. For manual investigation, we’ll use the Web Vitals addon. Then, dive deeper into what causes the issue. For that, we can leverage the Profiler available in the browser’s DevTools tab.

Using Web Vitals addon (Chrome)
Requirements:

Chrome browser
Web Vitals addon
React Developer Tools addon (Useful for highlighting component updates)

Let’s start with a simple configuration of the Web Vitals addon. This will display HUD overlay as well as log information about each interaction with the page in the console.

We’ll use React Developer Tools to highlight component updates. This will help us with identifying what’s happening in our application.

💡 Once React Developer Tools is installed, open Chrome developer tools, then “Profiler” section. Enable “Highlight updates when components render”.

Let’s go back to previously introduced app which allows us to filter cats by their breeds. First issue we can observe is that the list itself is long. We’re displaying over 1900 results at once. That’s already something we could optimize, but let’s leave it for later. Let’s try to interact with the page by typing “B” into the input, and then clearing it.

Few things are happening here:

A controlled Input component updates immediately on each character we type.
A list updates on each input change.

The moment of “freeze” shown on the video is the main issue that causes INP score to suffer. We can verify that by observing the INP delta times logged in the console on the right. This translates into general poor result of the INP score for our website (344 ms).

What exactly causes poor INP results?

Figuring out what's causing poor INP is the most important, and the most difficult step on the road of enhancing the User Experience (UX).

As Google Chrome’s team suggest, interactions can be broken down into three phases:

The input delay: It starts when the user initiates an interaction with the page, and ends when the event callbacks for the interaction begin to run.
The processing time: It consist of the time it makes for event callbacks to run to completion.
The presentation delay: It’s the time it takes for the browser to present the next frame which contains the visual result of the interaction.

I found a bottleneck - now what? Optimizing your React app for performance.

As a React developer you’re probably aware of core concepts used to optimize the app.

I’d distinguish following:

Memoization memo, useMemo, useCallback
Debouncing/Throttling (using a hook or a higher-order-function that triggers the callback)
List virtualization
List pagination

Let’s go back to our simple app about cats. Take a look at the component that provides browsing functionality:

export function CatsBreedsBrowserTemplate() {
  // 1. Fetching a list of all cats from the API (no limitting the response).
  const { data, isLoading } = useCatsData();

  // 2. Storing a `SearchBar` value.
  const [filterByBreed, setFilterByBreed] = useState("");

  // 3. Filtering the `data` array based on the `filterByBreed`.
  const searchResults = useMemo(() => {
    const catsData = data ?? [];

    if (!filterByBreed || !catsData.length) {
      return catsData;
    }

    return catsData.filter((cat) => cat.breed.includes(filterByBreed));
  }, [data, filterByBreed]);

  // 4. Rendering the list of cats
  const renderCatsSearchResult = useCallback(() => {
    if (isLoading) {
      // 5. Early-return: show loading indicator
      return <CircularProgress />
      );
    }
    if (!searchResults.length) {
      // 6. Early-return: show empty placeholder
      return <p>No cats found :(</p>
    }
    // 7. Return actual list component.
    return <CatsList cats={searchResults} />
  }, [isLoading, searchResults]);

  return (
    <div>
      <SearchBar
        label="Filter by cat's breed"
                // 8. On each search input change, set `filterByBreed` state.
        onChange={(e) => setFilterByBreed(e.target.value)}
      />

      {renderCatsSearchResult()}
    </div>
  );
}

Let’s see how can we utilize already known concepts to optimize the INP time.

Issue #1 - List of cats re-renders on each change of the SearchBar component.

As we know from the React’s Reconciliation model, when state changes, React re-renders the components and all its descendants. By calling setFilterByBreed on each SearchBar change, we perform unnecessary re-renders of the expensive CatsList component.

Instead of triggering onChange immediately, let’s apply debouncing on the SearchBar component.

You can use a library, or implement a simple debounce utility by yourself. Here’s a simple higher-order-function that triggers callback on timeout:

const DEFAULT_DEBOUNCE_DELAY = 500;

export const debounce = <T,>(
  callback: (value: T) => void,
  delay = DEFAULT_DEBOUNCE_DELAY
) => {
  let timer: number;
  return (value: T) => {
    clearTimeout(timer);
    timer = setTimeout(() => callback(value), delay);
  };
};

A SearchBar component is a simple wrapper around mui’s TextField component:

export interface SearchBarProps extends Partial<TextFieldProps<'standard'>> {}

function SearchBar({ onChange, ...rest }: SearchBarProps) {
  return <TextField onChange={onChange} {...rest} />;
}

Using debounced onChange:

export interface SearchBarProps extends Partial<TextFieldProps<'standard'>> {}

function SearchBar({ onChange, ...rest }: SearchBarProps) {
  // Re-initializing debounced onChange on `onChange` change (no pun's intented).
  const debouncedOnChange = useMemo(
    () => onChange && debounce(onChange),
    [onChange]
  );

  return <TextField onChange={debouncedOnChange} {...rest} />;
}

This improves the INP, because we no longer try to re-render expensive component on each user interaction.

We still however, render a long list of elements in the DOM, synchronously.

Issue #2- List renders all elements at once causing low INP score by freezing the layout

This time we could:
a) Virtualize the list (by limiting amount of elements attached to the DOM tree, and displaying only such that are visible in the viewport + some offset).
b) Paginate the data (combine intersection-observer with appending next elements to the DOM on user scroll).

Both solutions allow us to keep initial list lightweight, and get rid of the layout “freeze” that blocks us from interacting with the website. There’s a plenty of resources for implementing both Virtualization, and Pagination in React. You can check one of them.

React ≥18, concurrency, and the new mental model of reactivity

Apart from what I already mentioned, there’s a relatively new approach to reactivity in React using its latest features. By introducing concurrency, we’re now able to transition (no pun intended) from well-known optimization techniques, take the responsibility from the developer, and introduce less intrusive optimizations managed by the tool itself (with a bit of our help).

Non-urgent updates

With React < 18, every update is urgent. From React 18, every update is urgent by default, but now we’re able to set a low priority on updates that could potentially block the UI.

Instead of thinking about re-renders, we should start thinking about the urgency of updates we perform.

Let’s once again go back to our unoptimized cats browser (Fig. 1).

export function CatsBreedsBrowserTemplate() {
  ...

  return (
      ...

      <SearchBar
        ...
    // 1. On each search input change, set `filterByBreed` state.
        onChange={(e) => setFilterByBreed(e.target.value)}
      />

      ...
  );
}

Remember the part (Fig. 1 [8], Fig. 5 [1]) where, we update the state of the filterByBreed on each SearchBar change? We’ll utilize the useTransition hook to mark this as non-urgent update.

// 1. Import the `useTransition` hook.
import { useTransition } from "react";

export function CatsBreedsBrowserTemplate() {
  // 2. Declare `useTransition` fields.
  const [isPending, startTransition] = useTransition();

  ...

  return (
      ...

      <SearchBar
        ...
        onChange={(e) =>
          // 3. Mark state update as non-urgent,
          //    by wrapping it with the `startTransition` function.
          startTransition(() => {
            setFilterByBreed(e.target.value);
          })
        }
      />

      ...
  );
}
view raw

(Fig 6)

As simple as that, we were able to eliminate the “freezing” UI issue.

Why this works?‍

Under the hood, react maintains a queue of updates ready to be performed. Before React 18, all updates were performed sequentially until there were no more updates left. When we update the component’s state, React will queue this update, and perform it until it completes. This however, due to the JavaScript event loop’s “Run-to-completion” strategy, blocks processing of other tasks (that’s why we can see the UI “freeze” with non-optimized example).

With the release of React 18, a significant change introduced a new mechanism of processing the updates queue. Instead of forcefully processing an update, React now schedules a unit of update’s work, checks if the browser allows to perform it, and then completes it. Otherwise, it gives the control back to the browser. This way, long-running non-urgent updates no longer blocks the browser’s work that needs to be performed due to the higher priority.

Transition is only one of the concurrent scheduling mechanisms. Make sure to check all already-existing, and incoming hooks such as useDeferredValue, or use.

Is concurrent scheduling a solution to all problems?

While the new concurrency model effectively addresses common challenges, it's not a silver bullet applicable to every aspect of your code.

Since useTransition splits the work in chunks, for expensive-to-render components, splitting the work is impossible. That’s where we need to combine the new concurrent scheduling with already known optimization strategies.

Avoid premature optimization

Understanding the optimization techniques mentioned and comprehending how the INP calculates the Web Vitals score is crucial for delivering a seamless and responsive user experience for your product.

While optimization enhances performance, it often comes with associated costs. While it's tempting to immediately implement efficiency improvements, many enhancements result in higher memory consumption and maintenance challenges.

It is essential to prioritize simplicity in our solutions. I advise vigilant monitoring of your application to identify areas causing issues and optimizing those selectively, rather than prematurely optimizing, which can introduce unnecessary complexity to your project.

Workshop: Build Your Agentic Knowledge Base Using Open-Source Tools

Chris Yaowen Zhang — Wed, 09 Jul 2025 09:19:57 +0000

Hi everyone!

If you're exploring how to build your own internal “Company GPT” that runs on your stack, connects to your private data, and doesn’t compromise on control or security, we’re hosting a live hands-on workshop just for that!

In this workshop, we will use open-source tools, so no black boxes, no vendor lock-in. You’ll set up custom AI agents that can handle repetitive workflows like content generation, product queries, and support requests, all in a few clicks.

Whether you're curious about LLM-based retrieval, self-hosted agents, or just want to see a clean architecture for internal AI, this is a practical deep dive to get you started fast.

We will also show you how to ensure reliable answers with layered validation, so your AI agents stay accurate and trustworthy

By the end of this session, you’ll walk away with:
✅ A fully functional, AI-powered knowledge base
✅ Customizable AI agents trained on your internal content
✅ Ready-to-use code and templates to expand or scale
✅ Live expert guidance (no prior AI experience required)

⏰ Time: 5:00 PM CET / 11:00 AM ET July 16th
📍 Location: Online
✋ Format: Live workshop with Q&A
🎁 Access to source code and practical templates (GitHub link here)
❗️❗️Note: This is an online event, register to get Zoom link here!

Looking forward to seeing you there!

Looking for Senior Frontend Engineer (React/Vue)

Chris Yaowen Zhang — Tue, 08 Jul 2025 08:11:08 +0000

Dear Dev community,

We are hiring a senior frontend engineer. If you are interested, please see below job description or share with people that might be relevant.

Thanks!

Job description

At Upside, we’re not just shipping features - we’re partnering with ambitious companies across the globe to solve complex, meaningful problems through technology. We work in small, focused teams, combining engineering, product thinking, and design, to deliver outcomes, not just output.

Open-source is a strategic part of our work. We build and maintain a number of open-source projects that support our clients and internal R&D. Our latest release is Enthusiast, an AI Agent framework tailored for e-commerce use cases.

Role Overview:

We are seeking a Senior Frontend Engineer to drive the design, development, and maintenance of high-quality platforms for global clients. In this role, you will be a key contributor to our client-facing solutions, ensuring performance, scalability, and reliability. You’ll collaborate within a dynamic team of 3 to 9 developers, working together to craft scalable and resilient systems.

Job requirements

Extensive commercial experience in building web-based applications.
In-depth knowledge of modern frontend stack and knowledge of at least one framework (React or Vue).
Experience with Next.js or other SSR frameworks.
Experience with GraphQL.
Solid FP/OOP skills.
Practical experience with Continuous Integration/Continuous Deployment (CI/CD) practices and modern cloud environments.
Strong consulting mindset - balancing technical excellence with client needs and communicating effectively with both technical and non-technical stakeholders.
Fluent English, with clear written and verbal communication.

Bonus Points if you have:
Experience in advanced web animations (e.g. framer-motion, GSAP, Rive/Lottie).

Experience in advanced web technologies such as WebGL, WebSockets, WebAssembly, web workers, and PWA.

Your responsibilities would be

Developing complex digital platforms - from architecture decisions to implementation.
Work in autonomous teams (3-9 people) delivering mission-critical systems.
Collaborate directly with technology leaders, founders, and domain experts to shape solutions that drive real business outcomes.

What We Offer

Transparent progression path.
Flexible work: office in Kraków, hybrid or remote from anywhere in Poland, with company on-sites.
Additional paid days off beyond national holidays.
A deeply collaborative, engineering-led company culture.
Support for ongoing learning.

Salary:

Senior Engineer: 20 800 - 25 900 PLN net + VAT
Lead Engineer: >26 500 PLN net + VAT
Including additional days off

Location:
Office in Kraków, hybrid, fully-remote.

Apply via THIS LINK!

Rendering HEIC on the Web: How to Make Your Web App Handle iPhone Photos

Chris Yaowen Zhang — Wed, 02 Jul 2025 15:50:06 +0000

App developers, your iPhone users are uploading photos, and it’s breaking your app!

That’s because Apple’s default image format, HEIC, isn’t supported by most browsers.

We've seen this issue firsthand with quite a few clients that have iOS-heavy user bases. You might not notice it during testing, but the moment a real user uploads a photo from their iPhone, things fall apart.

In this post, we’ll walk through why it breaks your app, and how to fix it—with practical options for frontend and backend conversion, plus third-party tools. Alright, let's dive in!

First, let’s render it on our site 🔥

<img src="/assets/dog.heic" alt="">

Yeahhh… You see HEIC, while being highy efficient, is not highly supported in the browser environments. As you can see here none of the browsers besides latest releases of Safari support this format.

Consequently the code snippet above won’t work in most browsers.

Why should we care?

Implementing HEIC support in your web app is a time consuming task that often requires revision to make sure that your solution is compatible with your target users’s browsers. If 99% of them use Android then you can probably prioritise other tickets that have been sitting in your backlog.

Let’s look at some data though. According to Statcounter around 28% of smartphones worldwide are running IOS. Those 28% might want to upload a HEIC image they took with their iPhone to your app.

However, most of the businesses are running locally and are limited to a specific region or a country. If your business is targeting the United States market, the share of iOS users on mobile reaches almost 60%. In that case you can be sure that sooner or later a HEIC image will land in your uploader and there has to be a piece of code that will handle it.

Possible solutions

As you can see the only way to be able to reliably render HEIC in a browser is to convert it into a format that is widely supported. Right now, no one knows if Apple is going to change its licensing and allow other browsers to use HEIC images, and even if it does, waiting for it to happen doesn’t seem like a quick solution.

Here are a few approaches that I’m going to go through so that you can compare and choose the one that suits your case best:

Converting on the front-end
Converting on the back-end
Using third party solutions

Front-end conversion

When searching for HEIC conversion tools you are most likely to stumble upon libheif which is a great open source library written in C. But how do we get to run it with JavaScript in the browser environment?

There is the libheif-js library which lets you directly interact with the precompiled JavaScript libheif or run it using WebAssembly. You can also use it in node.js but I’ll cover that later. However, this is a low level solution that requires you to decode the image yourself and then render it using canvas. You can then preview it with a canvas or convert it into a jpg blob and display it with an img tag.

Here is an example of how that can be done. As you can see it’s complicated and requires a deep understanding of each step which might not be ideal if you only want to get the job done without getting into details of how libheif works.

Let’s take a look at the library called heic2any which is using libheif under the hood and makes it super simple to get started with HEIC conversion in the browser. Here’s an example implementation:

We have a file picker that accepts an HEIC image and previews it using the img tag. As you can see it takes a few lines of code to convert the image and generate a URL which we can then preview like any other static image.

I think it’s worth elaborating on one thing here…

Good to know: What is a blob?

You might be asking yourself - what is this whole blob thing in JavaScript? You see it everywhere when dealing with files and sometimes it might not be obvious why it’s being used in the first place.

Blobs allow you to create file-like objects made out of raw data and a specified MIME type. They are stored in the browser’s memory and can be accessed using a generated url. It is a perfect solution for any image manipulation because Blobs allow for storing the image data and then easily referencing it - in the example above by passing it as a src attribute.

This video gives a great introduction to how to create a blob yourself and explains a few related concepts like ArrayBuffers:

And just like that we managed to handle an HEIC image and view it without using any backend. However, keep in mind that converting images on the client is not an ideal solution and has its drawbacks that shouldn’t be undermined.

First of all, it’s heavy for the browser to first load the heic2any (2.7 MB) and then to run the actual image conversion. You never know if someone is using your app on the latest MacBook Pro or on the iPhone SE from 2016.

Besides that, you should never trust the client and handing over such crucial task as converting files to the client might be dangerous.

In our opinion frontend conversion should be only used for previewing and if you want to upload a file to the server you should send it in the original HEIC format to continue processing on the backend.

Backend conversion

Depending on your stack there are many tools you can use for handling the HEIC conversion on the backend.

We mentioned libheif before which has dedicated libraries for various languages like Node.js, C#, Python, Rust etc. Let’s take a look at an example implementation in Node.js, specifically using the heic-convert package which is an official user friendly wrapper for libheif.

Using Express.js you can quickly create a server that will handle an HEIC file and return the converted jpeg file. Multer is a middleware that lets you easily handle form data and files - I’m using the memory storage here which doesn’t save any file to the disk but with Multer you can also make the files get saved and return the image url that can be accessed multiple times.

To take it one step further in a production environment you would probably want to save those files to S3 or other storage solution and host them using a CDN.

In this example you simply get the image data which on the client side can be easily parsed to a blob and rendered. To use our backend implementation in the frontend demo I showed above you simply need to replace the heic2any call with a POST request to the server:

const formData = new FormData();
formData.append("file", blob);

fetch("http://localhost:4000", {
  method: "POST",
  body: formData,
})
  .then((res) => res.blob())
  .then((res) => {
    setPreview(URL.createObjectURL(res));
    setLoading(false);
  });

Here you go!

We have moved the crucial process of converting an image to the backend making it more reliable. Compared to the frontend conversion the performance will be more consistent as it’s always your server that is running the conversion.

The biggest disadvantage here is the effort that you have to put into spinning up your own server and maintaining it (unless you already have it).

Overall, if you need to reliably handle HEIC conversion with a prospect of also handling storage and referencing those images in a database, implementing a backend conversion is probably the best way to go.

Third party solutions

While implementing the file conversion yourself is not that hard nowadays due to a lot of libraries that get the job done for you, it can still be tricky to choose the right one and be sure that your custom implementation will work well in all the cases.

For example if your output image seems to have a slightly different color palette, you either have to tweak the parameters to match your expected output (assuming the library you’re using implements such interface) or switch to another library that might or might not solve the issue.

Also, the solutions I showed in the previous sections do not handle the file metadata. You would need to implement it separately.

You will also need to perform updates of those libraries and test if the update didn’t break anything in your code.

If you don’t want to struggle with all those things and just want to get a solution that is easy to implement and always works, you should look for an API that will outsource this process for you. For example:

They provide a straightforward documentation and you can always reach out to their support if your business has more specific requirements.

There are also more profound, end-to-end solutions for handling images which go beyond just image conversion:

They allow you to upload images, convert them, process them in many different ways like cropping, resizing or applying filters and access using a CDN. You can also specify the storage solution so it doesn’t have to be their servers. Filestack also provides you with an embedded file picker which means you don’t even have to make any requests yourself.

Using GPT for natural language querying

Chris Yaowen Zhang — Fri, 27 Jun 2025 15:09:48 +0000

Year 2023 was supposed to be the year of AI. For sure we all see some changes, especially in social media walls. For example, my LinkedIn is flooded with AI generated texts or images.
However, I prefer to focus on the realms of Artificial Intelligence that are equally fascinating and beneficial. I tried using AI, specifically GPT, to transform natural language into structured, machine-readable forms. This approach is used to generate machine-readable forms that translate into other use cases as well, i.e. the insights in this blog post can inform other equally interesting and useful applications. In this article you will learn how to use GPT for data processing and allowing users to provide natural input to an application. Let’s take a look into some interesting use cases.

It turns out that GPT does a pretty good job of parsing natural language input into data structures defined by a schema. We can use it for transforming manuals into a JSON graph, that can be then used for, e.g. presenting step-by-step instructions or visualizing dependencies.

Example: Let’s cook!

Example: Let’s cook!
We can use GPT to transform a recipe into a graph, defined by a JSON structure. It can then be used to, for example, present step-by-step instructions or visualizing dependencies between them.

The Upside team found a pretty good spaghetti recipe available here.
We can try transforming it into a JSON.

First, let’s define a JSON schema of the JSON that we’d like to get from GPT:

The schema can look like this:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://upsidelab.io/recipe.schema.json",
  "title": "Recipe",
  "description": "A recipe definition",
  "type": "object",
  "properties": {
    "ingredients": {
      "description": "The list of ingredients required to prepare the recipe",
      "type": "array",
      "items": {
        "$ref": "#Ingredient"
      }
    },
    "steps": {
      "description": "The list of steps required to prepare the recipe",
      "type": "array",
      "items": {
        "$ref": "#Step"
      }
    }
  },
  "$defs": {
    "ingredient": {
      "$anchor": "Ingredient",
      "type": "object",
      "properties": {
        "id": { "type": "number" },
        "name": { "type": "string" },
        "quantity": { "type": "string" }
      }
    },
    "step": {
      "$anchor": "Step",
      "type": "object",
      "properties": {
        "id": { "type": "number" },
        "name": { "type": "string" },
        "description": { "type": "string" },
        "dependsOnSteps": {
          "type": "array",
          "items": { "type": "number" }
        },
        "dependsOnIngredients": {
          "type": "array",
          "items": { "type": "number" }
        }
      }
    }
  }
}

Then a prompt for GPT. We’ll split it into three parts:

Schema definition
The recipe to be parsed
A summary of what we expect it to return (just the JSON, conforming to the schema)

Summarize the following recipe into a json that conforms to the following json schema delimited by three backticks

\`
${schemaDefinition}
\`
Use the following recipe, which is delimited by three backticks
\`
${recipe}
\`
Respond with the JSON only. Your output should be parsable by a JSON parser and should conform to the JSON schema definied before.

To get stable results, you may need to tweak the prompt. We’ve noticed that more complex domains GPT will often require a more complex prompt that more explicitly specifies our requirements. For a spaghetti recipe it’s enough though.
Sample output:

{
    "ingredients": [
        { "id": 1, "name": "olive oil", "quantity": "1 tbsp" },
        { "id": 2, "name": "smoked streaky bacon", "quantity": "4 rashers" },
        { "id": 3, "name": "medium onions", "quantity": "2" },
        { "id": 4, "name": "carrots", "quantity": "2" },
        { "id": 5, "name": "celery sticks", "quantity": "2" },
        { "id": 6, "name": "garlic cloves", "quantity": "2" },
        { "id": 7, "name": "rosemary sprigs", "quantity": "2-3" },
        { "id": 8, "name": "beef mince", "quantity": "500g" },
        { "id": 9, "name": "plum tomatoes", "quantity": "2 x 400g tins" },
        { "id": 10, "name": "basil", "quantity": "small pack" },
        { "id": 11, "name": "dried oregano", "quantity": "1 tsp" },
        { "id": 12, "name": "bay leaves", "quantity": "2" },
        { "id": 13, "name": "tomato purée", "quantity": "2 tbsp" },
        { "id": 14, "name": "beef stock cube", "quantity": "1" },
        { "id": 15, "name": "red chilli", "quantity": "1 (optional)" },
        { "id": 16, "name": "red wine", "quantity": "125ml" },
        { "id": 17, "name": "cherry tomatoes", "quantity": "6" },
        { "id": 18, "name": "parmesan", "quantity": "75g" },
        { "id": 19, "name": "spaghetti", "quantity": "400g" },
        { "id": 20, "name": "crusty bread", "quantity": "to serve (optional)" }
    ],
    "steps": [
        { 
            "id": 1, 
            "name": "Prepare the saucepan", 
            "description": "Put a large saucepan on a medium heat and add 1 tbsp olive oil.", 
            "dependsOnSteps": [],
            "dependsOnIngredients": [1] 
        },
        { 
            "id": 2, 
            "name": "Cook bacon", 
            "description": "Add 4 finely chopped bacon rashers and fry for 10 mins until golden and crisp.", 
            "dependsOnSteps": [1],
            "dependsOnIngredients": [2] 
        },
        { 
            "id": 3, 
            "name": "Add vegetables and herbs", 
            "description": "Reduce the heat and add the 2 onions, 2 carrots, 2 celery sticks, 2 garlic cloves and the leaves from 2-3 sprigs rosemary, all finely chopped, then fry for 10 mins.", 
            "dependsOnSteps": [2],
            "dependsOnIngredients": [3, 4, 5, 6, 7] 
        },
        { 
            "id": 4, 
            "name": "Brown the mince", 
            "description": "Increase the heat to medium-high, add 500g beef mince and cook stirring for 3-4 mins until the meat is browned all over.", 
            "dependsOnSteps": [3],
            "dependsOnIngredients": [8] 
        },
        { 
            "id": 5, 
            "name": "Add bolognese sauce ingredients", 
            "description": "Add 2 tins plum tomatoes, the finely chopped leaves from ¾ small pack basil, 1 tsp dried oregano, 2 bay leaves, 2 tbsp tomato purée, 1 beef stock cube, 1 deseeded and finely chopped red chilli (if using), 125ml red wine and 6 halved cherry tomatoes. Stir well.", 
            "dependsOnSteps": [4],
            "dependsOnIngredients": [9, 10, 11, 12, 13, 14, 15, 16, 17] 
        },
        { 
            "id": 6, 
            "name": "Simmer the sauce", 
            "description": "Bring to the boil, reduce to a gentle simmer and cover with a lid. Cook for 1 hr 15 mins stirring occasionally, until you have a rich, thick sauce.", 
            "dependsOnSteps": [5],
            "dependsOnIngredients": [] 
        },
        { 
            "id": 7, 
            "name": "Add parmesan and season", 
            "description": "Add the 75g grated parmesan, check the seasoning and stir.", 
            "dependsOnSteps": [6],
            "dependsOnIngredients": [18] 
        },
        { 
            "id": 8, 
            "name": "Cook spaghetti", 
            "description": "When the bolognese is nearly finished, cook 400g spaghetti following the pack instructions.", 
            "dependsOnSteps": [7],
            "dependsOnIngredients": [19] 
        },
        { 
            "id": 9, 
            "name": "Serve", 
            "description": "Drain the spaghetti and either stir into the bolognese sauce, or serve the sauce on top. Serve with more grated parmesan, the remaining basil leaves and crusty bread, if you like.", 
            "dependsOnSteps": [8],
            "dependsOnIngredients": [10, 18, 20] 
        }
    ]
}

This is something we can easily display on a graph:

If we aren’t satisfied with the outcome, we can also use a follow-up prompt to mutate the structure. If you look at the recipe, step 8 (cook spaghetti) is independent of the previous steps.

Let’s ask GPT to change that:

Step 8 doesn't depend on any of the previous steps and can be done in parallel. Change the resulting JSON to reflect that
{
    "ingredients": [
        { "id": 1, "name": "olive oil", "quantity": "1 tbsp" },
        { "id": 2, "name": "smoked streaky bacon", "quantity": "4 rashers" },
        { "id": 3, "name": "medium onions", "quantity": "2" },
        { "id": 4, "name": "carrots", "quantity": "2" },
        { "id": 5, "name": "celery sticks", "quantity": "2" },
        { "id": 6, "name": "garlic cloves", "quantity": "2" },
        { "id": 7, "name": "rosemary sprigs", "quantity": "2-3" },
        { "id": 8, "name": "beef mince", "quantity": "500g" },
        { "id": 9, "name": "plum tomatoes", "quantity": "2 x 400g tins" },
        { "id": 10, "name": "basil", "quantity": "small pack" },
        { "id": 11, "name": "dried oregano", "quantity": "1 tsp" },
        { "id": 12, "name": "bay leaves", "quantity": "2" },
        { "id": 13, "name": "tomato purée", "quantity": "2 tbsp" },
        { "id": 14, "name": "beef stock cube", "quantity": "1" },
        { "id": 15, "name": "red chilli", "quantity": "1 (optional)" },
        { "id": 16, "name": "red wine", "quantity": "125ml" },
        { "id": 17, "name": "cherry tomatoes", "quantity": "6" },
        { "id": 18, "name": "parmesan", "quantity": "75g" },
        { "id": 19, "name": "spaghetti", "quantity": "400g" },
        { "id": 20, "name": "crusty bread", "quantity": "to serve (optional)" }
    ],
    "steps": [
        { 
            "id": 1, 
            "name": "Prepare the saucepan", 
            "description": "Put a large saucepan on a medium heat and add 1 tbsp olive oil.", 
            "dependsOnSteps": [],
            "dependsOnIngredients": [1] 
        },
        { 
            "id": 2, 
            "name": "Cook bacon", 
            "description": "Add 4 finely chopped bacon rashers and fry for 10 mins until golden and crisp.", 
            "dependsOnSteps": [1],
            "dependsOnIngredients": [2] 
        },
        { 
            "id": 3, 
            "name": "Add vegetables and herbs", 
            "description": "Reduce the heat and add the 2 onions, 2 carrots, 2 celery sticks, 2 garlic cloves and the leaves from 2-3 sprigs rosemary, all finely chopped, then fry for 10 mins.", 
            "dependsOnSteps": [2],
            "dependsOnIngredients": [3, 4, 5, 6, 7] 
        },
        { 
            "id": 4, 
            "name": "Brown the mince", 
            "description": "Increase the heat to medium-high, add 500g beef mince and cook stirring for 3-4 mins until the meat is browned all over.", 
            "dependsOnSteps": [3],
            "dependsOnIngredients": [8] 
        },
        { 
            "id": 5, 
            "name": "Add bolognese sauce ingredients", 
            "description": "Add 2 tins plum tomatoes, the finely chopped leaves from ¾ small pack basil, 1 tsp dried oregano, 2 bay leaves, 2 tbsp tomato purée, 1 beef stock cube, 1 deseeded and finely chopped red chilli (if using), 125ml red wine and 6 halved cherry tomatoes. Stir well.", 
            "dependsOnSteps": [4],
            "dependsOnIngredients": [9, 10, 11, 12, 13, 14, 15, 16, 17] 
        },
        { 
            "id": 6, 
            "name": "Simmer the sauce", 
            "description": "Bring to the boil, reduce to a gentle simmer and cover with a lid. Cook for 1 hr 15 mins stirring occasionally, until you have a rich, thick sauce.", 
            "dependsOnSteps": [5],
            "dependsOnIngredients": [] 
        },
        { 
            "id": 7, 
            "name": "Add parmesan and season", 
            "description": "Add the 75g grated parmesan, check the seasoning and stir.", 
            "dependsOnSteps": [6],
            "dependsOnIngredients": [18] 
        },
        { 
            "id": 8, 
            "name": "Cook spaghetti", 
            "description": "When the bolognese is nearly finished, cook 400g spaghetti following the pack instructions.", 
            "dependsOnSteps": [],
            "dependsOnIngredients": [19] 
        },
        { 
            "id": 9, 
            "name": "Serve", 
            "description": "Drain the spaghetti and either stir into the bolognese sauce, or serve the sauce on top. Serve with more grated parmesan, the remaining basil leaves and crusty bread, if you like.", 
            "dependsOnSteps": [7, 8],
            "dependsOnIngredients": [10, 18, 20] 
        }
    ]
}

As the previous response is a part of the follow-up prompt, GPT does a pretty good job at keeping the unchanged elements stable.

Use case - build a BPMN specification of a business process

Let’s expand this further - this approach allows us to generate a BPMN definition of a business process, which can subsequently be used by a tool like Camunda.

Let’s try the following prompt:

Generate camunda xml file that describes a process delimited by three backticks 

\`
${processDescription}
\`

Respond with the XML only. The XML should contain process definition and a diagram of the process. The XML should be a valid camunda xml file and should be parsable by its modeller tool. There should be nothing except the xml in the response.

Where process description could be for example:

An invoice is received by a system, parsed by an OCR system, then reviewed by an accountant. After the initial review, it's sent to a manager for an approval. When the approval is made, it's sent to account payable. If the approval is not given by the manager, it is sent to accounting for further clarification.

After sending that to GPT, we’ll get the following output:

<?xml version="1.0" encoding="UTF-8"?>
<definitions xmlns="http://www.omg.org/spec/BPMN/20100524/MODEL"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xmlns:camunda="http://camunda.org/schema/1.0/bpmn"
             xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI"
             xmlns:dc="http://www.omg.org/spec/DD/20100524/DC"
             xmlns:di="http://www.omg.org/spec/DD/20100524/DI"
             xsi:schemaLocation="http://www.omg.org/spec/BPMN/20100524/MODEL BPMN20.xsd"
             id="Definitions_1"
             targetNamespace="http://bpmn.io/schema/bpmn">

  <process id="invoiceProcess" name="Invoice Processing" isExecutable="true">
    <startEvent id="startEvent" name="Invoice Received">
      <outgoing>Flow_1</outgoing>
    </startEvent>

    <sequenceFlow id="Flow_1" sourceRef="startEvent" targetRef="parseInvoice" />

    <serviceTask id="parseInvoice" name="Parse Invoice with OCR" camunda:type="external" camunda:topic="parseInvoice">
      <incoming>Flow_1</incoming>
      <outgoing>Flow_2</outgoing>
    </serviceTask>

    <sequenceFlow id="Flow_2" sourceRef="parseInvoice" targetRef="reviewByAccountant" />

    <userTask id="reviewByAccountant" name="Review by Accountant">
      <incoming>Flow_2</incoming>
      <outgoing>Flow_3</outgoing>
    </userTask>

    <sequenceFlow id="Flow_3" sourceRef="reviewByAccountant" targetRef="managerApproval" />

    <exclusiveGateway id="managerApproval" name="Manager Approval">
      <incoming>Flow_3</incoming>
      <outgoing>Flow_4</outgoing>
      <outgoing>Flow_5</outgoing>
    </exclusiveGateway>

    <sequenceFlow id="Flow_4" sourceRef="managerApproval" targetRef="sendToAccountPayable" name="Approved">
      <conditionExpression xsi:type="tFormalExpression">\${approved}</conditionExpression>
    </sequenceFlow>

    <sequenceFlow id="Flow_5" sourceRef="managerApproval" targetRef="sendToAccounting" name="Not Approved">
      <conditionExpression xsi:type="tFormalExpression">not(\${approved})</conditionExpression>
    </sequenceFlow>

    <userTask id="sendToAccountPayable" name="Send to Account Payable">
      <incoming>Flow_4</incoming>
      <outgoing>Flow_6</outgoing>
    </userTask>

    <userTask id="sendToAccounting" name="Send to Accounting for Clarification">
      <incoming>Flow_5</incoming>
      <outgoing>Flow_7</outgoing>
    </userTask>

    <endEvent id="endEvent" name="Process End">
      <incoming>Flow_6</incoming>
      <incoming>Flow_7</incoming>
    </endEvent>

    <sequenceFlow id="Flow_6" sourceRef="sendToAccountPayable" targetRef="endEvent" />
    <sequenceFlow id="Flow_7" sourceRef="sendToAccounting" targetRef="endEvent" />

  </process>

  <bpmndi:BPMNDiagram id="BPMNDiagram_1">
    <bpmndi:BPMNPlane id="BPMNPlane_1" bpmnElement="invoiceProcess">
      <bpmndi:BPMNShape id="StartEvent_1_di" bpmnElement="startEvent">
        <dc:Bounds x="173" y="102" width="36" height="36" />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNEdge id="Flow_1_di" bpmnElement="Flow_1">
        <di:waypoint x="209" y="120" />
        <di:waypoint x="261" y="120" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNShape id="ServiceTask_1_di" bpmnElement="parseInvoice">
        <dc:Bounds x="261" y="80" width="100" height="80" />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNEdge id="Flow_2_di" bpmnElement="Flow_2">
        <di:waypoint x="361" y="120" />
        <di:waypoint x="411" y="120" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNShape id="UserTask_1_di" bpmnElement="reviewByAccountant">
        <dc:Bounds x="411" y="80" width="100" height="80" />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNEdge id="Flow_3_di" bpmnElement="Flow_3">
        <di:waypoint x="511" y="120" />
        <di:waypoint x="561" y="120" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNShape id="ExclusiveGateway_1_di" bpmnElement="managerApproval" isMarkerVisible="true">
        <dc:Bounds x="561" y="95" width="50" height="50" />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNEdge id="Flow_4_di" bpmnElement="Flow_4">
        <di:waypoint x="611" y="120" />
        <di:waypoint x="661" y="120" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNEdge id="Flow_5_di" bpmnElement="Flow_5">
        <di:waypoint x="586" y="145" />
        <di:waypoint x="586" y="195" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNShape id="UserTask_2_di" bpmnElement="sendToAccountPayable">
        <dc:Bounds x="661" y="80" width="100" height="80" />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNShape id="UserTask_3_di" bpmnElement="sendToAccounting">
        <dc:Bounds x="536" y="195" width="100" height="80" />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNEdge id="Flow_6_di" bpmnElement="Flow_6">
        <di:waypoint x="761" y="120" />
        <di:waypoint x="811" y="120" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNEdge id="Flow_7_di" bpmnElement="Flow_7">
        <di:waypoint x="586" y="275" />
        <di:waypoint x="586" y="325" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNShape id="EndEvent_1_di" bpmnElement="endEvent">
        <dc:Bounds x="811" y="102" width="36" height="36" />
      </bpmndi:BPMNShape>
    </bpmndi:BPMNPlane>
  </bpmndi:BPMNDiagram>
</definitions>

Looking good!

Remarkable insights:

GPT seems to omit some of the XMLNS references quite often. This can be easily fixed in post-processing, which may be way more cost-effective than regenerating the response.
I once got a response where the diagram definition looked like the following:

<bpmndi:BPMNPlane id="BPMNPlane_1" bpmnElement="invoiceProcess">
    <!-- Diagram layout omitted for brevity -->
</bpmndi:BPMNPlane>

Even though it’s quite amusing, this is an edge case that we’d likely need to support if aiming for production use. The most common approaches are to add more validations and retries to our application’s code, improve the prompt to direct GPT, or try different temperatures of the model.

Use case - build dynamic forms in React

Another common use case that we can try to automate is building custom forms dynamically. Many applications, such as surveys, custom forms etc., rely on this feature to gain feedback from its candidates, customers or potential leads.

GPT makes it incredibly easy to build a tool that allows a regular user to create such a form by providing natural language input. Let’s use a library called react-jsonschema-form. It’s a React-based library that generates forms based on a JSON definition. It also contains an extensive schema definition that such JSONs can be validated against.

We can leverage the fact that GPT was trained on a large dataset that also includes definitions of commonly used JSON schemas. That means we don’t have to provide the exact schema, and we can refer to it in our prompt by name.

Let’s create a prompt that:

Specifies that we would like to use the schema of react-jsonschema-form
Include the user's definition of the ideal form
Will specify some additional criteria for the output (i.e. make sure that we only get a JSON as a response)

Generate a JSON definition that follows the schema of react-jsonschema-form. Use the following form that is delimited by three backticks

\`
${userInput}
\`

Respond with the JSON only and don't include any additional characters. Make sure that the output is a valid JSON and that it passes validation with the @rjsf/validator-ajv8 validator

Where userInput could be:

The form should allow a customer to provide an order number (starting with letter R followed by 9 digits), customer's phone number (limited only to country code and digits) and a customer support request. There should also be a field for selecting the type of the issue from a list of the following options: order missing, change in order, return request. At the end of the form, there should be a checkbox labeled "I agree to the privacy policy of company X"

In this case GPT gets back to us with

{
  "title": "Customer Support Form",
  "type": "object",
  "required": [
    "orderNumber",
    "phoneNumber",
    "supportRequest",
    "issueType",
    "privacyPolicyAgreement"
  ],
  "properties": {
    "orderNumber": {
      "type": "string",
      "title": "Order Number",
      "pattern": "R[0-9]{9}"
    },
    "phoneNumber": {
      "type": "string",
      "title": "Phone Number",
      "pattern": "\\+[0-9]+"
    },
    "supportRequest": {
      "type": "string",
      "title": "Customer Support Request"
    },
    "issueType": {
      "type": "string",
      "title": "Type of Issue",
      "enum": [
        "order missing",
        "change in order",
        "return request"
      ]
    },
    "privacyPolicyAgreement": {
      "type": "boolean",
      "title": "I agree to the privacy policy of company X",
      "const": true
    }
  }
}

We can quickly verify it in react-jsonschema-form’s playground: https://rjsf-team.github.io/react-jsonschema-form/

As you can see, it correctly generated a form, together with relevant validations. As with previous examples, we can make modifications by follow-up prompts.

Let’s try making the privacy policy field selectable by default. Sending the following prompt:

The privacy policy agreement should be checked by default

Will result in a new JSON definition of the form:

{
  "title": "Customer Support Form",
  "type": "object",
  "required": ["orderNumber", "phoneNumber", "supportRequest", "issueType"],
  "properties": {
    "orderNumber": {
      "type": "string",
      "title": "Order Number",
      "pattern": "R[0-9]{9}"
    },
    "phoneNumber": {
      "type": "string",
      "title": "Phone Number",
      "pattern": "\\+[0-9]+"
    },
    "supportRequest": {
      "type": "string",
      "title": "Customer Support Request"
    },
    "issueType": {
      "type": "string",
      "title": "Type of Issue",
      "enum": ["order missing", "change in order", "return request"]
    },
    "privacyPolicyAgreement": {
      "type": "boolean",
      "title": "I agree to the privacy policy of company X",
      "default": true
    }
  }
}

Which looks like that:

As you can see, the checkbox is now selected by default. Similar to that, we can make further tweaks to the existing form.

Use case - build dynamic UIs

In the previous example, we were bound to forms, which are useful but limited in terms of what we can do. To build more general UIs dynamically, we can use a tool called react-json-schema, combine it with GPT, and build fully dynamic complex UIs.

To start with, let’s define some components. We’ll use react-bootstrap to simplify the UI work needed.

Let’s make a wrapper component for the whole app:

function AppRoot({ children }) {
  return (
    <>
      {children}
    </>
  )
}

export default AppRoot

Then a basic layout component:

import { Container } from 'react-bootstrap';

function AppMainLayout({ children }) {
  return (
    <Container>
      {children}
    </Container>
  )
}

export default AppMainLayout

A customizable navbar:

import Container from 'react-bootstrap/Container';
import Nav from 'react-bootstrap/Nav';
import Navbar from 'react-bootstrap/Navbar';

function AppNavbar({ title, links }) {
  return (
    <Navbar bg="primary" data-bs-theme="dark">
    <Container>
      <Navbar.Brand href="/">{title}</Navbar.Brand>
      <Nav className="me-auto">
        {links && links.map((link) => {
          return <Nav.Link href={link.url} key={link.title}>{link.title}</Nav.Link>
        })}
      </Nav>
    </Container>
  </Navbar>
  )
}

export default AppNavbar;

A toolbar with some action buttons (that we can wire together later):

import Button from 'react-bootstrap/Button';

function AppToolbar() {
  return (
    <div className="py-2">
      <Button variant="primary" className="mx-1">New user</Button>
      <Button variant="secondary" className="mx-1">Refresh</Button>
      <Button variant="secondary" className="mx-1">Deactivate</Button>
      <Button variant="secondary" className="mx-1">Reset password</Button>
    </div>
  )
}

export default AppToolbar

And a table for displaying a list of users:

import Table from 'react-bootstrap/Table'

function AppUserList() {
  return (
    <Table striped bordered hover>
      <thead>
        <tr>
          <th>#</th>
          <th>Email</th>
          <th>First Name</th>
          <th>Last Name</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <td>1</td>
          <td>test@example.com</td>
          <td>John</td>
          <td>Doe</td>
        </tr>
        <tr>
          <td>2</td>
          <td>jack@example.com</td>
          <td>Jack</td>
          <td>Dummy</td>
        </tr>
        <tr>
          <td>3</td>
          <td>thomas@example.com</td>
          <td>Tomas</td>
          <td>Jonas</td>
        </tr>
        <tr>
          <td>4</td>
          <td>paul@example.com</td>
          <td>Paul</td>
          <td>Test</td>
        </tr>
      </tbody>
    </Table>
  )
}

export default AppUserList
view raw

Then connect it all together in our web application:

import ReactDOM from 'react-dom/client';
import reportWebVitals from './reportWebVitals';
import ReactJsonSchema from 'react-json-schema';
import AppNavbar from './components/AppNavbar'
import AppRoot from './components/AppRoot'
import AppMainLayout from './components/AppMainLayout'
import AppToolbar from './components/AppToolbar'
import AppUserList from './components/AppUserList'

const uiRenderer = new ReactJsonSchema();
uiRenderer.setComponentMap({ AppRoot, AppNavbar, AppMainLayout, AppToolbar, AppUserList });

const uiSchema = await fetchGPTResponse(prompt);

const root = ReactDOM.createRoot(document.getElementById('root'));
root.render(
  uiRenderer.parseSchema(uiSchema)
);

It’s easy:

Initialize react-json-schema parser
Configure it to recognize our custom components
Prompt GPT for a schema based on user’s description
Render the schema that GPT responded with

Now it’s time to define the actual prompt to be used:

Generate a JSON that follows the react-json-schema format for the following set of components and their props delimited by three backticks:
\`
AppRoot (children: Node[]) , AppNavbar (title: string, links: Link[]), AppMainLayout (children: Node[]) , AppToolbar (), AppUserList (), h1 (text: string), h2 (text: string)
\`
The following custom types are defined in the application
\`
type Link = {
title: string;
url: string;
}
\`

The JSON should build a UI following a specification delimited by three backticks
\`
The UI should start with a navbar, a title specifying that this is a dashboard for managing users. Below the title there should be a more detailed description of the dashboard. The main element of the UI should be a table with a list of users. Above the table there should be a toolbar allowing to modify the list of users.
\`
The UI must be wrapped in the AppRoot component. Specify the props directly in the same object as the component, without an additional props object. Only use the props that are defined above. The output should be a valid react-json-schema specification. Don't output any additional characters.

Remarkable insights: As you can see, we started with a list of available components, followed by an additional list of props available for each of them. My tests have shown that it’s also helpful to explicitly state how to provide props (without a nested props object). Without that remark, GPT put it inside a nested object, which didn’t work with react-json-schema library. My assumption is that react-json-schema is not a very popular library and lacks extensive examples online - hence the need to point GPT in the right direction.

A sample GPT response will look like:

{
  "component": "AppRoot",
  "children": [
    {
      "component": "AppNavbar",
      "title": "User Management Dashboard",
      "links": [
        {
          "title": "Home",
          "url": "/"
        },
        {
          "title": "Settings",
          "url": "/settings"
        }
      ]
    },
    {
      "component": "AppMainLayout",
      "children": [
        {
          "component": "h1",
          "text": "User Management Dashboard"
        },
        {
          "component": "h2",
          "text": "Manage and oversee user accounts and permissions"
        },
        {
          "component": "AppToolbar",
          "children": []
        },
        {
          "component": "AppUserList",
          "children": []
        }
      ]
    }
  ]
}

PICS Which renders a quite decent dashboard! GPT was able to infer that the majority of the layout should go inside a component called AppMainLayout, but didn’t put the AppNavbar there. It also filled the navbar with some example links.

Use case - querying an ElasticSearch index

ElasticSearch is commonly used for building indexes of products or application logs. We can use GPT to translate user’s queries to an ElasticSearch query.

The approach would be:

Specify that we’d like to get an ElasticSearch query as a result
Provide information about the structure of the index
Provide information about the user's query
Specify additional criteria for the output

Example: We have an ElasticSearch index with the following definition:

{
  "products_production": {
    "aliases": {},
    "mappings":{
      "spree_product": {
        "properties": {
          "available_on": {
            "type":"date",
            "format":"dateOptionalTime"
          },
          "brand_id": {
            "type":"keyword"
          },
          "created_at": {
            "type":"date",
            "format":"dateOptionalTime"
          },
          "description": {
            "type":"text",
            "analyzer":"snowball"
          },
          "discontinue_on": {
            "type":"date",
            "format":"dateOptionalTime"
          },
          "featured": {
            "type":"boolean"
          },
          "name": {
            "type":"text",
            "boost":100.0,
            "fields": {
              "untouched": {
                "type":"keyword"
              }
            },
            "analyzer":"snowball"
          },
          "price": {
            "type":"double"
          },
          "sku": {
            "type":"keyword"
          },
          "suggestion_terms": {
            "type":"text",
            "analyzer":"simple"
          },
          "taxon_ids": {
            "type":"keyword"
          }
        }
      }
    },
    "settings": {
      "index": {
        "number_of_shards":"1",
        "provided_name":"products_production",
        "creation_date":"1684177003621",
        "analysis": {
          "filter": {
            "nGram_filter": {
              "token_chars": [
                "letter",
                "digit",
                "punctuation",
                "symbol"
              ],
              "min_gram": "3",
              "type": "nGram",
              "max_gram": "4"
            }
          },
          "analyzer": {
            "nGram_analyzer": {
              "filter":[
                "lowercase",
                "asciifolding",
                "nGram_filter"
              ],
              "type": "custom",
              "tokenizer": "whitespace"
            },
            "whitespace_analyzer": {
              "filter": [
                "lowercase",
                "asciifolding"
              ],
              "type": "custom",
              "tokenizer": "whitespace"
            }
          }
        },
        "number_of_replicas": "1",
        "uuid": "WV8HWbS_QsudGKL223hwrA",
        "version": {
          "created": "6080091"
        }
      }
    }
  }
}
Create an elasticsearch query json that operates on an index with the following definition delimited by three backticks
\`
${indexDefinition}
\`

The query should return the results described here delimited by three backticks:
\`
Return the list of the most expensive products that are currently available, aren't featured and don't have any taxons associated with them.
\`

Return only a JSON and make sure it can be interpreted by an elasticsearch 6.8 instance

It’s worth mentioning that this approach should never be used on an ElasticSearch that’s not meant to be fully accessible by the user due to the potential risk of making arbitrary queries to the cluster.
It comes back with the desired query:

{
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "available_on": {
              "lte": "now"
            }
          }
        },
        {
          "term": {
            "featured": false
          }
        },
        {
          "bool": {
            "must_not": {
              "exists": {
                "field": "taxon_ids"
              }
            }
          }
        }
      ]
    }
  },
  "sort": [
    {
      "price": {
        "order": "desc"
      }
    }
  ]
}

We can now pass it to ElasticSearch and get the results:

{
    "took": 51,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": 435,
        "max_score": null,
        "hits": [
            {
                "_index": "products_production",
                "_type": "product",
                "_id": "1694",
                "_score": null,
                "_source": {
                    "name": "Asus ROG Zephyrus S Ultra Slim Gaming Laptop, 15.6\u0026quot; 144Hz IPS Type FHD, GeForce RTX 2070, Intel Core i7-9750H, 16GB DDR4, 512GB PCIe Nvme SSD, Aura Sync RGB, Windows 10, GX531GW-AB76",
                    "description": "\u003cp\u003eNvidia GeForce RTX 2070 8GB DDR6 (base: 1080 MHz, Boost: 1305 MHz; TDP: 90W). 230W Power Adaptor. Support Type-c PD 3.0 up to 65W Latest Intel Core i7-9750h Hexa-Core processor; Bluetooth 5.0 15.6\u0026rdquo; 144Hz 3ms IPS-Type full HD (1920x1080) Display 16GB 2666Hz DDR4 | 512GB PCIe NVMe SSD | Windows 10 Home 0.62\u0026rdquo; thin, 4.6 lbs ultraportable military-grade magnesium alloy body with premium metal cover Rog active Aerodynamic System (AAs) with upgraded 12V fans and anti-dust tunnels to preserve cooling performance and system stability Gigabit wave 2 Wi-Fi 5 (802.11AC) | customizable 4-zone ASUS Aura RGB Gaming Keyboard The all new ROG Zephyrus s gaming Laptop is redefining ultra-slim gaming laptops yet again with innovative engineering to cool Its 9th Gen Intel Core i7 processor and GeForce RTX Max-Q graphics, so you can immerse yourself in its no-compromise 144Hz/3ms Display. *The actual transfer speed of USB 3.0, 3.1 (Gen 1 and 2), and/or Type-C will vary depending on many factors including the processing speed of the host device, file attributes and other factors related to system configuration and your operating environment.\u003c/p\u003e",
                    "taxon_ids": [],
                    "brand_id": null,
                    "available_on": "2019-12-02T00:00:00.000Z",
                    "discontinue_on": null,
                    "featured": false,
                    "price": "1595.73",
                    "created_at": "2019-12-02T20:07:26.594Z",
                    "suggestion_terms": ["Asus ROG Zephyrus S Ultra Slim Gaming Laptop, 15.6\u0026quot; 144Hz IPS Type FHD, GeForce RTX 2070, Intel Core i7-9750H, 16GB DDR4, 512GB PCIe Nvme SSD, Aura Sync RGB, Windows 10, GX531GW-AB76"]
                },
                "sort": [1595.73]
            },  
            {
                "_index": "products_production",
                "_type": "product",
                "_id": "1470",
                "_score": null,
                "_source": {
                    "name": "Weber - Genesis II S-435 62006001 Stainless Steel",
                    "description": "\u003cp\u003eA grill that has it all. A powerful grilling engine, impressive features, and an outstanding 10 year warranty on all parts of the grill to back it up. The immense space and sear station will execute as many steaks as you need to perfection, while the side burner saut\u0026eacute;s veggies. Premium stainless steel cooking grates retain even heat and are easy to clean. The Tuck-Away warming rack keeps food warm while the main dish grills underneath. And between cookouts, store your grilling tools and accessories in the enclosed grill cabinet. Those are just a few features that set the Genesis II S-435 apart from your typical grill.\u003c/p\u003e\r\n\u003cul\u003e\r\n\u003cli\u003ePropane Grill fueled by propane\u003c/li\u003e\r\n\u003cli\u003eElectric Ignition Electronic ignition makes it easy to start\u003c/li\u003e\r\n\u003cli\u003eSide Burner Side burner for cooking versatility.\u003c/li\u003e\r\n\u003cli\u003e9,000 BTU-per-hour input Sear Station burner\u003c/li\u003e\r\n\u003cli\u003eStainless steel heat deflectors\u003c/li\u003e\r\n\u003cli\u003eStainless steel lid with thermometer\u003c/li\u003e\r\n\u003cli\u003eEnclosed cabinet with stainless steel doors\u003c/li\u003e\r\n\u003cli\u003eSix tool hooks\u003c/li\u003e\r\n\u003cli\u003eTwo heavy-duty front locking swivel casters and two large wheels\u003c/li\u003e\r\n\u003c/ul\u003e",
                    "sku": "62006001",
                    "taxon_ids": [],
                    "brand_id": null,
                    "available_on": "2019-11-03T00:00:00.000Z",
                    "discontinue_on" :"2019-11-04T00:00:00.000Z", 
                    "featured": false,
                    "price": "1315.5",
                    "created_at": "2019-11-03T19:02:48.694Z",
                    "suggestion_terms": ["Weber - Genesis II S-435 62006001 Stainless Steel"]
                },
                "sort" :[1315.5]
            },
            // more results here
        ]
    }
}

Follow-up use case - building a query in a query language that the user may not know

This feature is useful when a user needs to query a tool with its own specific query language or with a structure that the user doesn’t know. AWS seems to be running an A/B test of a feature where you can generate a CloudWatch search query based on a natural language input.

Use case - data visualization

Interesting use case - you allow users to pull report data (by e.g. querying an ElasticSearch index or a database). This is often combined with visualizing the data. Could we allow the user to also define how they’d like the data presented? Let’s say a manager needs a pie chart for their PowerPoint presentation.

Fortunately, there’s a tool called Vega which is a visualization grammar that allows you to define graphs using a JSON schema. It also comes with Vega-Lite, which allows us to easily render these charts using JavaScript. There’s also an interactive playground that we can use for testing our diagrams.

Let’s take a sample CSV file from Vega-Lite’s demo that contains the stock prices history of a few tech companies. It will be easy for us to reference it on the playground.

The file is a regular CSV that contains a few columns:

symbol,date,price
MSFT,Jan 1 2000,39.81
MSFT,Feb 1 2000,36.35
MSFT,Mar 1 2000,43.22

Let’s try to generate some charts that utilize this structure.

Generate a JSON using vega-lite schema that takes input data from file data/stocks.csv. The visualization should work on a dataset with the following columns delimited by three backticks
\`
symbol date price
\`
The visualization should result in the following result delimited by three backticks
\`
I would like to get a line chart of each of the stocks over time. There should also be a second line chart that shows only MSFT stock. Finally, I would like to see a pie chart of the average stock prices for each symbol
\`

Return only the resulting JSON and don't include any additional characters. Make sure that the visualization conforms to the vega-lite schema.

After trying it out with GPT we get the following definition

{
  "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
  "data": {"url": "data/stocks.csv"},
  "vconcat": [
    {
      "title": "Stock Prices Over Time",
      "mark": "line",
      "encoding": {
        "x": {"field": "date", "type": "temporal", "title": "Date"},
        "y": {"field": "price", "type": "quantitative", "title": "Price"},
        "color": {"field": "symbol", "type": "nominal", "title": "Symbol"}
      }
    },
    {
      "title": "MSFT Stock Price Over Time",
      "transform": [{"filter": "datum.symbol === 'MSFT'"}],
      "mark": "line",
      "encoding": {
        "x": {"field": "date", "type": "temporal", "title": "Date"},
        "y": {"field": "price", "type": "quantitative", "title": "Price"}
      }
    },
    {
      "title": "Average Stock Prices",
      "transform": [{"aggregate": [{"op": "mean", "field": "price", "as": "average_price"}], "groupby": ["symbol"]}],
      "mark": "arc",
      "encoding": {
        "theta": {"field": "average_price", "type": "quantitative", "title": "Average Price"},
        "color": {"field": "symbol", "type": "nominal", "title": "Symbol"}
      }
    }
  ]
}

Which gives us the desired set of charts

There’s also an interesting project on Github that showcases a similar concept. Their base prompt looks slightly differently:

You are a great assistant at vega-lite visualization creation. No matter what the user asks, you should always respond with a valid vega-lite specification in JSON.

You should create the vega-lite specification based on user's query.

Besides, Here are some requirements:
1. Do not contain the key called 'data' in vega-lite specification.
2. If the user ask many times, you should generate the specification based on the previous context.
3. You should consider to aggregate the field if it is quantitative and the chart has a mark type of react, bar, line, area or arc.
4. Consider to use bin for field if it is a chart like heatmap or histogram.
5. The available fields in the dataset and their types are:
${metas
    .map((field) => `${field.name} (${field.semanticType})`)
    .join(", ")}

Good practices - learn from the Upside team's experiences

Once the request was submitted to ChatGPT, we should run a JSON schema (or XML) validator on the response it provides. Despite its reliability, it may act differently than expected. Sometimes a retry will be enough to get a result that’s consistent with the defined structure. On the user experience side, there are several approaches, but you should tailor them to your specific needs. A good example of JSON schema validation is using Vega-Lite’s JSON Schema, available here.

You can apply the same method with any JSON schema you create, like for a spaghetti recipe.

Complexity - try not to bite more than GPT can chew‍

The prompts can’t be too long, as GPT may lose information and context with complicated inputs. You can use clever prompting engineering to break a task into smaller ones that gradually reduces the cognitive load and follows a logical reasoning process. For example, instead of asking GPT to format a recipe in a specific JSON schema, at first break it down into steps “Break this recipe down into steps". Once that’s achieved - "Structure this as the following JSON schema". It’s about breaking it down into steps, where each step builds on the understanding of the previous one and aligns it with the defined structure.

Performance - being patient is the key‍

Generating outputs with GPT is quite time-consuming, especially if we need to process multiple prompts in order to achieve the desired use case. We can expect improvements when it comes to hardware, so this won’t always be an issue, but for now, we need to focus on the UX level to cover up these issues. Improving UX in such cases is challenging, and often the best solution is to stream the response in smaller parts. However, if the model needs to return a full result (e.g. a JSON schema), this approach doesn’t work.

Cost - manage your expenses

However, using e.g. GPT-4 can be quite costly at scale. I wouldn’t be too worried about that in the long term, as like any technology, it will get cheaper over time, but for now it’s good to keep an eye on how much money is spent. The most sensible approach is to perform as much preprocessing as possible before pushing input to GPT, in order to limit the cost of the actual GPT usage.

Reliability - fine-tuning for diverse inputs

Playing around with these examples makes it quite easy to get it to a point where it fairly consistently delivers a great result. However, once you launch a GPT-based feature to a wider audience that will use it with different input data, you may need plenty of fine-tuning in order to ensure the solution's reliability. You never really know what the user’s will type into the chat box once you deploy your app. As developers of the feature, we are usually biased. User testing those solutions is important, especially since using a chat interface to solve actual problems (rather than just getting answers) is still a pretty novel idea and people have different expectations.

Prompt injection risks - always double check

It goes without saying that you need to be careful about the responses it may generate // (Optional) Screenshot of DPD example (the one where a customer prompted it to write a song about a shitty delivery company) ![[dpd_chatbot_fail-1.png]]

Summary

GPT does a pretty good job in parsing data. Basing on the experiments described in the post, we can define a simple framework that can handle diverse input and output formats:

Specify the desired schema
Specify the details of available fields (when needed)
Specify the input data
Reinforce important details (e.g. that the response should be a JSON and only a JSON)

Even now, there are many applications that can utilize GPT, without being AI-first. It doesn’t need to be a whole GPT-based experience, it can supplement the existing experiences and improve user productivity.

Limiting Data Egress: Challenges, Solutions, and Best Practices

Chris Yaowen Zhang — Wed, 28 May 2025 14:52:04 +0000

Long story Short or simply put - Introduction
In today's interconnected digital landscape, data flows freely across borders, networks, and platforms, driving innovation, powering economies, and shaping industries. However, amid this digital renaissance, data egress emerges as a critical consideration for businesses navigating the complex terrain of data management and compliance.

Generally speaking, data egress, which is the movement of data out of a controlled environment, poses multifaceted challenges and opportunities for organizations across various sectors. From ensuring regulatory compliance to safeguarding intellectual property and maintaining data sovereignty, the implications of data egress are far-reaching and demand meticulous attention.

To address these challenges, we will use the Health Data Nexus as an example of a strategy we have implemented for the Temerty Centre for Artificial Intelligence Research and Education in Medicine (T-CAIREM) at the University of Toronto.

What is Health Data Nexus?
The Health Data Nexus is the infrastructure deployed to manage the data and analytical environments for advancing health through AI at T-CAIREM. The platform itself stores de-identified data provided by various parties to help facilitate new advancements in healthcare.

While we could just share de-identified medical data by sending files to the parties that would like to do research on them, it is not the best approach. In such a scenario, we would not have much control over the files themselves. Instead, we have decided to build a platform deployed to Google Cloud, that allows credentialed users to access and work on the data but only within the boundaries that are being set by cloud environments that we provide. That approach, besides making it easier for us to manage the process more efficiently, also makes it possible to introduce additional security measures against malicious actions taken on data.

From the user’s perspective, accessing the datasets is really simple - they go to the web platform, request for access to data, then spin up a dedicated server in the cloud for their research. That server gives a limited Jupyter/RStudio interface that also implements access controls. The user then gets access to the data via a web interface and can execute their own scripts that use the dataset.

When the workbench with a predefined environment is up and ready, all the data is being fused into the working space directly from a cloud storage service. At that point, the user has access to the health data that is stored by the service itself. With great power comes great responsibility, one would say.

While we trust the users of our platform to have good intentions, and even if the aforementioned data is anonymized, we would like to minimize the possibility of the data getting outside of the platform. A merely accidental leakage of data may happen during work, for instance with a remote version controls by committing the files to a public repository.

With that in mind, let’s see what are the solutions that we can implement to limit the risk of data leaking outside of the platform.

General solution graph:

Easy, not so reliable, a little bit hacky but still worth mentioning
There are a few simple, rather obvious, but easily overlooked ways to attempt to prevent data leakage, but overall speaking, they can be bypassed with some knowledge and are not the cleanest solutions.

The first one would be setting proper permissions for the files that are said to have crucial data inside.

When fusing/adding those files to the container itself, we can set only read permissions to them(either setting those on the data container/bucket that the data is being stored in or by changing the default unix permissions on a profile to 744 - read only to group and others) and disabling root access by switching its shell from /bin/bash or /bin/sh (or any other shell that permits user login) to /sbin/nologin, in the /etc/passwd file. That will definitely lower the chance to easily modify the data or execute some script on them that will spread the data to the internet.

Another approach - that can and should be combined with previous one - would be to configure tools like RStudio or Vertex AI to prevent file downloads in their UI - such a simple change may reduce risks when not as experienced users will use our environment. Unfortunately, since the users can execute arbitrary code in the IDE, we cannot stop here.

Egress Prevention based on Google Cloud Platform
Google Cloud Platform, same as AWS or Azure - has a few out-of-the-box ways to secure a web interface, and hence prevent unauthorized persons from accessing it. We can archive that by using IAP (Identity-Aware Proxy), which is integrated with Google’s Cloud Identity. By having it connected to Google we can use Cloud Permissions systems consistently. When I first had to integrate security measures I assumed it would be a few simple firewall policy rules, but going through the implementation it is much more complex - adding IAP to non native environment makes the process quite complicated, the whole process will be described later. In addition to that “adding a few simple firewall rules” will not cover all edge cases so we need to actually add quite complex configuration, which also will not limit the user itself - quite interesting examples will be shown later.

Going to the point, which is IAP since we need to add a layer of authentication before accessing the internet exposed endpoint. In the case of GCP there are several ways to achieve that. The easiest one is using AppEngine environments that are under the hood based on virtual machines but are restricted in some areas - one of those is not having access to persistent storage. Going straight to code, a simple App Engine with IAP can be created using Terraform:

resource "google_app_engine_application" "app" {  
  project = google_project.my_project.project_id  
 location_id = "us-central"  
  iap {  
   enabled = true  
   oauth2_client_id = var.oauth2_client_id  
   oauth2_client_secret = var.oauth2_client_secret  
  }  
}

Now, this only covers a simple AppEngine setup. What if our application is more complex, and we would like to have access to a more customisable environment to which persistent storage can be attached - in such case we will need to create everything that AppEngine is providing us out of the box, which is a point of connection to the environment itself. Starting from the top - to add IAP to the environment we need a FQDN that has SSL certificate attached to it, that means we need to add a dns entry. In addition to that load balancer is needed to separate the internet facing part and backend part of our configuration. With such a setup we have all the basic requirements fulfilled to enable IAP, the only part missing is an actual connection between IAP config and the environment itself.

The last part in our case would be adding a network endpoint which is connected to designated vm and also attached to previously mentioned load balancer. There is also one hidden advantage of using a load balancer to access our environment - it serves as proxy and by default most of the cloud services provide out of the box traffic filtering and ddos attacks prevention.

resource "google_compute_address" "frontend-ip" {  
  name = "${var.name}-ipv4-address-frontend"  
}   

resource "google_dns_record_set" "a" {  
  name = "${var.dns_name}"  
  type = "A"  
  managed_zone = var.dns_zone  
 ttl = 180  
  rrdatas = [module.lb-http.external_ip]  
  project = var.dns_project  
}   

resource "google_compute_instance" "instance" {  
 name = var.name  
 zone = var.zone  
 machine_type = var.machine_type  
 allow_stopping_for_update = true  

 boot_disk {  
    initialize_params {  
      image = var.vm_image  
      type = "pd-standard"  
      size = 100  
  }  
  }  

  attached_disk {  
    source = google_compute_disk.persistent.name  
    device_name = google_compute_disk.persistent.name  
  }  

  network_interface {  
    network = var.network_name  
    subnetwork = var.region  
    access_config {  
      nat_ip = google_compute_address.static-ip.address  
      }  
  }  

  metadata = {  
   terraform = "true" 
   }  
  tags = ["ssh"]  
  service_account {  
    email = "${var.service_account_name}@${var.project_id}.iam.gserviceaccount.com"  
  }
  scheduling {  
    on_host_maintenance = "TERMINATE"  
  }  
}  

resource "google_compute_global_network_endpoint_group" "neg" {  
  name = "neg-${var.name}"  
  network_endpoint_type = "INTERNET_IP_PORT"  
  default_port = var.default_neg_port  
}  

resource "google_compute_global_network_endpoint" "endpoint" {  
 global_network_endpoint_group = google_compute_global_network_endpoint_group.neg.id  
 ip_address = google_compute_address.static-ip.address  
 port = google_compute_global_network_endpoint_group.neg.default_port  
}  

resource "google_iap_client" "oauth-client" {  
  display_name = "${var.name}-client"  
  brand = var.brand_name  
}  

resource "google_iap_web_iam_member" "access-role" {  
  member = "user:${var.emailid}"  
  role = "roles/iap.httpsResourceAccessor"  
}  

module "lb-http" {  
 project = var.project_id  
 source = "terraform-google-modules/lb-http/google//modules/serverless_negs"  
 name = var.name  
 ssl = true  
 create_ssl_certificate = true  
 private_key = var.private_key  
 certificate = var.certificate  
 address = google_compute_address.frontend-ip.address  

 backends = {  
    default = {  
      load_balancing_scheme = "EXTERNAL_MANAGED"  
  name = "backend-${var.name}"  
  enable_cdn = false  
  timeout_sec = var.lb_timeout  
  connection_draining_timeout_sec = var.lb_connection_drain  

  iap_config = {  
    enable = true  
    oauth2_client_id = google_iap_client.oauth-client.client_id  
    oauth2_client_secret = google_iap_client.oauth-client.secret  
  }  

      log_config = {  
        enable = false  
  }  

      groups = [  
        {  
          group = google_compute_global_network_endpoint_group.neg.id  
  }  
      ]  
    }  
  }  
}

We cannot trust that a wrong person will not know our IP or FQDN (fully qualified domain name). Having such a setup, which will prevent them from doing malicious actions, is an obvious thing to do to protect user environments.

VPC, Firewall rules and policies
Another option to make an instance more secure is creating a VPC network with subnets and defined firewall rules. With VPC in place, we can now proceed with adding rules and policies that will be our shield.

Firewall rules in GCP are configurations that control the traffic to and from resources based on specified criteria, such as IP addresses, protocols, and ports. Firewall policies offer a higher level of control and standardization, enabling customization on FQDN and geolocalization. Furthermore, policies are coupled with a few out-of-the-box rules that should exclude threats that are widely known. The downside of the higher standardization is that it has to be used on a larger resource pool, like a project or even organization. On the other hand, each Firewall rule can be applied to a single resource like a VM.

Using a simple terraform script, it can be done in seconds.

resource "google_compute_network" "vpc" {
  project                 = google_project.project.project_id
  name                    = var.network_name
  auto_create_subnetworks = "false"
  routing_mode            = "GLOBAL"
}

//Subnet
resource "google_compute_subnetwork" "subnet" {
  depends_on    = [google_compute_network.vpc]
  project       = google_project.project.project_id
  name          = var.subnet_name
  ip_cidr_range = var.subnet_ip
  network       = var.network_name
  region        = var.region
}

A firewall rules can be created by using terraform:


resource "google_compute_firewall" "sample_rule" {
 project       = google_project.project.project_id
 name          = var.rule_name
 network       = var.network_name
 source_ranges = ["0.0.0.0/0"]

 allow {
   protocol = "tcp"
   ports    = ["8888", "8787", "80"]
 }
 target_tags = ["http"]
}

By creating a proper setup of Firewall rules we can allow only IPs that are associated with a specific user and therefore block anyone else from accessing them; furthermore, by doing that, most of the egress points will be blocked. Using policy rules would be a little bit more convenient since we do not always know what IP address the end user will be using and we would spare the need to adjust the rules whenever the IP changes. Creating a simple policy rule can be also accomplished by a Terraform script:

resource "google_compute_firewall_policy_rule" "primary" {
  firewall_policy = google_compute_firewall_policy.default.name
  description     = "Resource created for Terraform testing"
  priority        = 9000
  enable_logging  = true
  action          = "allow"
  direction       = "EGRESS"
  disabled        = false
  match {
    layer4_configs {
      ip_protocol = "tcp"
      ports = [8080]
    }
    dest_ip_ranges = ["0.0.0.0/0"]
    dest_fqdns = []
    dest_region_codes = ["US"]
    dest_threat_intelligences = ["iplist-known-malicious-ips"]
    src_address_groups = []
    dest_address_groups = [google_network_security_address_group.basic_global_networksecurity_address_group.id]
  }
  target_service_accounts = ["my@service-account.com"]
}

Limiting Outbound Connections
Theoretically speaking we should be able to block outbound connections on the firewall level and allow the application itself to respond to incoming ones, which in our case would be users connecting to the Web UI. In most cases that is the course of actions that should be pursued, but in case anyone would like to use Vertex AI Workbenches it is not archivable easy way. Vertex AI Workbenches are heavily reliant on GCP itself so they need to have access to making outbound connections, to be precise we need to enable access to both private and restricted GCP IP pools or the environment will try to boot up infinitely.

In addition to that when the user is trying to connect the environment itself is adding custom elements to the UI so for that brief moment it needs to connect to the internet, but all is not lost yet - if the user is already connected there is no more need for such requests so we can modify our firewall policies “on the fly”.

The flow is quite simple: an environment is being created with an IP open for the user, the user itself starts establishing connection to the Vertex AI workbench, then the moment connection succeeds we initiate a process to modify our policy rules to block that point of connection. Since the user is already in he/she will be able to use that workbench but we will be protected from other connections. Having such a setup is hard to make but definitely would serve its purpose.

On the other hand combining a few policies - including those that exclude typical egress points like github, pastebin, pip and other data storage services - can create a solid shield for data which is much more maintainable. Excluding pip is a bit troublesome since the user will not be able to install packages but that can be addressed by pre installing those libraries on environment start or providing a private pip cache server that will be allowed to connect.

Conclusion
In today's data-driven world, safeguarding data egress is crucial for organizations. Implementing proper permissions, leveraging built-in egress prevention tools, and using advanced configurations like load balancers with IAP and VPC networks are essential steps in securing data environments. These measures help prevent accidental leaks, data reidentification, and malicious activities, but are not definitely foul proof and those accidents can still occur, so keep in mind that we also should have some steps to proceed prepared for such scenarios. On the other hand, keep in mind it is always a tradeoff between security and usability. The more secure the environment, the less the user can do and experiment on. Having a proper balance between those two can create a user-friendly space.

Disclaimer: This article was originally written by Upside Labs.