DEV Community: Austin Gil

The C̶a̶k̶e̶ User Location is a Lie!!!

Austin Gil — Wed, 31 Jul 2024 18:29:32 +0000

I recently sat in on a discussion about programming based on user location. Folks that are way smarter than me covered technical limitations, legal concerns, and privacy rights. It was nuanced, to say the least.

So, I thought I’d share some details.

Location, Location, Location

There are several common examples when you may want to add location-based logic to your app:

You want to set the language or currency of your app based on the region.
You’re offering discounts to people in a given country.
You have a store locator that should show user’s their nearest location.
Your weather app relies on a location before it can offer any sort of data.
You want to geofence your app for legal reasons (eg. cookie banners).

These are just a few use-cases. There are plenty more, but from these, we can identify some common themes:

Presentation/User experience: Using location information to improve or streamline the user experience.
Function/Logic: The application’s business logic changes based on location.
Policy/Compliance: You have legal requirements to include or exclude functionality.

It’s not always this clear-cut. There is overlap in some cases, but it’s important to keep these distinctions in mind, because getting it wrong has different levels of severity. Showing the wrong currency is not as bad as miscalculating tax rates, which is still not as bad as violating an embargo, for example.

With that in mind, let’s look at the options we have.

Getting User Location

There are four ways I know of to access the user’s location, each with their pros and cons.

User reporting
Device heuristics
IP Address
Edge compute

Getting User Location From the User

This is when you have a form on your website that explicitly asks a user where they are. It may offer user experience improvements like auto-completing an address, but ultimately, you are taking the user at their word.

This method has the benefits of being easy to get started (an HTML form will do), provides as reliable information as the user allows, is flexible to support different locations.

The most obvious down-side is that it may not be accurate if the user mistypes or omits information. Furthermore, it’s very easy for a user to provide false information. This can be allowed in some cases, and a big mistake in others.

Take this, for example.

This is a legitimate place in New Jersey…it’s ok to laugh. I actually went down a bit of a rabbit hole “researching” real places with funny names and spent way too much time, but I came across some real gems: Monkey’s Eyebrow – Kentucky, Big Butt Mountain – North Carolina, Unalaska – Alaska, Why – Arizona, Whynot – North Carolina.

Full list of real places with funny names

Accident, Maryland
Bacon Level, Alabama
Bald Head, Maine
Bat Cave, North Carolina
Batman
Beaverlick
Bell End
Bigfoot, Texas
Bitter End, Tennessee
Booger Hole, West Virginia
Boring, Oregon (I’ve been there!)
Breeding, Kentucky
Burnout, Alabama
Burnt Store, Florida
Butternuts, New York
Butthole Lane
Carefree, Arizona
Center of the World, Ohio
Cheesequake, New Jersey
Chicken, Alaska
Chugwater, Wyoming
Cookietown, Oklahoma
Correct, Indiana
Dick’s Knob
Ding Dong, Texas
Disappointment Islands
Earth, Texas
Eggnog, Utah
Fifty-Six, Arkansas
Funk, Nebraska
Greasy Corner, Arkansas
Hell, Michigan
Hot Coffee, Mississippi
Humptulips, Washington
Idiotville
Imalone, Wisconsin
Intercourse, Pennsylvania
Ketchuptown, South Carolina
Kickapoo, Kansas
Looneyville, Texas
Moose Factory
Mosquitoville, Vermont
Neutral, Kansas
New Erection
No Name, Colorado
Normal, Illinois
Nothing, Arizona
Peculiar, Missouri
Pee Pee, Ohio
Red Shirt, South Dakota
Sandwich, Massachusetts
Satan’s Kingdom
Scratch My Arse Rock
Slaughterville, Oklahoma
Stupid Lake
Sweet Lips, Tennessee
Worms, Nebraska
Zzyzx, California

Anyway, if you decide to take this approach, it’s a good idea to either use a form control with pre-selected options (select or radio), or integrate some sort of auto-complete (location API). This provides a better use-experience, and usually leads to more complete/reliable/accurate data.

Getting User Location From the Device

Modern devices like smartphones and laptops have access to their location information through GPS, Wi-Fi data, cell towers, and IP address. As web developers, we don’t get direct access to this information, for security reasons, but there are some things we can do.

The first thing that comes to mind is the Geolocation API built into the browser. This provides a way for websites to request access to the user’s location with the getCurrentPosition method:

navigator.geolocation.getCurrentPosition(data => {
  console.log(data)
})

The function provides you with a GeolocationPosition object containing latitude, longitude, and other information:

{
  coords: {
    accuracy: 1153.4846436496573
    altitude: null
    altitudeAccuracy: null
    heading: null
    latitude: 28.4885376
    longitude: 49.6407936
    speed: null
  },
  timestamp: 1710198149557
}

Great! Just one problem:

The first time a website tries to use the Geolocation API, the user will be prompted with a request to share their information.

Best case: the user understands the extra step and accepts.
Mid case: the user gets annoyed and has a 50/50 chance of accepting or denying.
Worst case: the user is paranoid about government surveilance, assumes worst intentions, and never comes back to your app (this is me).

When using an API that requires a user verification, it’s often a good idea to let the user know ahead of time to expect the popup, and only trigger it right when you need it. In other words, don’t request access as soon as your app loads. Wait until the user has focused on the location input field, for example.

Getting User Location From Their IP Address

In case you’re not familiar, an IP address looks like this, 192.0.2.1. They are used to uniquely identify and locate devices in a network. This is how computers communicate over the internet, and each packet of data contains information about the IP address of the sender. Your home internet modem is a good example of a device in a network with an IP address.

The relevant thing to note is that you can get location information from an IP address. Each chunk of numbers (separated by periods) represents a subnet from broader to finer scope. You can think of it as going from country to ISP, to region, to user. It doesn’t get fine enough to know someone’s specific address, but it’s possible to get the city or zip code.

Here are two great resources if you want to know more about how this works:

For JavaScript developers like myself, you can access the remote IP in Node.js with response.socket.remoteAddress. And note that you are not getting the user’s IP, technically. You’re getting the IP address for the user’s connection (and anyone else on their connection), by way of their modem and ISP.

Internet user -> ISP -> IP address.

An IP address alone is not enough to know where a user is coming from. You’ll need to look up the IP address subnets against a database of known subnet locations. It usually doesn’t make sense to maintain your own list. Instead, you can download an existing one, or ping a 3rd party service to look it up.

For basic needs, ip2location.com and KeyCDN offer free, limited options. For apps that rely heavily on determining geolocation from IP addresses or need a higher level of accuracy, you’ll want something more robust.

So now, we have a solution that requires no work from the user, and has a pretty high level of accuracy. Pretty high accuracy is not a guarantee that the user’s IP address is accurate, as we will see.

Getting User Location From Edge Compute

I’ve written several articles about edge compute in the past, so I won’t go too deep, but edge compute is a way to run dynamic, server-side code against a user’s request from the nearest server. It works by routing all requests through a network of globally distributed servers, or nodes, and allowing the network to choose the nearest node to the user.

The great thing about edge compute is that the platforms provide you with user location information without the need to ask the user for permission or look up an IP address. It can provide this information because every node knows where it lives.

Akamai’s edge compute platform, EdgeWorkers, gives you access to a request object with a userLocation property. This property is a User Location Object that looks something like this:

{
  areaCodes: ["617"],
  bandwidth: "257",
  city: "CAMBRIDGE",
  continent: "NA", // North America
  country: "US",
  dma: "506",
  fips: ["25"],
  latitude: "42.364948",
  longitude: "-71.088783",
  networkType: "mobile",
  region: "MA",
  timezone: "GMT",
  zipCode: "02114+02134+02138-02142+02163+02238",
}

So now we have a reliable source of location information with little effort, the only issue is that it’s not technically the user’s location. The User Location Object actually represents the edge node that received the user’s request. This will be the closest node to the user; likely in the same area. This is a subtle distinction, but depending on your needs, it can make a big difference.

This is Why We Can’t Have Nice Things!

So we’ve covered some options along with their benefits and caveats, but here’s the real kicker. None of the options we’ve looked at can be trusted.

Can’t trust the user

As mentioned above, we can’t trust users to always be honest and put in their actual location. And even if we could, they could make mistakes. And even if they don’t some data can be mistaken. For example, if I ask someone for their city, and they put “Portland” how can I be certain they mean Portland, OR (the best Portland), and not one of the 18+ others (in the US, alone).

Can’t trust the device

The first issue with things like the Geolocation API is that the user can just disallow using it. To which you may respond, “Fine, they can’t use my app then.” But this also fails to address another issue, which is the fact that the Geolocation API information can actually be overwritten by the user in their browser settings. And it’s not even that hard.

Can’t trust the IP address

I’m not sure if it’s possible to spoof an IP address for the computer that is connecting to your website, but it’s pretty easy for a user to route their request through a proxy client. Commonly, this is referred to as a Virtual Private Network, or VPN. The user connects to a VPN, their request goes to the VPN first, then the VPN connects to your website. As a result, the IP address you see is the VPN’s, not the user’s. This means any location data you get will be for the VPN, and not the user.

Can’t trust edge compute

Edge compute offers reliable information, but that information is the location of the edge node, and not the actual user. Often, they can be close enough, but it’s possible that the user lives near the border of one region and their nearest edge node is on the other side of that border. What happens if you have distinct behavior based on those regional differences? Also, edge compute is not free from the same VPN issues as IP addresses. With Akamai’s Enhanced Proxy Detection, you can identify if someone is using a VPN, but you still can’t access their original IP address.

What Can We Do About It?

So, there are a lot of ways to get location information, but none of them are entirely reliable. In fact, browser extensions can make it trivial for users to circumvent our efforts. Does that mean we should give up?

No!

I want to leave you better informed and prepared. So let’s look at some examples.

Content Translation

Say we have a website written in English, but also supports other languages. We’d like to improve the user experience by loading the local language of the user.

How should we treat users from Belgium, where they speak Dutch (Flemish), French, and German? Should we default to the most common language (Dutch)? Should default to the default website language (English)?

For the first render of the page, I think it’s safe to either use the default language or the best guess, but the key thing is to let the user decide which is best for them (maybe they only speak French) and honor their decision on subsequent visits.

It could look like this:

User requests the website.
Request passes through edge compute to determine it’s coming from Belgium.
Edge compute looks for the language preference from an HTTP cookie.
If the cookie is present, use the preferred language.
If the cookie is not present, use the English or Dutch version.
In the website, provide the user with a list of predefined, supported languages (maybe using a <select> field).
When the user selects a language preference, store the value in a cookie for future sessions.

In this scenario, we combine edge compute with user reporting to get location information to improve the experience. I don’t think it makes sense to use the Geolocation API at all. There is a risk of showing the wrong language, but the cost is low. The website works even if the location information is wrong or missing.

Weather App

In this example, we have an application that shows the weather information based on location. In this case, the app requires the location information in order to work. How else can we show the weather?

In this scenario, it’s still safe to assume the user’s location on first load. We can pull that information either from edge compute, or from the IP address, then show (what we think is) the user’s local weather. In addition to that, because the website’s main focus relies on location, we can use the Geolocation API to ask for more accurate data. We’ll also want to offer a flexible user reporting option in case the user want information for a different location. For that, a search input with auto-complete to fill in the location information with as much detail as possible. How you handle future visits may vary. You could always default to the “local” weather, or you could remember the location from the previous visit.

User requests the website.
On the first request, start the app assuming location information from edge compute or IP address.
On the first client-side load, initiate the Geolocation API and update the information if necessary.
You can store location information in a cookie for future loads.
For other location searches, provide a flexible input that auto-completes location information and updates the app on submission.

The important thing to note here is that the app doesn’t actually care about where the user is located. We just care about having a location. User reported location (search) takes precedence over a location found in a cookie, edge compute, or IP address.

Due to the daily change in weather, it’s also worth considering caching strategy and whether the app should be primarily server-rendered or client-rendered.

Store Locator

Imagine you run a brick-and-mortar business with multiple locations. You might show your product catalog and inventory online, but a good practice is to offer up-to-date information about the in-store inventory. For that, you would need to know which store to show inventory for, and for the best user experience, it should be the store closest to the user.

Once again, it makes sense to predict the user’s location using edge compute or IP address. Then, you also want to offer a flexible input that allows the user to put in their location information, but any auto-complete should be limited to the list of stores, sorted by proximity. It’s also good to initiate the Geolocation API.

The difference between this example and the last is that the main purpose of the site is not location dependent. Therefore, you should wait until the user has interacted with the location-dependent feature. In other words, only ask the user for their location when they’ve focused on the store locator field.

Regional Pricing

This one’s a little tricky, but how would you handle charging different prices based on the user’s location? For example, some airlines and hotels have been reported to have higher prices for users booking from one region vs. another.

Ethics aside, this is a question about profits, which is highly impactful. So, you probably don’t want to allow users to easily change their prices through user-reported location information.

In this case, you’d probably only use edge compute or IP address. It’s possible for users to get around it with a VPN, but it’s probably the best you could do. If you’re really concerned about avoiding scammers, you could use Akamai’s Enhanced Proxy Detection and try blocking requests from VPN users, but that could lead to a no-sale instead of a discounted sale. Up to you.

Cookie Banners

This last example focuses more on the legal compliance side, so I’ll start with a small disclaimer: I AM NOT A LAWYER!!! This is a hypothetical example and should not be taken as legal advice.

In 2016, the European Union passed the General Data Protection Regulation (GDPR). It’s a law that protects the privacy of internet users in the EU, and it applies to companies that offer goods or services to individuals in the EU, even if the company is based elsewhere.

It has a lot of requirements for website owners, but the one I’ll focus on is the blight of cookie banners we now see everywhere online.

I’ll avoid discussing privacy issues, whether cookie banners are right or wrong, the effectiveness or ineffectiveness of them, or if there is a better approach. Instead, I’ll just say that you may want to only show cookie banners when you are legally required, and avoid them otherwise.

Once again, knowing the user’s location is pretty important. This is very similar to the previous case, and the implementation is similar too. The main difference is the severity of getting it wrong, and therefore the level of effort to get it right.

Cookie banners might be the most ubiquitous example of how legislation and user location can impact a website, but if you’re looking for the most powerful, it’s probably The Great Firewall of China.

Closing

Alright, hopefully this long and windy road has brought us all to the same place: The magical land of nuance.

We still didn’t touch on a couple of other challenges:

What happens when a user changes their location mid-session?
What happens if there time zones are involved?
How do you report location information for disputed territories?

Still, I hope you found it useful in learning how user location is determined, what challenges it faces, and some ways you might approach various scenarios. Unfortunately, there is no one right way to approach location data. Some scenarios are better suited for user reporting, some are better for device heuristics, and some are better for edge compute or IP address. In most cases, it’s some sort of combination.

The important things you need to ask yourself are:

Do you need the user’s location or just any location?
How accurate does the data need to be?
Is it OK if the user location is falsified?

You also legal compliance, regulations, functionality, is 95% reliable ok?

If any of your location logic is for legal reasons, you’ll want to take steps to protect yourself. Account for data privacy laws like CCPA and GDPR. Include messaging in your terms of service to disallow bad behavior. These are some things to consider, but I’m no lawyer. Consult your legal team.

That’s all I have to say about that, but if you’re interested in cloud computing, new customers can sign up at linode.com/austingil for some free credits :)

Thank you so much for reading. If you liked this article, and want to support me, the best ways to do so are to share it, sign up for my newsletter, and follow me on Twitter.

Originally published on austingil.com.

I Deployed My Own Cute Lil’ Private Internet (a.k.a. VPC)

Austin Gil — Mon, 18 Mar 2024 16:24:42 +0000

Recently, I had the pleasure of attending DeveloperWeek in Oakland, CA. In addition to working the Akamai booth, making new friends, and spreading the good word of cloud computing, my team mate, Talia and I were tasked with creating a demo to showcase the new VPC product.

Background

A Virtual Private Cloud (VPC) enables private communication between two cloud compute instances, isolating network traffic from other internet users, thus improving security.

So, how did I decide to showcase this? By building a little Pokémon dashboard, of course.

I deployed two apps, each consisting of an app server and a database server (four servers total). The first app + database server pair is deployed normally, the second is configured to run within a VPC.

Each app’s front end is built with Qwik and uses Tailwind for styling. The server-side is powered by Qwik City (Qwik’s official meta-framework) and runs on Node.js hosted on a shared Linode VPS. The apps also use PM2 for process management and Caddy as a reverse proxy and SSL provisioner. The data is stored in a PostgreSQL database that also runs on a shared Linode VPS. The apps interact with the database using Drizzle, an Object-Relational Mapper (ORM) for JavaScript. The entire infrastructure for both apps is managed with Terraform using the Terraform Linode provider, which was new to me, but made provisioning and destroying infrastructure really fast and easy (once I learned how it all worked).

If you’re interested, you can find all the code here: github.com/AustinGil/linode-vpc-demo

Demo

As I mentioned above, the demo deploys two identical apps. There isn’t anything remarkably special about it, but here’s a screenshot.

(I had to change the Pokémon names for reasons…)

There’s nothing special about this tech stack. I chose these tools because I like them, not necessarily because they were the best tools for the job.

The interesting part is the infrastructure.

When we consider app #1, it’s essentially made up of two servers hosted inside the Akamai cloud, one server for the app and one server for the database. When a user loads the app, the app server pulls the data from the database, constructs the HTML, and returns the result to the user.

The problem here is how the database connection is configured. In some cases, you may deploy a database server without knowing the IP addresses of the computers you want to allow access from (like the app server). In these cases, it’s not uncommon to allow any computer with the right credentials to connect to the database. This presents a security vulnerability because it could allow for a bad actor to connect to the database and steal sensitive data.

A bad actor would still need the database host, port, username, and password to get access, so it’s not trivial. And as I said, this is not an uncommon practice, but we can do better.

If you know the IP address for every computer that needs access, a good solution might be to set up a firewall or VLAN. But if your infrastructure is more dynamic, with servers coming up and down, maintaining lists of IP addresses can be cumbersome. And that’s where VPCs shine. You can configure servers to live within a VPC, and allow communication to flow freely, only between other computers in the network.

That’s how app #2 is set up. Users can connect to the app server, which allows traffic from the public internet, but also lives within the VPC. The app server connects to the database, which is also in the VPC and only allows connections from within the same network. Then the app server takes the data, builds the HTML, and returns the page to the user.

For a normal user, the experience is identical. The browser loads the table with the modified Pokémon data just fine. The VPC doesn’t change anything for normal users.

For bad actors, however, the experience is different. Even if they somehow manage to get the database access credentials, they would not be able to connect because the network isolation from the VPC. Here, the VPC acts as a virtual firewall, ensuring that only devices with the same network are able to access the database.

(This concept is sometimes referred to as “segmentation”)

Evidence

It’s cool to show a demo and talk about the infrastructure with cute diagrams, but I always want to prove, even if just to myself, that things work as expected. So I thought a good way to test it would be to try connecting directly to both databases using my database client, DBeaver.

For database #1, I set up a Postgres connection using the host IP address I got from my Akamai dashboard and the port, username, and password I had set up in my Terraform script. The connection worked as expected.

For database #2, all I had to change was the IP address, since all the database provisioning was handled by the same script using Terraform. The only difference was that the database server was put inside the same VPC as the app server, and it was configured to only allow connections from any computer within the same network.

As expected, I got an error when trying to connect, even though I had all the correct information.

The error doesn’t mention anything about the VPC. It just says that my IP address is not in the allowed list in the configuration file. This makes sense. I could explicitly add my home’s IP address and gain access to the database if needed, but that’s beside the point.

The key point is that I did not explicitly add any IP address to the Postgres allow-list. Yet, the app server was able to connect just fine, and everyone else was blocked, thanks to the VPC.

Code

The last thing I’ll touch on is the Terraform code for deploying this application. You can find the whole file here: github.com/AustinGil/linode-vpc-demo/blob/main/terraform/terraform.tf

It’s also worth mentioning that I tried to make this Terraform file reusable for other people (or future me). That required a bit more variable and config setup based around the tfvars file: github.com/AustinGil/linode-vpc-demo/blob/main/terraform/terraform.tfvars.example

Anyway, I’m just going to highlight the key parts.

1. Configure Terraform Provider

First, since I used the Linode Terraform provisioner, it makes sense to know how to set that up:

terraform {
  required_providers {
    linode = {
      source = "linode/linode"
      version = "2.13.0"
    }
  }
}

variable "LINODE_TOKEN" {}

provider "linode" {
  token = var.LINODE_TOKEN
}

This part sets up the provider as well as a variable which Terraform will either ask you for, or you can provide with the tfvars file.

2. Set Up VPC and VPC Subnet

Next, I set up the actual vpc resource along with a subnet resource. This part required a lot of learning on my part.

resource "linode_vpc" "vpc" {
  label = "${local.app_name}-vpc"
  region = var.REGION
}
resource "linode_vpc_subnet" "vpc_subnet" {
  vpc_id = linode_vpc.vpc.id
  label = "${local.app_name}-vpc-subnet"
  ipv4 = "${var.VPC_SUBNET_IP}"
}

Servers can only be added to VPCs in the same region. At the time of writing, there are thirteen regions where VPCs are supported. For the most up-to-date details, refer to the docs: linode.com/docs/products/networking/vpc/.

I tried deploying my servers to San Francisco and ran into errors several times before realizing that it wasn’t an available region. So I went with Seattle ("us-sea") instead.

The subnets were also a learning point for me. As a web application developer, I haven’t had to do much networking, so when I was asked to provide “The IPv4 range of this subnet in CIDR format” I had to research.

Turns out, there are three IPv4 address ranges that are reserved for private networks (such as a VPC):

10.0.0.0 – 10.255.255.255
172.16.0.0 – 172.31.255.255
192.168.0.0 – 192.168.255.255

You have to choose one of these three options, but you have to use CIDR format, which is a way of representing the IP range you want to use. Don’t ask me for more details, because that’s all I know. Akamai has more documentation around the subnets. I just went with 10.0.0.0/24.

Every server in the private network will have an IPv4 address within that range.

3. Set Up Application Servers

For Terraform to deploy my application servers, I used the linode_instance resource. I also used the stackscript resource to create a reusable deployment script for installing and configuring software. It’s like a Bash script that lives in your Akamai cloud dashboard that you can reuse on new servers.

I won’t include the code here, but it installs Node.js 20 via NVM, installs PM2, clones my project repo, runs the app, and sets up Caddy. You can view the StackScript contents in the source code, but I want to focus on the Terraform stuff.

resource "linode_instance" "application1" {
  depends_on = [
    linode_instance.database1
  ]
  image = "linode/ubuntu20.04"
  type = "g6-nanode-1"
  label = "${local.app_name}-application1"
  group = "${local.app_name}-group"
  region = var.REGION
  authorized_keys = [ linode_sshkey.ssh_key.ssh_key ]

  stackscript_id = linode_stackscript.configure_app_server.id
  stackscript_data = {
    "GIT_REPO" = var.GIT_REPO,
    "START_COMMAND" = var.START_COMMAND,
    "DOMAIN" = var.DOMAIN1,
    "NODE_PORT" = var.NODE_PORT,
    "DB_HOST" = linode_instance.database1.ip_address,
    "DB_PORT" = var.DB_PORT,
    "DB_NAME" = var.DB_NAME,
    "DB_USER" = var.DB_USER,
    "DB_PASS" = var.DB_PASS,
  }
}
resource "linode_instance" "application2" {
  depends_on = [
    linode_instance.database2
  ]
  image = "linode/ubuntu20.04"
  type = "g6-nanode-1"
  label = "${local.app_name}-application2"
  group = "${local.app_name}-group"
  region = var.REGION
  authorized_keys = [ linode_sshkey.ssh_key.ssh_key ]

  stackscript_id = linode_stackscript.configure_app_server.id
  stackscript_data = {
    "GIT_REPO" = var.GIT_REPO,
    "START_COMMAND" = var.START_COMMAND,
    "DOMAIN" = var.DOMAIN2,
    "NODE_PORT" = var.NODE_PORT,
    "DB_HOST" = var.DB_PRIVATE_IP,
    "DB_PORT" = var.DB_PORT,
    "DB_NAME" = var.DB_NAME,
    "DB_USER" = var.DB_USER,
    "DB_PASS" = var.DB_PASS,
  }

  interface {
    purpose = "public"
  }
  interface {
    purpose   = "vpc"
    subnet_id = linode_vpc_subnet.vpc_subnet.id
  }
}

Configuring the two resources is almost identical, with only a few significant things to note:

Application #2 includes configuration to add it to the VPC.
The StackScript needs the IP address for the database. Application #1 uses the public IP address from database #1 (linode_instance.database1.ip_address). Application #2 uses a variable (var.DB_PRIVATE_IP). This variable will come up later, but it’s the private IP address assigned to database #2, running within the VPC. This can be manually assigned, so I set it to 10.0.0.3.

Also note that they are deployed to the same region as the VPC, for the reasons I mentioned above.

4. Set Up Database Servers

The databases are also set up using the linode_instance and linode_stackscript resources. Once again, I’ll skip the StackScript contents which you can find in the repo. It installs Postgres, sets up the database and credentials, and provides some configuration options.

resource "linode_instance" "database1" {
  image = "linode/ubuntu20.04"
  type = "g6-nanode-1"
  label = "${local.app_name}-db1"
  group = "${local.app_name}-group"
  region = var.REGION

  authorized_keys = [ linode_sshkey.ssh_key.ssh_key ]
  stackscript_id = linode_stackscript.configure_db_server.id
  stackscript_data = {
    "DB_NAME" = var.DB_NAME,
    "DB_USER" = var.DB_USER,
    "DB_PASS" = var.DB_PASS,
    "PG_HBA_ENTRY" = "host all all all md5"
  }
}
resource "linode_instance" "database2" {
  image = "linode/ubuntu20.04"
  type = "g6-nanode-1"
  label = "${local.app_name}-db2"
  group = "${local.app_name}-group"
  region = var.REGION
  authorized_keys = [ linode_sshkey.ssh_key.ssh_key ]

  stackscript_id = linode_stackscript.configure_db_server.id
  stackscript_data = {
    "DB_NAME" = var.DB_NAME,
    "DB_USER" = var.DB_USER,
    "DB_PASS" = var.DB_PASS,
    "PG_HBA_ENTRY" = "host all all samenet md5"
  }

  interface {
    purpose = "public"
  }
  interface {
    purpose   = "vpc"
    subnet_id = linode_vpc_subnet.vpc_subnet.id
    ipv4 {
      vpc = var.DB_PRIVATE_IP
    }
  }
}

As with the application servers, the two database servers are mostly the same, with just a couple of key differences:

The second database includes configuration to add it to the VPC.
Different settings are written to the Client Authentication file (pg_hba.conf). Database #1 allows all internet connections ("host all all all md5") while database #2 only allows access from the same network ("host all all samenet md5").

It’s also worth noting that we explicitly assign the server’s private IP address when configuring the VPC settings (var.DB_PRIVATE_IP). This is the same static value that was given to the application server so it can connect to the database from within the VPC.

Closing

Hopefully this post has opened our eyes to what VPCs are, why they’re cool, and when you might consider one. It’s like having your own little private internet. It’s not strictly a replacement for VLANs or firewalls, but it’s a great addition to any existing security practice, or at least something to keep in the back of your head.

Building out the demo was interesting in itself, and there were a lot of things that were totally new to me. I spent a lot of time learning:

What VPCs are and how they work.
It was my first time using Terraform, so that involved installation, usage, terminology, etc.
I’ve used Postgres before, but have never had to manually configure client access.
This was my second project using Drizzle and although it was very limited, the migrations process was challenging.
I learned more than I care to know about networking, computer interfaces, IP ranges, and CIDR. I have much more respect for folks working on the networking layer.
Linode StackScripts are also super cool. It was my preferred way to configure a server using Terraform and I want to see how they work otherwise.

There were also a couple of resources that I found particularly helpful:

And in case you want to keep going on with this or related topics, Talia put together some excellent posts recently:

And of course, if you are interested in trying out the VPC or any other Akamai cloud computing products, new users can sign up at linode.com/austingil and get $100 in free credits :)

Thank you so much for reading. If you liked this article, and want to support me, the best ways to do so are to share it, sign up for my newsletter, and follow me on Twitter.

Originally published on austingil.com.

Advanced Architecture for AI Application (AKA AAAA!)

Austin Gil — Fri, 16 Feb 2024 19:25:15 +0000

Surprise! This is a bonus blog post for the AI for Web Devs series I recently wrapped up. If you haven’t read that series yet, I’d encourage you to check it out.

This post will look at the existing project architecture and ways we can improve it for both application developers and the end user.

I’ll be discussing some general concepts, and using specific Akamai products in my examples.

Basic Application Architecture

The existing application is pretty basic. A user submits two opponents, then the application streams back an AI-generated response of who would win in a fight.

The architecture is also simple:

The client sends a request to a server.
The server constructs a prompt and forwards the prompt to OpenAI.
OpenAI returns a streaming response to the server.
The server makes any necessary adjustments and forwards the streaming response to the client.

I used Akamai’s cloud compute services (formerly Linode) but this would be the same for any hosting service, really.

🤵 looks like a server at a fancy restaurant, and 👁️‍🗨️ is “a eye”, or AI. lolz

Technically this works fine, but there are a couple of problems, particularly when users make duplicate requests. It could be faster and more cost-effective to store responses on our server and only go to OpenAI for unique requests.

This assumes we don’t need every single request to be non-deterministic (the same input produces a different output). Let’s assume it’s OK for the same input to produce the same output. After all, a prediction for who would win in a fight wouldn’t likely change.

Add Database Architecture

If we want to store responses from OpenAI, a practical place to put them is in some sort of database that allows for quick and easy lookup using the two opponents. This way, when a request is made, we can check the database first:

The client sends a request to a server.
The server checks for an existing entry in the database that matches the user’s input.
If a previous record exists, the server responds with that data, and the request is complete. Skip the following steps.
If not, the server follows from step three in the previous flow.
Before closing the response, the server stores the OpenAI results in the database.

Dotted lines represent optional requests, and the 💽 kind of looks like a hard disk.

With this setup, any duplicate requests will be handled by the database. By making some of the OpenAI requests optional, we can potentially reduce the amount of latency users experience, plus save money by reducing the number of API requests.

This is a good start, especially if the server and the database exist in the same region. It would make for much quicker response times than going to OpenAI’s servers.

However, as our application becomes more popular, we may start getting users from all over the world. Faster database lookups are great, but what happens if the bottleneck is the latency from the time spent in flight?

We can address that concern by moving things closer to the user.

Bring in Edge Compute

If you’re not already familiar with the term “edge”, this part might be confusing, but I’ll try to explain it simply. Edge refers to content being as close to the user as possible. For some people, that could mean IoT devices or cellphone towers, but in the case of the web, the canonical example is a Content Delivery Network (CDN).

I’ll spare you the details, but a CDN is a network of globally distributed computers that can respond to user requests from the nearest node in the network (something I’ve written about in the past). While traditionally they were designed for static assets, in recent years, they started supporting edge compute (also something I’ve written about in the past).

With edge compute, we can move a lot of our backend logic super close to the user, and it doesn’t stop at compute. Most edge compute providers also offer some sort of eventually-consistent key-value store in the same edge nodes.

How could that impact our application?

The client sends a request to our backend.
The edge compute network routes the request to the nearest edge node.
The edge node checks for an existing entry in the key-value store that matches the user’s input.
If a previous record exists, the edge node responds with that data and the request is complete. Skip the following steps.
If not, the edge node forwards the request to the origin server, which passes it along to OpenAI and yadda yadda yadda.
Before closing the response, the server stores the OpenAI results in the edge key-value store.

The edge node is the blue box and represented by 🔪 because it has an edge, EdgeWorker is Akamai’s edge compute product represented by 🧑‍🏭, and EdgeKV is Akamai’s key-value store represented by 🔑🤑🏪. The edge box is closer to the client than the origin server in the cloud to represent physical distance.

The origin server may not be strictly necessary here, but I think it’s more likely to be there. For the sake of data, compute, and logic flow, this is mostly the same as the previous architecture. The main difference being the previously stored results now exist super close to users and can be returned almost immediately.

(Note: although the data is being cached at the edge, the response is still dynamically constructed. If you don’t need dynamic responses, it may be simpler to use a CDN in front of the origin server and set the correct HTTP headers to cache the response. There’s a lot of nuance here, and I could say more but…well, I’m tired and don’t really want to. Feel free to reach out if you have any questions.)

Now we’re cooking! Any duplicate requests will be responded to almost immediately, while also saving us unnecessary API requests.

This sorts out the architecture for the text responses, but we also have AI-generated images.

Cache Those Images

The last thing we’ll consider today is images. When dealing with images, we need to think about delivery and storage. I’m sure that the folks at OpenAI have their own solutions, but some organizations want to own the entire infrastructure for security, compliance, or reliability reasons. Some may even run their own image generation services instead of using OpenAI.

In the current workflow, the user makes a request that ultimately makes its way to OpenAI. OpenAI generates the image but doesn’t return it. Instead, they return a JSON response with the URL for the image, hosted on OpenAI’s infrastructure. With this response, an <img> tag can be added to the page using the URL, which kicks off another request for the actual image.

If we want to host the image on our own infrastructure, we need a place to store it. We could write the images onto the origin server’s disk, but that could quickly use up the disk space, and we’d have to upgrade our servers, which can be costly. Object storage is a much cheaper solution (I’ve also written about this). Instead of using the OpenAI URL for the image, we could upload it to our own object storage instance and use that URL instead.

That solves the storage question, but object storage buckets are generally deployed to a single region. This echoes the problem we had with storing text in a database. A single region may be far away from users, which could cause a lot of latency.

Having introduced the edge already, it would be pretty trivial to add CDN features for just the static assets (frankly, every site should have a CDN). Once configured, the CDN will pull images from object storage on the initial request and cache them for any future requests from visitors in the same region.

Here’s how our flow for images would look:

Client sends a request to generate an image based on their opponents
Edge compute checks if the image data for that request already exists. If so, it returns the URL.
The image is added to the page with the URL and the browser requests the image.
If the image has been previously cached in the CDN, the browser loads it almost immediately. This is the end of the flow.
If the image has not been previously cached, the CDN will pull the image from the object storage location, cache a copy of it for future requests, and return the image to the client. This is another end of the flow.
If the image data is not in the edge key-value store, the request to generate the image goes to the server and on to OpenAI, which generates the image and returns the URL information. The server starts a task to save the image in the object storage bucket, stores the image data in the edge key-value store, and returns the image data to edge compute.
With the new image data, the client creates the image which creates a new request and continues from step five above.

Content delivery network denoted by delivery truck (🚚) and network signal (📶), and object storage denoted by socks in a box (🧦📦), or objects in storage. This caption is probably not necessary, as I think these are clear, but I’m too proud of my emoji game and require validation. Thank you for indulging me. Carry on.

This last architecture is, admittedly, a little bit more complex, but if your application is going to handle serious traffic, it’s worth considering.

Voilà

Right on! With all those changes in place, we have created AI-generated text and images for unique requests and serve cached content from the edge for duplicate requests. The result is faster response times and a much better user experience (in addition to fewer API calls).

I kept these architecture diagrams applicable across various databases, edge compute, object storage, and CDN providers on purpose. I like my content to be broadly applicable. But it’s worth mentioning that integrating the edge is about more than just performance. There are a lot of really cool security features you can enable as well.

For example, on Akamai’s network, you can have access to things like web application firewall (WAF), distributed denial of service (DDoS) protection, intelligent bot detection, and more. That’s all beyond the scope of today’s post, though.

So for now, I’ll leave you with a big “thank you” for reading. I hope you learned something. And as always, feel free to reach out any time with comments, questions, or concerns.

Thank you so much for reading. If you liked this article, and want to support me, the best ways to do so are to share it, sign up for my newsletter, and follow me on Twitter.

Originally published on austingil.com.

AI for Web Devs: Deploying Your AI App to Production

Austin Gil — Wed, 07 Feb 2024 17:28:40 +0000

Welcome back to the series where we have been building an application with Qwik that incorporates AI tooling from OpenAI. So far we’ve created a pretty cool app that uses AI to generate text and images.

Now, there’s just one more thing to do. It’s launch time!

I’ll be deploying to Akamai‘s cloud computing services (formerly Linode), but these steps should work with any VPS provider. If you don’t already have a hosting provider, you can sign up at linode.com/austingil to get $100 in cloud computing credits.

Let’s do this!

Setup Runtime Adapter

There are a couple of things we need to get out of the way first: deciding where we are going to run our app, what runtime it will run in, and how the deployment pipeline should look.

As I mentioned before, I’ll be deploying to a VPS in Akamai’s connected cloud, but any other VPS should work. For the runtime, I’ll be using Node.js, and I’ll keep the deployment simple by using Git.

Qwik is cool because it’s designed to run in multiple JavaScript runtimes. That’s handy, but it also means that our code isn’t ready to run in production as is. Qwik needs to be aware of its runtime environment, which we can do with adapters.

We can access see and install available adapters with the command, npm run qwik add. This will prompt us with several options for adapters, integrations, and plugins.

For my case, I’ll go down and select the Fastify adapter. It works well on a VPS running Node.js. You can select a different target if you prefer.

Once you select your integration, the terminal will show you the changes it’s about to make and prompt you to confirm. You’ll see that it wants to modify some files, create some new ones, install dependencies, and add some new NPM scripts. Make sure you’re comfortable with these changes before confirming.

Once these changes are installed, your app will have what it needs to run in production. You can test this by building the production assets and running the serve command. (Note: For some reason, npm run build always hangs for me, so I run the client and server build scripts separately).

npm run build.client & npm run build.server & npm run serve

This will build out our production assets and start the production server listening for requests at http://localhost:3000. If all goes well, you should be able to open that URL in your browser and see your app there. It won’t actually work because it’s missing the OpenAI API keys, but we’ll sort that part out on the production server.

Push Changes to Git Repo

As mentioned above, this deployment process is going to be focused on simplicity, not automation. So rather than introducing more complex tooling like Docker containers or Kubernetes, we’ll stick to a simpler, but more manual process: using Git to deploy our code.

I’ll assume you already have some familiarity with Git, and a remote repo you can push to. If not, please go make one now.

You’ll need to commit your changes and push it to your repo.

git commit -am "ready to commit 💍" & git push origin main

Prepare Production Server

If you already have a VPS ready, feel free to skip this section. I’ll be deploying to an Akamai VPS. If you don’t already have an account, feel free to sign up at linode.com/austingil for $100 in free credits.

I won’t walk through the step-by-step process for setting up a server, but in case you’re interested, I chose the Nanode 1 GB shared CPU plan for $5/month with the following specs:

Operating system: Ubuntu 22.04 LTS
Location: Seattle, WA
CPU: 1
RAM: 1 GB
Storage: 25 GB
Transfer: 1 TB

Choosing different specs shouldn’t make a difference when it comes to running your app, although some of the commands to install any dependencies may be different. If you’ve never done this before, then try to match what I have above. You can even use a different provider, as long as you’re deploying to a server to which you have SSH access.

Once you have your server provisioned and running, you should have a public IP address that looks something like 172.100.100.200. You can log into the server from your terminal with the following command:

ssh root@172.100.100.200

You’ll have to provide the root password if you have not already set up an authorized key.

We’ll use Git as a convenient tool to get our code from our repo into our server, so that will need to be installed. But before we do that, I always recommend updating the existing software. We can do the update and installation with the following command.

sudo apt update && sudo apt install git -y

Our server also needs Node.js to run our app. We could install the binary directly, but I prefer to use a tool called NVM, which allows us to easily manage Node versions. We can install it with this command:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash

And once NVM is installed, you can install the latest version of Node with:

nvm install node

Note that the terminal may say that NVM is not installed. If you exit the server and sign back in, it should work.

Upload, Build, & Run App

With our server set up, it’s time to get our code installed. With Git, it’s relatively easy. We can copy our code into our server using the clone command. You’ll want to use your own repo, but it should look something like this:

git clone https://github.com/AustinGil/versus.git

Our source code is now on the server, but it’s still not quite ready to run. We still need to install the NPM dependencies, build the production assets, and provide any environment variables.

Let’s do!

First, navigate to the folder where you just cloned the project. I used:

cd versus

The install is easy enough:

npm install

The build command is:

npm run build

However, if you have any type-checking or linting errors, it will hang there. You can either fix the errors (which you probably should) or bypass them and build anyway with this:

npm run build.client & npm run build.server

The latest version of the project source code has working types if you want to check that.

The last step is a bit tricky. As we saw above, environment variables will not be injected from the .env file when running the production app. Instead, we can provide them at runtime right before the serve command like this:

OPENAI_API_KEY=your_api_key npm run serve

You’ll want to provide your own API key there in order for the OpenAI requests to work.

Also, for Node.js deployments, there’s an extra, necessary step. You must also set an ORIGIN variable assigned to the full URL where the app will be running. Qwik needs this information to properly configure their CSRF protection.

If you don’t know the URL, you can disable this feature in the /src/entry.preview.tsx file by setting the createQwikCity options property checkOrigin to false:

export default createQwikCity({
  render,
  qwikCityPlan,
  checkOrigin: false
});

This process is outlined in more detail in the docs, but it’s recommended not to disable, as CSRF can be quite dangerous. And anyway, you’ll need a URL to deploy the app anyway, so better to just set the ORIGIN environment variable. Note that if you make this change, you’ll want to redeploy and rerun the build and serve commands.

If everything is configured correctly and running, you should start seeing the logs from Fastify in the terminal, confirming that the app is up and running.

{"level":30,"time":1703810454465,"pid":23834,"hostname":"localhost","msg":"Server listening at http://[::1]:3000"}

Unfortunately, accessing the app via IP address and port number doesn’t show the app (at least not for me). This is likely a networking issue, but also something that will be solved in the next section, where we run our app at the root domain.

The Missing Steps

Technically, the app is deployed, built, and running, but in my opinion, there is a lot to be desired before we can call it “production ready.” Some tutorials would assume you know how to do the rest, but I don’t want to do you like that. We’re going to cover:

Running the app in background mode
Restarting the app if the server crashes
Accessing the app at the root domain
Setting up an SSL certificate

One thing you will need to do for yourself is buy the domain name. There are lots of good places. I’ve been a fan of Porkbun and Namesilo. I don’t think there’s a huge difference for which registrar you use, but I like these because they offer WHOIS privacy and email forwarding at no extra charge on top of their already low prices.

Before we do anything else on the server, it’ll be a good idea to point your domain name’s A record (@) to the server’s IP address. Doing this sooner can help with propagation times.

Now, back in the server, there’s one glaring issue we need to deal with first. When we run the npm run serve command, our app will run as long as we keep the terminal open. Obviously, it would be nice to exit out of the server, close our terminal, and walk away from our computer to go eat pizza without the app crashing. So we’ll want to run that command in the background.

There are plenty of ways to accomplish this: Docker, Kubernetes, Pulumis, etc., but I don’t like to add too much complexity. So for a basic app, I like to use PM2, a Node.js process manager with great features, including the ability to run our app in the background.

From inside your server, run this command to install PM2 as a global NPM module:

npm install -g pm2

Once it’s installed, we can tell PM2 what command to run with the “start” command:

pm2 start "npm run serve"

PM2 has a lot of really nice features in addition to running our apps in the background. One thing you’ll want to be aware of is the command to view logs from your app:

pm2 logs

In addition to running our app in the background, PM2 can also be configured to start or restart any process if the server crashes. This is super helpful to avoid downtime. You can set that up with this command:

pm2 startup

Ok, our app is now running, and will continue to run after a server restart. Great!

But we still can’t get to it. Lol!

My preferred solution is using Caddy. This will resolve the networking issues, work as a great reverse proxy, and takes care of the whole SSL process for us. We can follow the install instructions from their documentation and run these five commands:

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy

Once that’s done, you can go to your server’s IP address and you should see the default Caddy welcome page:

Progress!

In addition to showing us something is working, this page also gives us some handy information on how to work with Caddy.

Ideally, you’ve already pointed your domain name to the server’s IP address. Next, we’ll want to modify the Caddyfile:

sudo nano /etc/caddy/Caddyfile

As their instructions suggest, we’ll want to replace the :80 line with our domain (or subdomain), but instead of uploading static files or changing the site root, I want to remove (or comment out) the root line and enable the reverse_proxy line, pointing the reverse proxy to my Node.js app running at port 3000.

versus.austingil.com {
        reverse_proxy localhost:3000
}

After saving the file and reloading Caddy (systemctl reload caddy), the new Caddyfile changes should take effect. Note that it may take a few moments before the app is fully up and running. This is because one of Caddy’s features is to provision a new SSL certificate for the domain. It also sets up the automatic redirect from HTTP to HTTPS.

So now if you go to your domain (or subdomain), you should be redirected to the HTTPS version running a reverse-proxy in front of your generative AI application which is resilient to server crashes.

How awesome is that!?

Using PM2 we can also enable some load-balancing in case you’re running a server with multiple cores. The full PM2 command including environment variables and load-balancing might look something like this:

OPENAI_API_KEY=your_api_key ORIGIN=example.com pm2 start "npm run serve" -i max

Note that you may need to remove the current instance from PM2 and rerun the start command, you don’t have to restart the Caddy process unless you change the Caddy file, and any changes to the Node.js source code will require a rebuild before running it again.

Hell yeah! We did it!

Alright, that’s it for this blog post, and this series. I sincerely hope you enjoyed both and learned some cool things. Today, we covered a lot of things you need to know to deploy an AI-powered application:

Runtime adapters
Building for production
Environment variables
Process managers
Reverse-proxies
SSL certificates

If you missed any of the previous posts, be sure to go back and check them out.

I’d love to know what you thought about the whole series. If you want, you can play with the app I built at versus.austingil.com. Let me know if you deployed your own app, and I’ll link to it from here. Also, if you have ideas for topics you’d like me to discuss in the future I’d love to hear them :)

UPDATE: If you liked this project and are curious to see what it might look like as a SvelteKit app, check out this blog post by Tim Smith where he converts this existing app over.

Thank you so much for reading. If you liked this article, and want to support me, the best ways to do so are to share it, sign up for my newsletter, and follow me on Twitter.

Originally published on austingil.com.

AI for Web Devs: Addressing Bugs, Security, & Reliability

Austin Gil — Wed, 31 Jan 2024 17:58:48 +0000

Welcome back to this series where we have been learning how to build web applications with AI.

So far in this series, we’ve created a working app that uses AI to determine who would win in a fight between two user-provided opponents and generates text responses and images. It’s working, but we’ve been following the happy path.

In this post, we’re going to talk about what happens when things don’t follow the happy path by accounting for error handling and security concerns.

Dealing With Bad HTTP Requests

The first issue to deal with is around our HTTP requests. So far, we’ve just been assuming that the requests from the client to the server will just work.

const response = await jsFormSubmit(form)

// Do something with response

This is a mistake. We need to account for situations where the server experiences an error or returns a bad status code.

In a real-world application, we would want a sophisticated notification service to communicate to users in the event of different errors (server error, validation error, authorization error, not found error, etc.). It would also be good to tie in an error and bug-tracking software so you get notified of any issues.

For today’s example, the discount brand will have to do. We’ll check the response ok property and in case of a bad response, we’ll just alert the user that there was an error.

const response = await jsFormSubmit(form)

if (!response.ok) {
  state.isLoading = false
  alert("The request experienced an issue.")
  return
}

The code above only accounts for the HTTP request between the client and the server. Don’t forget, we have another request between the server and OpenAI.

Consider the scenario where OpenAI returns a bad status code. How should we communicate that to the end user on the client? This is also tricky and unique to each app. For the sake of convenience, we can do a similar check on the response.ok property. In the event of a bad request, you’ll once again want to report on the error, and maybe respond to the user with the same status code. I would recommend against passing the response message to the client in case it contains sensitive data.

const response = await fetch('https://api.openai.com/v1/chat/completions', {
  // ... fetch options
})
if (!response.ok) {
  reportError(response)
  throw error(response.status, 'ERROR: Service unavailable');
}

This error handling is very rudimentary, and I’ll leave it like that because unfortunately, I’ve never seen two apps that handle errors the same way. It’s highly subjective. Suffice it to say that you should spend time thinking about how your app should behave in the event of an error. How do you report it internally, and how do you communicate it to users?

And what happens when users deliberately try to break something…?

Dealing With Bad User Input

In addition to following the happy path where we assumed every HTTP request would always work, we assumed every user was benevolent. This is another mistake. Sometimes users are malicious. Oftentimes, they are just plain silly. We should account for both.

Any time you receive user-submitted data, you have to validate it. In our app, we expect the user to submit two opponents. What happens if they submit just one, or none, or empty strings? We probably should catch that early before sending the malstructured prompt to OpenAI.

We can add the HTML required attribute to the textareas to tell the form that both inputs need to be filled before the form can be submitted. If the user tries to submit the form without the controls filled in, the browser will prevent the submission, focus on the first invalid input, and provide a little error message telling the user what the problem is.

This is good for the user experience because it provides some early feedback, but client-side validation is easily bypassed, so we have to also validate data on the server. Fortunately, there are some very good validation libraries available that can help with this. My favorite is called Zod. We can install it with npm install zod.

Zod allows us to define a schema that will be used to validate input data. If the input doesn’t match the schema, Zod can either throw an error or report it.

In our app, we are receiving the user input through the requestEvent.parseBody() method, which returns the submitted form data as an object containing opponent1 and opponent2 properties. So, what we need to do is create a validation schema, then pass the form data into one of the schema validation methods.

I prefer not throwing an error, and instead getting an object back with the validation information. That way, I can add the logic myself to deal with bad data.

Inside my onPost middleware, before doing too much work, let’s make sure we have the right data:

import { z } from 'zod'
// ...

export const onPost: RequestHandler = async (requestEvent) => {
  const formData = await requestEvent.parseBody()

  const schema = z.object({
    opponent1: z.string().min(1),
    opponent2: z.string().min(1),
  })
  const validation = schema.safeParse(formData)

  if (!validation.success) {
    requestEvent.json(400, {
      errors: validation.error.issues
    })
    return 
  }

  // Continue with OpenAI API request and response
}

In the code above, I create an Object schema that should have two properties, opponent1 and opponent2. Both properties are required, must be strings, and cannot be empty. Passing the form data into the schema’s safeParse() method will return an object that can tell me if the validation was successful, what the error was, if any, and the validated data.

In the event of invalid data, I return early from the request handler with an HTTP 400 error response explaining the errors. 400 is the Bad Request status code.

One other thing I like to change is how I use the form data once it’s been validated. Zod also provides a data property on the returned object from safeParse.

const prompt = await promptTemplate.format({
  opponent1: validation.data.opponent1,
  opponent2: validation.data.opponent2
})

In our example, it doesn’t make too much of a difference whether we use this or the form data directly, but it’s nice to get in the habit of using the data property because Zod will coerce the data to the appropriate format. Form data and query parameters are almost always received as strings, but if your Zod schema was expecting a number, it will try to coerce it for you, turning something like the string "420" into the number 420.

So that covers users sending missing data or not enough data, but what about sending too much?

By giving users unbounded input length that gets injected directly into our prompt, we are opening the gates for users to create massive prompts that would require a lot of tokens and cost us money. Why don’t we add a maximum length to the inputs to something more appropriate for this app?

We can add a maximum length to both the server-side validation schema and the client-side validation attributes.

// Reusable constant
const MAX_INPUT_LENGTH = 50

// In our schema
const schema = z.object({
  opponent1: z.string().min(1).max(MAX_INPUT_LENGTH),
  opponent2: z.string().min(1).max(MAX_INPUT_LENGTH),
})

// In our template
<Input
  maxLength={MAX_INPUT_LENGTH}
/>

By reducing the amount of data a user can provide, we are reducing the amount of tokens that our API request could potentially use.

This step also limits the amount of flexibility a user has to manipulate our prompt. Consider the fact that a user could provide an “opponent” that actually contains malicious instructions for the app.

This segues nicely to a very important security concern for AI applications specifically.

Dealing With Injection Attacks

We’re doing some basic validation that the data we get from the user is the right type, but we aren’t checking the content that they send us. We’re just blindly sticking it into our prompt and sending it off, and this opens us up to a very interesting kind of attack called a prompt injection attack.

If you’ve done any work building applications with SQL, this may sound similar to an SQL injection attack, and that’s because it is. An SQL injection attack is when a user submits some data of the right type, but containing SQL commands that could be harmful if run.

Here’s an example. Let’s say our app had some SQL logic to select a user by ID based on the input provided:

const query = "SELECT * FROM Users WHERE UserId = " + inputId;

An attacker could provide the string '1 OR 1=1' as the input and would return the information of all the users. This is bad, but you can avoid it by using parameterized queries, stored procedures, or escaping user input. Unless you’re writing raw SQL queries, most tools protect against injection. If you’re interested, here’s more on prevention from OWASP.

Prompt injection is a bit different because prompts don’t have a structured language with specific keywords you can search for. Literally anything you (or the user) provide is a valid prompt, and there’s no easy delineation between what you write and what a user writes.

To their credit, OpenAI does include tactics for prompt engineering that encourage using delimiters to indicate separate parts of the input. This way, you can help the AI identify the different identities, like the system and the user.

It might look like this:

Translate the text after the delimiting characters "~~~~~":

~~~~~

[text to be translated]

This is a big improvement as it provides a clearer separation between the system and the user input. Still, it’s not without gaps.

Simon Willison has pointed out several examples of prompt injection attacks and why it may never be a solved problem:

It’s kind of scary and should make you think twice about building AI-powered apps. But this is a bit of a Pandora’s box. Even with its inherent vulnerabilities, AI is here to stay. My suggestion is to keep yourself up to date on attack vectors and incorporate several layers of security.

Conclusion

Building applications with the happy path in mind is great, but we must also address the not-so-happy path. It’s important to familiarize ourselves with points of failure and vulnerabilities and address them appropriately. This makes our applications more secure for our users and more reliable.

In this post, we discussed:

What happens when our app experiences HTTP errors?
How do we validate user input?
What are some security concerns specifically for AI apps?

This is not a comprehensive list, but I hope it serves as a good starting point. With this work out of the way, I think we are ready to launch our app to the world. We’ll do that in the next post.

Thank you so much for reading. If you liked this article, and want to support me, the best ways to do so are to share it, sign up for my newsletter, and follow me on Twitter.

Originally published on austingil.com.

AI for Web Devs: AI Image Generation

Austin Gil — Mon, 29 Jan 2024 21:48:18 +0000

Welcome back to this series where we are learning how to integrate AI products into web application.

In this post, we are going to use AI to generate images. Before we get to that, let’s add a dialog component to our app. This is a modal-like pop-up designed to show content and be dismissed with the mouse or keyboard.

Create A Dialog Component

Before showing an image, we need a place to put it. I think it would be nice to have a dialog that pops up to showcase the image. This is a good opportunity to spend more time with Qwik components.

By convention, our new component should go in the ./src/components folder. I’ll call mine Dialog.jsx (or .tsx if you prefer TypeScript). A Qwik component file should have a default export of a Qwick component. To create one, we use the component$ function from @builder.io/qwik. This function takes a function component that returns JSX:

import { component$ } from "@builder.io/qwik";

export default component$((props) => {

  return (
    <div>
      <!-- component markup -->
    </div>
  )
})

At the moment, it’s not very useful. Let’s plan out the API design for this dialog. It should have the following characteristics:

Opened by clicking on a button.
Can be opened programmatically from the parent.
Closed by pressing Esc key.
Closed by clicking the dialog background.
Can be closed programmatically from the parent.

HTML has a <dialog> element which is great, but it doesn’t quite offer the API that I’m looking for. With a little bit of work, we can fill in the gaps.

Let’s start with a component that provides a <button> element for controlling the dialog and the <dialog> element that will contain any content you put inside. Clicking the button should trigger the dialog.showModal() method. Clicking outside the dialog should trigger the dialog.close() method. Pressing the Esc key closes the dialog already, so that’s sorted. To trigger the dialog methods, we need access to the DOM node, which we can get by using a ref and a signal. In our dialog component, we can add content using a <Slot>, a flexible space inside the dialog where you can put content. We should also provide a btnText prop, to allow for customization of the text on the button that opens the dialogue.

Here’s what I have so far:

import { component$, useSignal, Slot } from "@builder.io/qwik";

export default component$(({ btnText }) => {
  const dialogRef = useSignal()

  return (
    <div>
      <button onClick$={() => dialogRef.value.showModal()}>
        {btnText}
      </button>

      <dialog
        ref={dialogRef}
        onClick$={(event) => {
          if (event.target.localName !== 'dialog') return
          dialogRef.value.close()
        }}
      >
        <div class="p-2">
          <Slot></Slot>
        </div>
      </dialog>
    </div>
  )
})

Note the additional <div> inside the <dialog>. This lets me add some padding, but more importantly, it helps me track whether a click on the dialog happened on the background (the dialog element) or on the content (the div and children). This lets us close the dialog only when the background is clicked.

That gets us some basic functionality, but for my use case, I want to be able to programmatically open the dialog from the parent, not just when the toggle button is clicked. For that, we need a small refactor.

To control the dialog from a parent, we need to add a new prop and trigger the dialog methods as it changes. Rather than repeat the open/close functionality for internal and external changes let’s create a local state with useStore() to track whether the dialog should be shown or not. Then we can simply toggle the state and respond to those changes using a Qwik task (it’s like useEffect in React). If the open prop from the parent changes, we need to respond to that change using another task, but this time we’ll use useVisibleTask$(). We also need to provide a way for the parent component to be aware of changes to the dialog’s visibility. We can do that by providing a custom onClose$ prop that will call the function whenever the dialog closes. And lastly, if we’re providing programmatic control from the parent, we may want to provide a way to hide the <button>.

import { component$, useSignal, Slot, useStore, useTask$, useVisibleTask$ } from "@builder.io/qwik";

export default component$(({ btnText, open, onClose$ }) => {
  const dialogRef = useSignal()
  const state = useStore({
    isOpen: false,
  })

  useTask$(({ track }) => {
    track(() => state.isOpen)

    const dialog = dialogRef.value
    if (!dialog) return

    if (state.isOpen) {
      dialog.showModal()
    } else {
      dialog.close()
      onClose$ && onClose$()
    }
  })
  useVisibleTask$(({ track }) => {
    track(() => open)
    state.isOpen = open || false
  })

  return (
    <div>
      {btnText && (
        <button onClick$={() => state.isOpen = true}>
          {btnText}
        </button>
      )}

      <dialog
        ref={dialogRef}
        onClick$={(event) => {
          if (event.target.localName !== 'dialog') return
          state.isOpen = false
        }}
      >
        <div class="p-2">
          <Slot></Slot>
        </div>
      </dialog>
    </div>
  )
})

This is a lot better, but there’s still a little work to do. I like to make sure my components are accessible and typed. In this case, since we’re using a button to control the visibility of the dialog, it makes sense to add aria-controls and aria-expanded attributes to the button. To connect them to the dialog, the dialog needs an ID, which we can either take from the props or dynamically generate. Lastly, pressing escape will close the dialog, but we also need to track the dialog’s native "close" event by attaching an onClose$ handler.

Here is my finished component, including JSDoc type definitions:

import {component$, useSignal, Slot, useStore, useTask$, useVisibleTask$ } from "@builder.io/qwik";
import { randomString } from "~/utils.js";

/**
 * @typedef {HTMLAttributes<HTMLDialogElement>} DialogAttributes
 *
 * @type {Component<DialogAttributes  & {
 * toggle: string|false,
 * open?: Boolean,
 * onClose$?: import('@builder.io/qwik').PropFunction<() => any>
 * }>}
 */
export default component$(({ toggle, open, onClose$, ...props }) => {
  const id = props.id || randomString(8)

  const dialogRef = useSignal()
  const state = useStore({
    isOpen: false,
  })

  useTask$(({ track }) => {
    track(() => state.isOpen)

    const dialog = dialogRef.value
    if (!dialog) return

    if (state.isOpen) {
      dialog.showModal()
    } else {
      dialog.close()
      onClose$ && onClose$()
    }
  })
  useVisibleTask$(({ track }) => {
    track(() => open)
    state.isOpen = open || false
  })

  return (
    <div>
      {toggle && (
        <button aria-controls={id} aria-expanded={state.isOpen} onClick$={() => state.isOpen = true}>
          {toggle}
        </button>
      )}

      <dialog
        ref={dialogRef}
        id={id}
        onClick$={(event) => {
          if (event.target.localName !== 'dialog') return
          state.isOpen = false
        }}
        onClose$={() => state.isOpen = false}
        {...props}
      >
        <div class="p-2">
          <Slot></Slot>
        </div>
      </dialog>
    </div>
  )
})

Cool! Now that we have a working dialog component, we need to put something in it.

Generate AI Images with OpenAI

Once the AI has determined a winner between opponent1 and opponent2, it would be cool to offer an image of them in combat. So why not add a button that says “Show me” after the results are available?

After the text in the template, we could add a conditional like this:

{state.winner && (
  <button>
    Show me
  </button>
)}

Awesome! It’s just too bad it doesn’t do anything…

To actually generate the AI image, we need to make another API request to OpenAI, which means we need another API endpoint in our backend. We’ve already assigned a request handler to the current route. Let’s add a new route to handle GET requests to /ai-image by adding a new file in /src/routes/ai-image/index.js.

In many cases, a new route may need to return HTML from the server to generate a page. That’s not the case for this route. This will only ever return JSON, so it doesn’t need to be a JSX file or return a component.

Instead, we can export a custom middleware like we did for post requests on the first page. To do that, we create a named export called onGet that can look something like this:

export const onGet = async (requestEvent) => {
  requestEvent.send(200, JSON.stringify({ some: 'data' }))
}

Now, to get the route working, we need to do the following:

Grab opponent1 and opponent2 from the request (we’ll use query parameters).
Use the opponents to construct a prompt using LangChain templates.
Create a body for the OpenAI API request containing the prompt and the image size we want (I’ll use "512x512").
Create the authenticated HTTP request to OpenAI.
Respond to the initial request with the URL of the generated image.

For more details on working with images through OpenAI, refer to their documentation.

Here’s how my implementation looks:

import { PromptTemplate } from 'langchain/prompts'

const mods = [
  'cinematic',
  'high resolution',
  'epic',
];

const promptTemplate = new PromptTemplate({
  template: `{opponent1} and {opponent2} in a battle to the death, ${mods.join(', ')}`,
  inputVariables: ['opponent1', 'opponent2']
})

/** @type {import('@builder.io/qwik-city').RequestHandler} */
export const onGet = async (requestEvent) => {
  const OPENAI_API_KEY = requestEvent.env.get('OPENAI_API_KEY')

  const opponent1 = requestEvent.query.get('opponent1')
  const opponent2 = requestEvent.query.get('opponent2')

  const prompt = await promptTemplate.format({
    opponent1: opponent1,
    opponent2: opponent2
  })

  const body = {
    prompt: prompt,
    size: '512x512',
  }

  const response = await fetch('https://api.openai.com/v1/images/generations', {
    method: 'post',
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${OPENAI_API_KEY}`,
    },
    body: JSON.stringify(body)
  })
  const results = await response.json()

  requestEvent.send(200, JSON.stringify(results.data[0]))
}

It’s worth noting that the response from OpenAI should look something like this:

{
  "created": 1589478378,
  "data": [
    {
      "url": "https://..."
    },
    {
      "url": "https://..."
    },
  ]
}

Also worth mentioning is the lack of validation for the opponents. It’s always a good idea to validate user input before processing it. Why don’t you try addressing that as an additional challenge?

Provide AI Images with Art Direction

In the code example above, you may have noticed the mods array that gets joined and appended to the end of the prompt. This is worth a callout.

Generative images are tricky because the same prompt can return drastically different results. You might get a cartoon or an oil painting or a sketch. So it’ important to include a couple of hints to the AI to match the aesthetic of your application.

I found that using an array with several different options allowed me to easily toggle off various features. In fact, in my actual project, I keep a much longer list of options, organized by style, format, quality, and effect.

const mods = [
  /** Style */
  // 'Abstract',
  // 'Academic',
  // 'Action painting',
  // 'Aesthetic',
  // 'Angular',
  // 'Automatism',
  // 'Avant-garde',
  // 'Baroque',
  // 'Bauhaus',
  // 'Contemporary',
  // 'Cubism',
  // 'Cyberpunk',
  // 'Digital art',
  // 'photo',
  // 'vector art',
  // 'Expressionism',
  // 'Fantasy',
  // 'Impressionism',
  // 'kiyo-e',
  // 'Medieval',
  // 'Minimal',
  // 'Modern',
  // 'Pixel art',
  // 'Realism',
  // 'sci-fi',
  // 'Surrealism',
  // 'synthwave',
  // '3d-model',
  // 'analog-film',
  // 'anime',
  // 'comic-book',
  // 'enhance',
  // 'fantasy-art',
  // 'isometric',
  // 'line-art',
  // 'low-poly',
  // 'modeling-compound',
  // 'origami',
  // 'photographic',
  // 'tile-texture',

  /** Format */
  // '3D render',
  // 'Blender Model',
  // 'CGI rendering',
  'cinematic',
  // 'Detailed render',
  // 'oil painting',
  // 'unreal engine 5',
  // 'watercolor',
  // 'cartoon',
  // 'anime',
  // 'colored pencil',

  /** Quality */
  'high resolution',
  // 'high-detail',
  // 'low-poly',
  // 'photographic',
  // 'photorealistic',
  // 'realistic',

  /** Effects */
  // 'Beautiful lighting',
  // 'Cinematic lighting',
  // 'Dramatic',
  // 'dramatic lighting',
  // 'Dynamic lighting',
  'epic',
  // 'Portrait lighting',
  // 'Volumetric lighting',
];

I’ve also found that it’s better to include these modifiers at the end of the prompt, otherwise they can be forgotten.

Request an AI-Generated Image

Now that our endpoint is ready, we can start using it. We need to send an HTTP request with query parameters including opponent1 and opponent2. We can pull those values from the <textarea>s on demand, but I prefer to maintain some reactive state that gets updated any time a user types into the <textarea>s .

Let’s modify our state to include properties for opponent1 and opponent2:

const state = useStore({
  isLoading: false,
  text: '',
  winner: '',
  opponent1: '',
  opponent2: '',
})

Next, let’s add an onInput$ event handler that will update the state. The event handler should probably also clear any previous text results and winners. Note that we need to do this for both inputs.

<Input
  label="Opponent 1"
  name="opponent1"
  value={state.opponent1}
  class={{
    rainbow: state.winner === 'opponent1'
  }}
  onInput$={(event) => {
    state.winner = ''
    state.text = ''
    state.opponent1 = event.target.value
  }}
/>

Now that we have the values conveniently available, we can construct the HTTP request. We could do this when the “Show me” button is clicked, but we already made the jsFormSubmit function in the third post of the series. Might as well reuse it. All it needs is a <form> with the data to send.

Let’s create a form that submits to our /ai-image route, prevents the default behavior, and submits the data with jsFormSubmit instead. We can use hidden inputs to put the data in the form without impacting the UI.

{state.winner && (
  <form
    action="/ai-image"
    preventdefault:submit
    onSubmit$={async (event) => {
      const form = event.target
      console.log(await jsFormSubmit(form))
    }}
    class="mt-4"
  >
    <input
      type="hidden"
      name="opponent1"
      value={state.opponent1}
      required
    />
    <input
      type="hidden"
      name="opponent2"
      value={state.opponent2}
      required
    />
    <button type="submit">
      Show me
    </button>
  </form>
)}

It looks the same as it did before, but now it actually does something. I would show you a screenshot, but it’s pretty much just a button. Unremarkable, but effective.

Show the Image in the Dialog

The last step is to put it all together.

The user submits the two opponents and the AI returns a winner.
The image generation <form> will be available, showing the user the “Show me” button.
When the user clicks the button, the API request gets submitted.
At the same time, we’ll programmatically open the dialog with some initial loading state.
When the request returns, we’ll display the image.

For that, I’ll create a new store for the image state using useStore. It’ll hold a showDialog state, an isLoading state, and the url. I’m also going to move the form’s submit handler into a dedicated function called onSubmitImg so it’s not nested in the template and all the logic can live together.

The body of the obSubmitImg function will activate the dialog, set the loading state, submit the form with jsFormSubmit, set the image URL from the results, and disable the loading state.

const imgState = useStore({
  showDialog: false,
  isLoading: false,
  url: ''
})
const onSubmitImg = $(async (event) => {
  imgState.showDialog = true
  imgState.isLoading = true

  const form = event.target

  const response = await jsFormSubmit(form)
  const results = await response.json()

  imgState.url = results.url
  imgState.isLoading = false
})

Man, I love that jsFormSubmit function! It lets the HTML provide the declarative HTTP logic and simplifies the business logic.

Ok – with the state setup, the last thing to do is connect the <Dialog> component. Since we’ll be opening it programmatically, we don’t need the built-in button. We can disable that by making the btnText prop falsy. We can connect the open prop to imgState.showDialog. We’ll also want to update that state when the dialog closes via the onClose$ event. And the contents of the dialog should either show something for the loading state or show the generated image.

<Dialog
  btnText={false}
  open={imgState.showDialog}
  onClose$={() => imgState.showDialog = false}
>
  {imgState.isLoading && (
    "Working on it..."
  )}
  {!imgState.isLoading && imgState.url && (
    <img src={imgState.url} alt={`An epic battle between ${state.opponent1} and ${state.opponent2}`} />
  )}
</Dialog>

Unfortunately, we now have a button that programmatically opens a dialog element without accessibility considerations. I thought about including the same ARIA attributes as we have on the toggle, but if I’m honest, I’m not sure if a submit button should control a dialog.

In this case, I’m leaving it out because I don’t know the right approach, and sometimes doing accessibility wrong leads to a worse experience than doing nothing at all. Open to suggestions. :)

Closing

Okay, I think that’s as far as we’ll get today. It’s time to stop and enjoy the fruits of our labor.

Okay, OpenAI isn’t the best AI image generator, or maybe my prompt skills need some work. I’d love to see how yours turned out.

A lot of the things we covered today were a review: building Qwik components, making HTTP requests to OpenAI, state, and conditional rendering in the template.

The biggest difference, I think, is how we treat prompts for AI images. I find they require a little more creative thinking and finagling to get right. Hopefully, you found this helpful.

In the video version, we covered a really cool SVG component that I think is worth checking out, but this post was already long enough.

Functionally, the app is as far as I want to take it, so In the next post, we’ll focus on reliability and security before we get into launching to production.

Thank you so much for reading. If you liked this article, and want to support me, the best ways to do so are to share it, sign up for my newsletter, and follow me on Twitter.

Originally published on austingil.com.

AI for Web Devs: Prompt Engineering

Austin Gil — Mon, 22 Jan 2024 18:38:47 +0000

Welcome back to this series where we are building web applications that incorporate AI tooling. In the previous post, we covered what AI is, how it works, and some related terminology.

In this post, we’re going to cover prompt engineering, which is a way to modify your application’s behavior without changing the code. Since it’s challenging to explain without seeing the code, let’s get to it.

%[https://www.youtube.com/watch?v=pK6WzlTOlYw]

Start Adapting the UI

I hope you’ve come up with your own idea for an AI app, because this is where we’ll write mostly the same code, but could end up with different apps.

My app will take two different opponents and tell you who would win in a fight. I’ll start on the UI side of things because that’s easier for me.

So far, we’ve been giving users a single <textarea> and expecting them to write the entire prompt body to send to OpenAI. We can reduce the work users need to do and get more accurate prompts by modifying the UI to only ask for the missing details instead of the whole prompt.

In my app’s case, we really only need two things: opponent 1 and opponent 2. So instead of one input, we’ll have two.

This is a good opportunity to replace the <textarea> HTML with a reusable input component.

I’ll add a file called Input.jsx to the /src/components folder. The most basic example of a Qwik component is a function that uses the component$ function from "@builder.io/qwik" and returns JSX.

import { component$ } from "@builder.io/qwik";

export default component$((props) => {
  return (
    <div>
    </div>
  )
})

Our Input component should be reusable and accessible. For that, it needs a required label prop, a required name attribute, and an optional id which will default to a random string if not provided. And any other HTML attribute can be applied directly on the form control.

Here’s what I came up with along with JSDocs type definitions (note that the randomString function comes from this utility repo):

import { component$ } from "@builder.io/qwik";
import { randomString } from "~/utils.js";

/**
 * @typedef {import("@builder.io/qwik").HTMLAttributes<HTMLTextAreaElement>} TextareaAttributes
 */

/**
 * @type {import("@builder.io/qwik").Component<TextareaAttributes  & {
 * label: string,
 * name: string,
 * id?: string,
 * value?: string
 * }>}
 */
export default component$(({id, label, value, ...props}) => {
  const id = id || randomString(8)

  return (
    <div>
      <label for={id}>{label}</label>
      <textarea id={id} {...props}>{value}</textarea>
    </div>
  )
})

It’s rudimentary, but works for our app. If you’re feeling spunky, I encourage you to modify it to support the other input and select elements.

Now, instead of using a single <textarea> for the whole prompt, we can replace it with one of our new Input components for each opponent. I’ll put them in a two-column grid, so they sit next to each other on large screens.

<div class="grid gap-4 sm:grid-cols-2">
  <Input label="Opponent 1" name="opponent1" />
  <Input label="Opponent 2" name="opponent2" />
</div>

Side-Quest: `global.d.ts`

If you’re interested in using TypeScript or JSDocs, it may be useful to make the Qwik HTMLAttributes and Component global declarations so they’re easier to use across the application.

To do that, create a file at ./src/global.d.ts. Inside it, we’ll import HTMLAttributes and Component from "@builder.io/qwik" with aliases, then create global declarations with their original names that implement their functionality:

import type { Component as QC, HTMLAttributes as QH } from "@builder.io/qwik"

declare global {
  export type Component<T> = QC<T>
  export type HTMLAttributes<T> = QH<T>
}

This is just an optional step, but I like to do it because I use these two type definitions frequently. It’s nice to not have to import them all the time.

Adjust the Backend

Now that we’ve changed our UI to reduce the amount of information we ask for, we can move to the backend.

In the previous version, we were sending the entire prompt content using a form field named “prompt”. Now, we’re sending the two individual opponents, and we need to construct the prompt in the request handler.

export const onPost = async (requestEvent) => {
  // ...
  const formData = await requestEvent.parseBody()

  const { opponent1, opponent2 } = formData
  const prompt = `Who would win in a fight between ${opponent1} and ${opponent2}?`

  // ...
}

Functionally, this brings us back to where we were, except now there’s less work for the user to do and they have better guidance on what they need. That’s great! Unfortunately, the AI response is still something like, “As an AI language model I can’t predict hypothetical fights or determined definite winners blah blah blah…”

It’s not very helpful.

But because we’ve moved the control of the prompt to the backend, we’ve set the stage for prompt engineering because now we are in control of it instead of the user.

Here’s where our apps can take wildly different routes depending on how closely you want to follow the prompt that I write or if you make your own.

Begin Prompt Engineering

The AI already told us that there is no way it could know who could win in a fight, but what if we’re a little more persuasive? Let’s change our prompt to something like this:

const prompt = `Who would win in a fight between ${opponent1} and ${opponent2}?

Provide a creative and detailed explanation of why they would win and what tactics they might use.`

Now, instead of asking the AI for a wishy-washy answer, we’re encouraging it to provide a creative explanation. The result?

“In a hypothetical battle between a ninja and a pirate, the outcome would depend on several factors. Both ninjas and pirates possess unique skill sets and tactics that make them formidable opponents, so let’s imagine a thrilling encounter between the two and explore the possible outcome…”

That’s much better!

Of course, it’s going to be different each time, so I don’t expect you to get the same results, but the key thing is that the AI is cooperating.

Character Building

Our app is mostly working now, but I think we can also make it more interesting. One way to do that is to give the AI some context about the role it should play as it answers the questions. For example, why not make it answer questions as if it were a professional fighting judge from Liverpool who speaks mostly with Cockney slang?

To do that, we simply need to modify our prompt, but I also like to break up my prompt into various sections so it’s easier to manage.

const context = `You're a professional fighting judge from Liverpool that speaks mostly with Cockney slang`

const question = `Who would win in a fight between ${opponent1} and ${opponent2}?`

const format = `Provide a creative and detailed explanation of why they would win and what tactics they might use.`

const prompt = [context, question, format].join(' ')

This way, each separate section is captured by its own variable, which makes things easier for me to follow when I look at this later on.

What’s the result?

“Alright, mate! Let me put on me Cockney cap and dive into this lively debate between a ninja and a pirate. Picture meself in Liverpool, surrounded by kickin’ brick walls, ready to analyze this rumble in the most creative way…”

It spits out over three thousand words of ridiculousness, which is a lot of fun, but highlights another problem. The output is too long.

Understanding Tokens

Something worth understanding with these AI tools is “tokens”. From the OpenAI help article, “What are tokens and how to count them?“:

“Tokens can be thought of as pieces of words. Before the API processes the prompts, the input is broken down into tokens. These tokens are not cut up exactly where the words start or end – tokens can include trailing spaces and even sub-words.”

A token accounts for roughly four characters, they are calculated based on the text that the AI receives and produces, and there are two big reasons we need to be aware of them:

The platform charges based on the volume of tokens used.
Each LLM has a limit on the maximum tokens it can work with.

So it’s worth being cognizant of the length of text we send as a prompt as well as what we receive as a response. In some cases, you may want a lengthy response to achieve a better product, but otherwise, in other cases, it’s better to use fewer tokens.

In our case, a three thousand character response is not only a less-than-ideal user experience, it’s also costing us more money.

Reducing Tokens

Now that we’ve decided to reduce the tokens we use, the next question is, how?

If you’ve read through the OpenAI docs, you may have noticed a max_tokens parameter that we can set when we make the API request. Also, good on you for reading the docs. Five stars.

const body = {
  model: 'gpt-3.5-turbo',
  messages: [{ role: 'user', content: prompt }],
  stream: true,
  max_tokens: 100,
}

const response = await fetch('https://api.openai.com/v1/chat/completions', {
  method: 'post',
  headers: {
    'Content-Type': 'application/json',
    Authorization: `Bearer ${OPENAI_API_KEY}`,
  },
  body: JSON.stringify(body)
})

Let’s see what happens when we set the max_tokens parameter to something like 100.

Ok, now this is about the right length that I want, but it looks like it’s getting cut off. That’s because the GPT was given a hard limit on how much it could return, but it doesn’t account for that when constructing the response. As a result, we end up with an incomplete thought.

Not ideal.

Programmatically limiting the allowed length probably makes sense in some applications. It may even make sense in this one to add an upper bound. But to get a short AND complete response, the solution comes back to prompt engineering.

Let’s modify our prompt to ask for a “short explanation” instead of a “creative and detailed” one.

const format = `Only tell me who would win and a short reason why.`

Okay, this is more like what I had in mind. This is about the right length and level of detail. If you want to massage it some more, I encourage you to do so, but I’m going to move on.

Introducing LangChain

I want to address the clunkiness of the current system. You can imagine if we had a lot more prompts and a lot more endpoints it might be hard to manage. That’s why I want to introduce a tool chain called LangChain. In this new and constantly shifting world of AI, it’s been emerging as the leading tool chain for working with prompts. Let’s see why.

First, install the package with npm install @langchain/core.

The most relevant thing we can do with LangChain for our project is to generate prompts using prompt templates. Instead of generating our prompt from within our route handler, we can create a shareable prompt template and only provide the variables (opponent 1 & 2) at runtime. It’s essentially a factory function for prompts.

We can import the PromptTeplate module from "@langchain/core/prompts", then create a template and configure any variables it will consume like this:

const promptTemplate = new PromptTemplate({
  inputVariables: ['opponent1', 'opponent2'],
  template: `You're a professional fighting judge from Liverpool that speaks mostly with Cockney slang. Who would win in a fight between {opponent1} and {opponent2}? Only tell me who would win and a short reason why.`,
})

Notice that we’re using two inputVariables called “opponent1” and “opponent2”. These will be referenced in the template inside curly braces. It tells LangChain what variables to expect at runtime and where to place them.

So now, within our route handler, instead of constructing the entire prompt, we can call promptTemplate.format and provide our variables.

const prompt = await promptTemplate.format({
  opponent1: opponent1,
  opponent2: opponent2
})

Separating our prompt template from the route handler’s business logic simplifies the handler, makes the template easier to maintain, and allows us to export and share the template across the codebase if needed.

It’s worth mentioning that prompt templates are not the only benefit that LangChain offers. They also have tooling for managing the memory in chat applications, caching, handling timeouts, rate limiting, and more. This is just an introduction, but it’s worth getting more familiar with the capabilities if you plan on going deeper.

Identifying the Winner

One last thing that I want to do before we finish up today is to highlight the winner based on the response. Unfortunately, it’s hard to know that from a large block of indeterminate text.

Now, you may be thinking it would be nice to use a JSON object containing the winner and the text, and you’d be right.

Just one problem, in order to parse JSON, we need the entire JSON string, which means we would need to wait until the entire text completes. This kind of defeats the purpose of streaming.

This was one of the tricky challenges I found dealing with AI APIs.

The solution I came up with was to format the streaming response like so:

winner: opponent1 (or opponent2). reason: the reason they won...

This way, I could grab the winner programmatically and continue writing the reason to the page as it arrived by skipping the unrelated text. I’d love to hear your thoughts or see what you come up with, but let’s see how this worked.

First, we need to modify the prompt. In order for the AI to know how to respond with the winner, both opponents need a label (“opponent1” and “opponent2”). We’ll add those labels in parentheses when we first mention the opponents. And since we have a more specific requirement on what the returned format needs to be, we should also include that in the template.

Here’s what my template looks like now:

`You're a professional fighting judge from Liverpool that speaks mostly with Cockney slang. Who would win in a fight between {opponent1} ("opponent1") and {opponent2}("opponent2")? Only tell me who would win and a short reason why.

Format the response like this:
"winner: 'opponent1' or 'opponent2'. reason: the reason they won."`

Notice how now I’m giving the AI an example of what the response should look like. This is sometimes referred to as a one-shot prompt. What we had before without any example would be a zero-shot prompt. You can also have a multi-shot where you provide multiple examples.

OK, so now we should get back some text that tells us who the winner is and the reasoning.

The last step is to modify the way the frontend deals with this response so we separate the winner from the reasoning.

Showing just the reason to the user is the easy part. The first bit of the response will always be “winner: opponent1 (or 2). reason: “. So we can store the whole string in state, but skip the first 27 characters and show just the reason to the user. There are certainly some more advanced ways to get just the reasoning, but sometimes I prefer a simple solution.

We can replace this:

<p>{state.text}</p>

With this:

<p>{state.text.slice(27)}</p>

Identifying the winner is a little more robust. When the streaming response comes back, it still gets pushed to state.text. And after the response completes, we can pluck the winner from the results. You could slice the string, but I chose to use a Regular Expression:

// Previous fetch request logic

const winnerPattern = /winner:\s+(\w+).*/gi
const match = winnerPattern.exec(state.text)
const winner = match?.length ? match[1].toLowerCase() : ''

This Regular Expression looks for a string beginning with “winner:”, has an optional white-space character, then captures the next whole word up until a period character. Compared to our template, the captured word should either be “opponent1” or “opponent2”, our winners ;)

Once you have the winner, what you do with that information is up to you. I thought it would be cool to store it in state, and apply a fun rainbow background animation and confetti explosion (party-js) to the corresponding <textarea> .

That’s so fun. I love it!

I’ll let you sort that out if you want to recreate it, but here’s some of the code in case you’re interested.

JS:

if (state.winner) {
  const winnerInput = document.querySelector(`textarea[name=${state.winner}]`)
  if (winnerInput) {
    party.confetti(winnerInput, {
      count: 40,
      size: 2,
      spread: 15
    })
  }
}

CSS:

.rainbow {
  color: #fff;
  background: linear-gradient(45deg, #cc0000, #c8cc00, #38cc00, #00ccb5, #0015cc, #5f00cc, #c200cc, #cc0000);
  background-size: 1600% 1600%;
  animation: BgSlide 2s linear infinite;
}
@keyframes BgSlide {
  0% { background-position: 0% 50%; }
  100% { background-position: 100% 50%; }
}

Review

Alright, in the end, we did get into a few code changes, but I don’t want that to overshadow the main focus of this article. Now, we can drastically change the behavior of our app just by tweaking the prompt.

Some things we covered were:

Providing the AI with some context about its role.
Formatting responses.
The importance of understanding tokens.
Tooling like LangChain
Zero-shot, one-shot, and n-shot prompts.

I also don’t want to understate how much work can go into getting a prompt just right. This post was a silly example, but it actually took me a very long time to figure out the right permutations of words and formats to get what I needed. Don’t feel bad if it takes you a while to get used to it as well.

I truly believe that becoming a good prompt engineer will serve you well in the future. Even if you’re not building apps, it’s helpful for interacting with GPTs. But if you’re building apps, the key differentiating factors between the winners and losers will be the secret sauce that goes into the prompts and the form factor of using the app. It will need to be intuitive and provide the user with the least friction to get what they want.

In the next post we’ll start playing around with AI image generation, which comes with its own fun and quirky experience.

I hope you stick around, and feel free to reach out any time.

Thank you so much for reading. If you liked this article, and want to support me, the best ways to do so are to share it, sign up for my newsletter, and follow me on Twitter.

Originally published on austingil.com.

AI for Web Devs: What Are Neural Networks, LLMs, & GPTs?

Austin Gil — Thu, 18 Jan 2024 23:34:19 +0000

Welcome back to this series where we’re building web applications with AI tooling.

In the previous post, we got AI generated jokes into our Qwik application from OpenAI API. It worked, but the user experience suffered because we had to wait until the API completed the entire response before updating the client.

A better experience, as you’ll know if you’ve used any AI chat tools, is to respond as soon as each bit of text is generated. It becomes a sort of teletype effect.

That’s what we’re going to build today using HTTP streams.

Prerequisites

Before we get into streams, we need to explore something with a Qwik quirk related to HTTP requests.

If we examine the current POST request being sent by the form, we can see that the returned payload isn’t just the plain text we returned from our action handler. Instead, it’s this sort of serialized data.

This is the result of how the Qwik Optimizer lazy loads assets and is necessary to properly handle the data as it comes back. Unfortunately, this prevents standard streaming responses.

So while routeAction$ and the Form component are super handy, we’ll have to do something else.

To their credit, the Qwik team does provide a well-documented approach for streaming responses. However, it involves their server$ function and async generator functions. This would probably be the right approach if we’re talking strictly about Qwik, but this series is for everyone. I’ll avoid this implementation, as it’s too specific to Qwik, and focus on broadly applicable concepts instead.

Refactor Server Logic

It sucks that we can’t use route actions because they’re great. So what can we use?

Qwik City offers a few options. The best I found is middleware. They provide enough access to primitive tools that we can accomplish what we need, and the concepts will apply to other contexts besides Qwik.

Middleware is essentially a set of functions that we can inject at various points within the request lifecycle of our route handler. We can define them by exporting named constants for the hooks we want to target (onRequest, onGet, onPost, onPut, onDelete).

So instead of relying on a route action, we can use a middleware that hooks into any POST request by exporting an onPost middleware. In order to support streaming, we’ll want to return a standard Response object. We can do so by creating a Response object and passing it to the requestEvent.send() method.

Here’s a basic (non-streaming) example:

/** @type {import('@builder.io/qwik-city').RequestHandler} */
export const onPost = (requestEvent) => {
  requestEvent.send(new Response('Hello Squirrel!'))
}

Before we tackle streaming, let’s get the same functionality from the old route action implemented with middleware. We can copy most of the code into the onPost middleware, but we won’t have access to formData.

Fortunately, we can recreate that data from the requestEvent.parseBody() method. We’ll also want to use requestEvent.send() to respond with the OpenAI data instead of a return statement.

/** @type {import('@builder.io/qwik-city').RequestHandler} */
export const onPost = async (requestEvent) => {
  const OPENAI_API_KEY = requestEvent.env.get('OPENAI_API_KEY')
  const formData = await requestEvent.parseBody()

  const prompt = formData.prompt
  const body = {
    model: 'gpt-3.5-turbo',
    messages: [{ role: 'user', content: prompt }]
  }

  const response = await fetch('https://api.openai.com/v1/chat/completions', {
    // ... fetch options
  })
  const data = await response.json()

  const responseBody = data.choices[0].message.content

  requestEvent.send(new Response(responseBody))
}

Refactor Client Logic

Replacing the route actions has the unfortunate side effect of meaning we also can’t use the <Form> component anymore. We’ll have to use a regular HTML <form> element and recreate all the benefits we had before, including sending HTTP request with JavaScript, tracking the loading state, and accessing the results. Let’s refactor our client-side to support those features again.

We can break these requirements down to needing two things, a JavaScript solution for submitting forms and a reactive state for managing loading states and results.

I’ve covered submitting HTML forms with JavaScript in depth several times in the past:

So today I’ll just share the snippet, which I put inside a utils.js file in the root of my project. This jsFormSubmit function accepts an HTMLFormElement then constructs a fetch request based on the form attributes and returns the resulting Promise:

/**
 * @param {HTMLFormElement} form 
 */
export function jsFormSubmit(form) {
  const url = new URL(form.action)
  const formData = new FormData(form)
  const searchParameters = new URLSearchParams(formData)

  /** @type {Parameters<typeof fetch>[1]} */
  const fetchOptions = {
    method: form.method
  }

  if (form.method.toLowerCase() === 'post') {
    fetchOptions.body = form.enctype === 'multipart/form-data' ? formData : searchParameters
  } else {
    url.search = searchParameters
  }

  return fetch(url, fetchOptions)
}

This generic function can be used to submit any HTML form, so it’s handy to use in a submit event handler. Sweet!

As for the reactive data, Qwik provides two options, useStore and useSignal. I prefer useStore, which allows us to create an object whose properties are reactive. Meaning changes to the object’s properties will automatically be reflected wherever they are referenced in the UI.

We can use useStore to create a “state” object in our component to track the loading state of the HTTP request as well as the text response.

import { $, component$, useStore } from "@builder.io/qwik";
// other setup logic

export default component$(() => {
  const state = useStore({
    isLoading: false,
    text: '',
  })

  // other component logic
})

Next, we can update the template. Since we can no longer use the action object we had before, we can replace references from action.isRunning and action.value to state.isLoading and state.text, respectively (don’t ask me why I changed the property names 🤷‍♂️). I’ll also add a “submit” event handler to the form called handleSbumit, which we’ll look at shortly.

<main>
  <form 
    method="post"
    preventdefault:submit
    onSubmit$={handleSubmit}
  >
    <div>
      <label for="prompt">Prompt</label>
      <textarea name="prompt" id="prompt">
        Tell me a joke
      </textarea>
    </div>

    <button type="submit" aria-disabled={state.isLoading}>
      {state.isLoading ? 'One sec...' : 'Tell me'}
    </button>
  </form>

  {state.text && (
    <article>
      <p>{state.text}</p>
    </article>
  )}
</main>

Note that the <form> does not explicitly provide an action attribute. By default, an HTML form will submit data to the current URL, so we only need to set the method to POST and submit this form to trigger the onPost middleware we defined earlier.

Now, the last step to get this refactor working is defining handleSubmit. Just like we did in the previous post, we need to wrap an event handler inside Qwik’s $ function.

Inside the event handler, we’ll want to clear out any previous data from state.text, set state.isLoading to true, then pass the form’s DOM node to our fancy jsFormSubmit function. This should submit the HTTP request for us. Once it comes back, we can update state.text with the response body, and return state.isLoading to false.

const handleSubmit = $(async (event) => {
  state.text = ''
  state.isLoading = true

  /** @type {HTMLFormElement} */
  const form = event.target

  const response = await jsFormSubmit(form)

  state.text = await response.text()
  state.isLoading = false
})

OK! We should now have a client-side form that uses JavaScript to submit an HTTP request to the server while tracking the loading and response states, and updating the UI accordingly.

That was a lot of work to get the same solution we had before but with fewer features. BUT the key benefit is we now have direct access to the platform primitives we need to support streaming.

Enable Streaming on the Server

Before we start streaming responses from OpenAI, I think it’s helpful to start with a very basic example to get a better grasp of streams. Streams allow us to send small chunks of data over time. So as an example, let’s print out some iconic David Bowie lyrics in tempo with the song, “Space Oddity“.

When we construct our Response object, instead of passing plain text, we’ll want to pass a stream. We’ll create the stream shortly, but here’s the idea:

/** @type {import('@builder.io/qwik-city').RequestHandler} */
export const onPost = (requestEvent) => {
  requestEvent.send(new Response(stream))
}

We’ll create a very rudimentary ReadableStream using the ReadableStream constructor and pass it an optional parameter. This optional parameter can be an object with a start method that’s called when the stream is constructed.

The start method is responsible for the steam’s logic and has access to the stream controller, which is used to send data and close the stream.

const stream = new ReadableStream({
  start(controller) {
    // Stream logic goes here
  }
})

OK, let’s plan out that logic. We’ll have an array of song lyrics and a function to ‘sing’ them (pass them to the stream). The sing function will take the first item in the array and pass that to the stream using the controller.enqueue() method. If it’s the last lyric in the list, we can close the stream with controller.close(). Otherwise, the sing method can call itself again after a short pause.

const stream = new ReadableStream({
  start(controller) {
    const lyrics = ['Ground', ' control', ' to major', ' Tom.']
    function sing() {
      const lyric = lyrics.shift()

      controller.enqueue(lyric)

      if (lyrics.length < 1) {
        controller.close()
      } else {
        setTimeout(sing, 1000)
      }
    }
    sing()
  }
})

So each second, for four seconds, this stream will send out the lyrics “Ground control to major Tom.” Slick!

Because this stream will be used in the body of the Response, the connection will remain open for four seconds until the response completes. But the frontend will have access to each chunk of data as it arrives, rather than waiting the full four seconds.

This doesn’t speed up the total response time (in some cases, streams can increase response times), but it does allow for a faster perceived response, and that makes a better user experience.

Here’s what my code looks like:

/** @type {import('@builder.io/qwik-city').RequestHandler} */
export const onPost: RequestHandler = async (requestEvent) => {
  const stream = new ReadableStream({
    start(controller) {
      const lyrics = ['Ground', ' control', ' to major', ' Tom.']
      function sing() {
        const lyric = lyrics.shift()

        controller.enqueue(lyric)

        if (lyrics.length < 1) {
          controller.close()
        } else {
          setTimeout(sing, 1000)
        }
      }
      sing()
    }
  })

  requestEvent.send(new Response(stream))
}

Unfortunately, as it stands right now, the client will still be waiting four seconds before seeing the entire response, and that’s because we weren’t expecting a streamed response.

Let’s fix that.

Enable Streaming on the Client

Even when dealing with streams, the default browser behavior when receiving a response is to wait for it to complete. In order to get the behavior we want, we’ll need to use client-side JavaScript to make the request and process the streaming body of the response.

We’ve already tackled that first part inside our handleSubmit function. Let’s start processing that response body.

We can access the ReadableStream from the response body’s getReader() method. This stream will have its own read() method that we can use to access the next chunk of data, as well as the information if the response is done streaming or not.

The only ‘gotcha’ is that the data in each chunk doesn’t come in as text, it comes in as a Uint8Array, which is “an array of 8-bit unsigned integers.” It’s basically the representation of the binary data, and you don’t really need to understand any deeper than that unless you want to sound very smart at a party (or boring).

The important thing to understand is that on their own, these data chunks aren’t very useful. To get something we can use, we’ll need to decode each chunk of data using a TextDecoder.

Ok, that’s a lot of theory. Let’s break down the logic and then look at some code.

When we get the response back, we need to:

Grab the reader from the response body using response.body.getReader()
Setup a decoder using TextDecoder and a variable to track the streaming status.
Process each chunk until the stream is complete, with a while loop that does this:
1. Grab the next chunk’s data and stream status.
2. Decode the data and use it to update our app’s state.text.
3. Update the streaming status variable, terminating the loop when complete.
Update the loading state of the app by setting state.isLoading to false.

The new handleSubmit function should look something like this:

const handleSubmit = $(async (event) => {
  state.text = ''
  state.isLoading = true

  /** @type {HTMLFormElement} */
  const form = event.target

  const response = await jsFormSubmit(form)

  // Parse streaming body
  const reader = response.body.getReader()
  const decoder = new TextDecoder()
  let isStillStreaming = true

  while(isStillStreaming) {
    const {value, done} = await reader.read()
    const chunkValue = decoder.decode(value)

    state.text += chunkValue

    isStillStreaming = !done
  }

  state.isLoading = false
})

Now, when I submit the form, I see something like:

“Ground

control

to major

Tom.”

Hell yeah!!!

OK, most of the work is down. Now we just need to replace our demo stream with the OpenAI response.

Stream OpenAI Response

Looking back at our original implementation, the first thing we need to do is modify the request to OpenAI to let them know that we would like a streaming response. We can do that by setting the stream property in the fetch payload to true.

const body = {
  model: 'gpt-3.5-turbo',
  messages: [{ role: 'user', content: prompt }],
  stream: true
}

const response = await fetch('https://api.openai.com/v1/chat/completions', {
  method: 'post',
  headers: {
    'Content-Type': 'application/json',
    Authorization: `Bearer ${OPENAI_API_KEY}`,
  },
  body: JSON.stringify(body)
})

UPDATE 2023/11/15: I used fetch and custom streams because at the time of writing, the openai module on NPM did not properly support streaming responses. That issue has been fixed, and I think a better solution would be to use that module and pipe their data through a TransformStream to send to the client. That version is not reflected here.

Next, we could pipe the response from OpenAI directly to the client, but we might not want to do that. The data they send doesn’t really align with that we want to send to the client because it looks like this (two chunks, one with data, and one representing the end of the stream):

data: {"id":"chatcmpl-4bJZRnslkje3289REHFEH9ej2","object":"chat.completion.chunk","created":1690319476,"model":"gpt-3.5-turbo-0613","choiced":[{"index":0,"delta":{"content":"Because"},"finish_reason":"stop"}]}

data: [DONE]

Instead, what we’ll do is create our own stream, similar to the David Bowie lyrics, that will do some setup, enqueue chunks of data into the stream, and close the stream. Let’s start with an outline:

const stream = new ReadableStream({
  async start(controller) {
    // Any setup before streaming   
    // Send chunks of data
    // Close stream
  }
})

Since we’re dealing with a streaming fetch response from OpenAI, a lot of the work we need to do here can actually be copied from the client-side stream handling. This part should look familiar:

const reader = response.body.getReader()
const decoder = new TextDecoder()
let isStillStreaming = true

while(isStillStreaming) {
  const {value, done} = await reader.read()
  const chunkValue = decoder.decode(value)

  // Here's where things will be different

  isStillStreaming = !done
}

This snippet was taken almost directly from the frontend stream processing example. The only difference is that we need to treat the data coming from OpenAI slightly differently. As we say, the chunks of data they send up will look something like”data: [JSON data or done]“. Another gotcha is that every once in a while, they’ll actually slip in TWO of these data strings in a single streaming chunk. So here’s what I came up with for processing the data.

Create a Regular Expression to grab the rest of the string after “data: “.
For the unlikely event there are more than one data strings, use a while loop to process every match in the string.
If the current matches the closing condition (“[DONE]“) close the stream.
Otherwise, parse the data as JSON and enqueue the first piece of text from the list of options (json.choices[0].delta.content). Fall back to an empty string if none is present.
Lastly, in order to move to the next match, if there is one, we can use RegExp.exec().

The logic is quite abstract without looking at code, so here’s what the whole stream looks like now:

const stream = new ReadableStream({
  async start(controller) {
    // Do work before streaming
    const reader = response.body.getReader()
    const decoder = new TextDecoder()
    let isStillStreaming = true

    while(isStillStreaming) {
      const {value, done} = await reader.read()
      const chunkValue = decoder.decode(value)

      /**
       * Captures any string after the text `data: `
       * @see https://regex101.com/r/R4QgmZ/1
       */
      const regex = /data:\s*(.*)/g
      let match = regex.exec(chunkValue)

      while (match !== null) {
        const payload = match[1]

        // Close stream
        if (payload === '[DONE]') {
          controller.close()
          break
        } else {
          try {
            const json = JSON.parse(payload)
            const text = json.choices[0].delta.content || ''

            // Send chunk of data
            controller.enqueue(text)
            match = regex.exec(chunkValue)
          } catch (error) {
            const nextChunk = await reader.read()
            const nextChunkValue = decoder.decode(nextChunk.value)
            match = regex.exec(chunkValue + nextChunkValue)
          }
        }
      }

      isStillStreaming = !done
    }
  }
})

UPDATE 2023/11/15: I discovered that OpenAI API sometimes returns the JSON payload across two streams. So the solution is to use a try/catch block around the JSON.parse and in the case that it fails, reassign the match variable to the current chunk value plus the next chunk value. The code above has the updated snippet.

Review

That should be everything we need to get streaming working. Hopefully it all makes sense and you got it working on your end.

I think it’s a good idea to review the flow to make sure we’ve got it:

The user submits the form, which gets intercepted and sent with JavaScript. This is necessary to process the stream when it returns.
The request is received by the action handler which forwards the data to the OpenAI API along with the setting to return the response as a stream.
The OpenAI response will be sent back as a stream of chunks, some of which contain JSON and the last one being “[DONE]“.
Instead of passing the stream to the action response, we create a new stream to use in the response.
Inside this stream, we process each chunk of data from the OpenAI response and convert it to something more useful before enqueuing it for the action response stream.
When the OpenAI stream closes, we also close our action stream.
The JavaScript handler on the client side will also process each chunk of data as it comes in and update the UI accordingly.

Conclusion

The app is working. It’s pretty cool. We covered a lot of interesting things today. Streams are very powerful, but also challenging and, especially when working within Qwik, there are a couple of little gotchas. But because we focused on low-level fundamentals, these concepts should apply across any framework.

As long as you have access to the platform and primitives like streams, requests, and response objects then this should work. That’s the beauty of fundamentals.

I think we got a pretty decent application going now. The only problem is right now we’re using a generic text input and asking users to fill in the entire prompt themselves. In fact, they can put in whatever they want. We’ll want to fix that in a future post, but the next post is going to step away from code and focus on understanding how the AI tools actually work.

I hope you’ve been enjoying this series and come back for the rest of it.

Thank you so much for reading. If you liked this article, and want to support me, the best ways to do so are to share it, sign up for my newsletter, and follow me on Twitter.

Originally published on austingil.com.

AI for Web Devs: Faster Responses with HTTP Streaming

Austin Gil — Tue, 16 Jan 2024 19:12:20 +0000

Welcome back to this series where we’re building web applications with AI tooling.

A better experience, as you’ll know if you’ve used any AI chat tools, is to respond as soon as each bit of text is generated. It becomes a sort of teletype effect.

That’s what we’re going to build today using HTTP streams.

Prerequisites

Before we get into streams, we need to explore something with a Qwik quirk related to HTTP requests.

This is the result of how the Qwik Optimizer lazy loads assets and is necessary to properly handle the data as it comes back. Unfortunately, this prevents standard streaming responses.

So while routeAction$ and the Form component are super handy, we’ll have to do something else.

Refactor Server Logic

It sucks that we can’t use route actions because they’re great. So what can we use?

Here’s a basic (non-streaming) example:

/** @type {import('@builder.io/qwik-city').RequestHandler} */
export const onPost = (requestEvent) => {
  requestEvent.send(new Response('Hello Squirrel!'))
}

Fortunately, we can recreate that data from the requestEvent.parseBody() method. We’ll also want to use requestEvent.send() to respond with the OpenAI data instead of a return statement.

/** @type {import('@builder.io/qwik-city').RequestHandler} */
export const onPost = async (requestEvent) => {
  const OPENAI_API_KEY = requestEvent.env.get('OPENAI_API_KEY')
  const formData = await requestEvent.parseBody()

  const prompt = formData.prompt
  const body = {
    model: 'gpt-3.5-turbo',
    messages: [{ role: 'user', content: prompt }]
  }

  const response = await fetch('https://api.openai.com/v1/chat/completions', {
    // ... fetch options
  })
  const data = await response.json()

  const responseBody = data.choices[0].message.content

  requestEvent.send(new Response(responseBody))
}

Refactor Client Logic

We can break these requirements down to needing two things, a JavaScript solution for submitting forms and a reactive state for managing loading states and results.

I’ve covered submitting HTML forms with JavaScript in depth several times in the past:

/**
 * @param {HTMLFormElement} form 
 */
export function jsFormSubmit(form) {
  const url = new URL(form.action)
  const formData = new FormData(form)
  const searchParameters = new URLSearchParams(formData)

  /** @type {Parameters<typeof fetch>[1]} */
  const fetchOptions = {
    method: form.method
  }

  if (form.method.toLowerCase() === 'post') {
    fetchOptions.body = form.enctype === 'multipart/form-data' ? formData : searchParameters
  } else {
    url.search = searchParameters
  }

  return fetch(url, fetchOptions)
}

This generic function can be used to submit any HTML form, so it’s handy to use in a submit event handler. Sweet!

We can use useStore to create a “state” object in our component to track the loading state of the HTTP request as well as the text response.

import { $, component$, useStore } from "@builder.io/qwik";
// other setup logic

export default component$(() => {
  const state = useStore({
    isLoading: false,
    text: '',
  })

  // other component logic
})

<main>
  <form 
    method="post"
    preventdefault:submit
    onSubmit$={handleSubmit}
  >
    <div>
      <label for="prompt">Prompt</label>
      <textarea name="prompt" id="prompt">
        Tell me a joke
      </textarea>
    </div>

    <button type="submit" aria-disabled={state.isLoading}>
      {state.isLoading ? 'One sec...' : 'Tell me'}
    </button>
  </form>

  {state.text && (
    <article>
      <p>{state.text}</p>
    </article>
  )}
</main>

Now, the last step to get this refactor working is defining handleSubmit. Just like we did in the previous post, we need to wrap an event handler inside Qwik’s $ function.

const handleSubmit = $(async (event) => {
  state.text = ''
  state.isLoading = true

  /** @type {HTMLFormElement} */
  const form = event.target

  const response = await jsFormSubmit(form)

  state.text = await response.text()
  state.isLoading = false
})

OK! We should now have a client-side form that uses JavaScript to submit an HTTP request to the server while tracking the loading and response states, and updating the UI accordingly.

That was a lot of work to get the same solution we had before but with fewer features. BUT the key benefit is we now have direct access to the platform primitives we need to support streaming.

Enable Streaming on the Server

When we construct our Response object, instead of passing plain text, we’ll want to pass a stream. We’ll create the stream shortly, but here’s the idea:

/** @type {import('@builder.io/qwik-city').RequestHandler} */
export const onPost = (requestEvent) => {
  requestEvent.send(new Response(stream))
}

The start method is responsible for the steam’s logic and has access to the stream controller, which is used to send data and close the stream.

const stream = new ReadableStream({
  start(controller) {
    // Stream logic goes here
  }
})

const stream = new ReadableStream({
  start(controller) {
    const lyrics = ['Ground', ' control', ' to major', ' Tom.']
    function sing() {
      const lyric = lyrics.shift()

      controller.enqueue(lyric)

      if (lyrics.length < 1) {
        controller.close()
      } else {
        setTimeout(sing, 1000)
      }
    }
    sing()
  }
})

So each second, for four seconds, this stream will send out the lyrics “Ground control to major Tom.” Slick!

This doesn’t speed up the total response time (in some cases, streams can increase response times), but it does allow for a faster perceived response, and that makes a better user experience.

Here’s what my code looks like:

/** @type {import('@builder.io/qwik-city').RequestHandler} */
export const onPost: RequestHandler = async (requestEvent) => {
  const stream = new ReadableStream({
    start(controller) {
      const lyrics = ['Ground', ' control', ' to major', ' Tom.']
      function sing() {
        const lyric = lyrics.shift()

        controller.enqueue(lyric)

        if (lyrics.length < 1) {
          controller.close()
        } else {
          setTimeout(sing, 1000)
        }
      }
      sing()
    }
  })

  requestEvent.send(new Response(stream))
}

Unfortunately, as it stands right now, the client will still be waiting four seconds before seeing the entire response, and that’s because we weren’t expecting a streamed response.

Let’s fix that.

Enable Streaming on the Client

We’ve already tackled that first part inside our handleSubmit function. Let’s start processing that response body.

The important thing to understand is that on their own, these data chunks aren’t very useful. To get something we can use, we’ll need to decode each chunk of data using a TextDecoder.

Ok, that’s a lot of theory. Let’s break down the logic and then look at some code.

When we get the response back, we need to:

Grab the reader from the response body using response.body.getReader()
Setup a decoder using TextDecoder and a variable to track the streaming status.
Process each chunk until the stream is complete, with a while loop that does this:
1. Grab the next chunk’s data and stream status.
2. Decode the data and use it to update our app’s state.text.
3. Update the streaming status variable, terminating the loop when complete.
Update the loading state of the app by setting state.isLoading to false.

The new handleSubmit function should look something like this:

const handleSubmit = $(async (event) => {
  state.text = ''
  state.isLoading = true

  /** @type {HTMLFormElement} */
  const form = event.target

  const response = await jsFormSubmit(form)

  // Parse streaming body
  const reader = response.body.getReader()
  const decoder = new TextDecoder()
  let isStillStreaming = true

  while(isStillStreaming) {
    const {value, done} = await reader.read()
    const chunkValue = decoder.decode(value)

    state.text += chunkValue

    isStillStreaming = !done
  }

  state.isLoading = false
})

Now, when I submit the form, I see something like:

“Ground

control

to major

Tom.”

Hell yeah!!!

OK, most of the work is down. Now we just need to replace our demo stream with the OpenAI response.

Stream OpenAI Response

const body = {
  model: 'gpt-3.5-turbo',
  messages: [{ role: 'user', content: prompt }],
  stream: true
}

const response = await fetch('https://api.openai.com/v1/chat/completions', {
  method: 'post',
  headers: {
    'Content-Type': 'application/json',
    Authorization: `Bearer ${OPENAI_API_KEY}`,
  },
  body: JSON.stringify(body)
})

data: {"id":"chatcmpl-4bJZRnslkje3289REHFEH9ej2","object":"chat.completion.chunk","created":1690319476,"model":"gpt-3.5-turbo-0613","choiced":[{"index":0,"delta":{"content":"Because"},"finish_reason":"stop"}]}

data: [DONE]

const stream = new ReadableStream({
  async start(controller) {
    // Any setup before streaming   
    // Send chunks of data
    // Close stream
  }
})

Since we’re dealing with a streaming fetch response from OpenAI, a lot of the work we need to do here can actually be copied from the client-side stream handling. This part should look familiar:

const reader = response.body.getReader()
const decoder = new TextDecoder()
let isStillStreaming = true

while(isStillStreaming) {
  const {value, done} = await reader.read()
  const chunkValue = decoder.decode(value)

  // Here's where things will be different

  isStillStreaming = !done
}

Create a Regular Expression to grab the rest of the string after “data: “.
For the unlikely event there are more than one data strings, use a while loop to process every match in the string.
If the current matches the closing condition (“[DONE]“) close the stream.
Otherwise, parse the data as JSON and enqueue the first piece of text from the list of options (json.choices[0].delta.content). Fall back to an empty string if none is present.
Lastly, in order to move to the next match, if there is one, we can use RegExp.exec().

The logic is quite abstract without looking at code, so here’s what the whole stream looks like now:

const stream = new ReadableStream({
  async start(controller) {
    // Do work before streaming
    const reader = response.body.getReader()
    const decoder = new TextDecoder()
    let isStillStreaming = true

    while(isStillStreaming) {
      const {value, done} = await reader.read()
      const chunkValue = decoder.decode(value)

      /**
       * Captures any string after the text `data: `
       * @see https://regex101.com/r/R4QgmZ/1
       */
      const regex = /data:\s*(.*)/g
      let match = regex.exec(chunkValue)

      while (match !== null) {
        const payload = match[1]

        // Close stream
        if (payload === '[DONE]') {
          controller.close()
          break
        } else {
          try {
            const json = JSON.parse(payload)
            const text = json.choices[0].delta.content || ''

            // Send chunk of data
            controller.enqueue(text)
            match = regex.exec(chunkValue)
          } catch (error) {
            const nextChunk = await reader.read()
            const nextChunkValue = decoder.decode(nextChunk.value)
            match = regex.exec(chunkValue + nextChunkValue)
          }
        }
      }

      isStillStreaming = !done
    }
  }
})

Review

That should be everything we need to get streaming working. Hopefully it all makes sense and you got it working on your end.

I think it’s a good idea to review the flow to make sure we’ve got it:

The user submits the form, which gets intercepted and sent with JavaScript. This is necessary to process the stream when it returns.
The request is received by the action handler which forwards the data to the OpenAI API along with the setting to return the response as a stream.
The OpenAI response will be sent back as a stream of chunks, some of which contain JSON and the last one being “[DONE]“.
Instead of passing the stream to the action response, we create a new stream to use in the response.
Inside this stream, we process each chunk of data from the OpenAI response and convert it to something more useful before enqueuing it for the action response stream.
When the OpenAI stream closes, we also close our action stream.
The JavaScript handler on the client side will also process each chunk of data as it comes in and update the UI accordingly.

Conclusion

As long as you have access to the platform and primitives like streams, requests, and response objects then this should work. That’s the beauty of fundamentals.

I hope you’ve been enjoying this series and come back for the rest of it.

Thank you so much for reading. If you liked this article, and want to support me, the best ways to do so are to share it, sign up for my newsletter, and follow me on Twitter.

Originally published on austingil.com.

AI for Web Devs: Your First API Request to OpenAI

Austin Gil — Tue, 16 Jan 2024 17:40:24 +0000

Welcome back to this series where we are learning how to integrate AI products into web applications.

Last time, we got all the boilerplate work out of the way.

In this post, we’ll learn how to integrate OpenAI’s API responses into our Qwik app using fetch. We’ll want to make sure we’re not leaking API keys by executing these HTTP requests from a backend.

By the end of this post, we will have a rudimentary, but working AI application.

Generate OpenAI API Key

Before we start building anything, you’ll need to go to platform.openai.com/account/api-keys and generate an API key to use in your application.

Make sure to keep a copy of it somewhere because you will only be able to see it once.

With your API key, you’ll be able to make authenticated HTTP requests to OpenAI. So it’s a good idea to get familiar with the API itself. I’d encourage you to take a brief look through the OpenAI Documentation and become familiar with some concepts. The models are particularly good to understand because they have varying capabilities.

If you would like to familiarize yourself with the API endpoints, expected payloads, and return values, check out the OpenAI API Reference. It also contains helpful examples.

You may notice the JavaScript package available on NPM called openai. We will not be using this, as it doesn’t quite support some things we’ll want to do, that fetch can.

Make Your First HTTP Request

The application we’re going to build will make an AI-generated text completion based on the user input. For that, we’ll want to work with the chat endpoint (note that the completions endpoint is deprecated).

We need to make a POST request to https://api.openai.com/v1/chat/completions with the 'Content-Type' header set to 'application/json', the 'Authorization' set to 'Bearer OPENAI_API_KEY' (you’ll need to replace OPENAI_API_KEY with your API key), and the body set to a JSON string containing the GPT model to use (we’ll use gpt-3.5-turbo) and an array of messages:

fetch('https://api.openai.com/v1/chat/completions', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Authorization': 'Bearer OPENAI_API_KEY'
  },
  body: JSON.stringify({
    'model': 'gpt-3.5-turbo',
    'messages': [
      {
        'role': 'user',
        'content': 'Tell me a funny joke'
      }
    ]
  })
})

You can run this right from your browser console and see the request in the Network tab of your dev tools.

The response should be a JSON object with a bunch of properties, but the one we’re most interested in is the "choices". It will be an array of text completions objects. The first one should be an object with a "message" object that has a "content" property with the chat completion.

{
  "id": "chatcmpl-7q63Hd9pCPxY3H4pW67f1BPSmJs2u",
  "object": "chat.completion",
  "created": 1692650675,
  "model": "gpt-3.5-turbo-0613",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Why don't scientists trust atoms?\n\nBecause they make up everything!"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 12,
    "completion_tokens": 13,
    "total_tokens": 25
  }
}

Congrats! Now you can request a mediocre joke whenever you want.

Build the Form

The fetch request above is fine, but it’s not quite an application. What we want is something a user can interact with to generate an HTTP request like the one above.

For that, we’ll probably want some sort to start with an HTML <form> containing a <textarea>. Below is the minimum markup we need, and if you want to learn more, consider reading these articles:

<form>
  <label for="prompt">Prompt</label>
  <textarea id="prompt" name="prompt"></textarea>

  <button>Tell me</button>
</form>

We can copy and paste this form right inside our Qwik component’s JSX template. If you’ve worked with JSX in the past, you may be used to replacing the for attribute on the <label> with htmlFor, but Qwik’s compiler actually doesn’t require us to do that, so it’s fine as is.

Next, we’ll want to replace the default form submission behavior. By default, when an HTML form is submitted, the browser will create an HTTP request by loading the URL provided in the form’s action attribute. If none is provided, it will use the current URL. We want to avoid this page load and use JavaScript instead.

If you’ve done this before, you may be familiar with the preventDefault method on the Event interface. As the name suggests, it prevents the default behavior for the event.

There’s a challenge here due to how Qwik deals with event handlers. Unlike other frameworks, Qwik does not download all the JavaScript logic for the application upon first page load. Instead, it has a very thin client that intercepts user interactions and downloads the JavaScript event handlers on-demand.

This asynchronous nature makes Qwik applications much faster to load, but introduces a challenge of dealing with event handlers asynchronously. It makes it impossible to prevent the default behavior the same way as synchronous event handlers that are downloaded and parsed before the user interactions.

Fortunately, Qwik provides a way to prevent the default behavior by adding preventdefault:{eventName} to the HTML tag. A very basic form example may look something like this:

import { component$ } from '@builder.io/qwik';

export default component$(() => {
  return (
    <form
      preventdefault:submit
      onSubmit$={(event) => {
        console.log(event)
      }}
    >
      <!-- form contents -->
    </form>
  )
})

Did you notice that little $ at the end of the onSubmit$ handler, there? Keep an eye out for those, because they are usually a hint to the developer that Qwik’s compiler is going to do something funny and transform the code. In this case, it’s due to that lazy-loading event handling system I mentioned above. If you plan on working with Qwik more, it’s worth reading more about that here.

Incorporate the Fetch Request

Now we have the tools in place to replace the default form submission with the fetch request we created above.

What we want to do next is pull the data from the <textarea> into the body of the fetch request. We can do so with FormData, which expects a form element as an argument and provides an API to access a form control values through the control’s name attribute.

We can access the form element from the event’s target property, use it to create a new FormData object, and use that to get the <textarea> value by referencing its name, “prompt”. Plug that into the body of the fetch request we wrote above, and you might get something that looks like this:

export default component$(() => {
  return (
    <form
      preventdefault:submit
      onSubmit$={(event) => {
        const form = event.target
        const formData = new FormData(form)
        const prompt = formData.get('prompt')
        const body = {
          'model': 'gpt-3.5-turbo',
          'messages': [{ 'role': 'user', 'content': prompt }]
        }

        fetch('https://api.openai.com/v1/chat/completions', {
          method: 'POST',
          headers: {
            'Content-Type': 'application/json',
            'Authorization': 'Bearer OPENAI_API_KEY'
          },
          body: JSON.stringify(body)
        })
      }}
    >
      <!-- form contents -->
    </form>
  )
})

In theory, you should now have a form on your page that, when submitted, sends the value from the textarea to the OpenAI API.

Protect Your API Keys

Although our HTTP request is working, there’s a glaring issue. Because it’s being constructed on the client side, anyone can open the browser dev tools and inspect the properties of the request. This includes the Authorization header containing our API keys.

I’ve blocked out my API token here with a red bar.

This would allow someone to steal our API tokens and make requests on our behalf, which could lead to abuse or higher charges on our account.

Not good!!!

The best way to prevent this is to move this API call to a backend server that we control that would work as a proxy. The frontend can make an unauthenticated request to the backend, and the backend would make the authenticated request to OpenAI and return the response to the frontend. But because users can’t inspect backend processes, they would not be able to see the Authentication header.

So how do we move the fetch request to the backend?

I’m so glad you asked!

We’ve been mostly focusing on building the frontend with Qwik, the framework, but we also have access to use Qwik City, the full-stack meta-framework with tooling for file-based routing, route middleware, HTTP endpoints, and more.

Of the various options Qwik City offers for running backend logic, my favorite is routeAction$. It allows us to create a backend function that can be triggered from the client over HTTP (essentially an RPC endpoint).

The logic would follow:

Use routeAction$() to create an action.
Provide the backend logic as the parameter.
Programmatically execute the action’s submit() method.

A simplified example could be:

import { component$ } from '@builder.io/qwik';
import { routeAction$ } from '@builder.io/qwik-city';

export const useAction = routeAction$((params) => {
  console.log('action on the server', params)
  return { o: 'k' }
})

export default component$(() => {
  const action = useAction()
  return (
    <form
      preventdefault:submit
      onSubmit$={(event) => {
        action.submit('data')
      }}
    >
      <!-- form contents -->
    </form>
    { JSON.stringify(action) }
  )
})

I included a JSON.stringify(action) at the end of the template because I think you should see what the returned ActionStore looks like. It contains extra information like whether the action is running, what the submission values were, what the response status is, what the returned value is, and more.

This is all very useful data that we get out of the box just by using an action, and it allows us to create more robust applications with less work.

Enhance the Experience

Qwik City actions are cool, but they get even better when combined with Qwik’s <Form> component:

Under the hood, the component uses a native HTML element, so it will work without JavaScript.

When JS is enabled, the component will intercept the form submission and trigger the action in SPA mode, allowing to have a full SPA experience.

By replacing the HTML <form> element with Qwik’s <Form> component, we no longer have to set up preventdefault:submit, onSubmit$, or call action.submit(). We can just pass the action to the Form‘s action prop, and it’ll take care of the work for us. Additionally, it will work if JavaScript is not available for some reason (we could have done this with the HTML version as well, but it would have been more work).

import { component$ } from '@builder.io/qwik';
import { routeAction$, Form } from '@builder.io/qwik-city';

export const useAction = routeAction$(() => {
  console.log('action on the server')
  return { o: 'k' }
});

export default component$(() => {
  const action = useAction()
  return (
    <Form action={action}>
      <!-- form contents -->
    </Form>
  )
})

So that’s an improvement for the developer experience. Let’s also improve the user experience.

Within the ActionStore, we have access to the isRunning data which keeps track of whether the request is pending or not. It’s handy information we can use to let the user know when the request is in flight.

We can do so by modifying the text of the submit button to say “Tell me” when it’s idle, then “One sec…” while it’s loading. I also like to assign the aria-disabled attribute to match the isRunning state. This will hint to assistive technology that it’s not ready to be clicked (though technically still can be). It can also be targeted with CSS to provide visual styles suggesting it’s not quite ready to be clicked again.

<button type="submit" aria-disabled={state.isLoading}>
  {state.isLoading ? 'One sec...' : 'Tell me'}
</button>

Show the Results

Ok, we’ve done way too much work without actually seeing the results on the page. It’s time to change that. Let’s bring the fetch request we prototyped earlier in the browser into our application.

We can copy/paste the fetch code right into the body of our action handler, but to access the user’s input data, we’ll need access to the form data that is submitted. Fortunately, any data passed to the action.submit() method will be available to the action handler as the first parameter. It will be a serialized object where the keys correspond to the form control names.

Note that I’ll be using the await keyword in the body of the handler, which means I also have to tag the handler as an async function.

import { component$ } from '@builder.io/qwik';
import { routeAction$, Form } from '@builder.io/qwik-city';

export const useAction = routeAction$(async (formData) => {
  const prompt = formData.prompt // From <textarea name="prompt">
  const body = {
    'model': 'gpt-3.5-turbo',
    'messages': [{ 'role': 'user', 'content': prompt }]
  }

  const response = await fetch('https://api.openai.com/v1/chat/completions', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Authorization': 'Bearer OPENAI_API_KEY'
    },
    body: JSON.stringify(body)
  })
  const data = await response.json()

  return data.choices[0].message.content
})

At the end of the action handler, we also want to return some data for the frontend. The OpenAI response comes back as JSON, but I think we might as well just return the text. If you remember from the response object we saw above, that data is located at responseBody.choices[0].message.content.

If we set things up correctly, we should be able to access the action handler’s response in the ActionStore‘s value property. This means we can conditionally render it somewhere in the template like so:

{action.value && (
  <p>{action.value}</p>
)}

Use Environment Variables

Alright, we’ve moved the OpenAI request to the backend, protected our API keys from prying eyes, we’re getting a (mediocre joke) response, and displaying it on the frontend. The app is working, but there’s still one more security issue to deal with.

It’s generally a bad idea to hard code API keys into your source code, for a number of reasons:

It means you can’t share the repo publicly without exposing your keys.
You may run up API usage during development, testing, and staging.
Changing API keys requires code changes and re-deploys.
You’ll need to regenerate API keys anytime someone leaves the org.

A better system is to use environment variables. With environment variables, you can provide the API keys only to the systems and users that need access to them.

For example, you can make an environment variable called OPENAI_API_KEY with the value of your OpenAI key for only the production environment. This way, only developers with direct access to that environment would be able to access it. This greatly reduces the likelihood of the API keys leaking, it makes it easier to share your code openly, and because you are limiting access to the keys to the least number of people, you don’t need to replace keys as often because someone left the company.

In Node.js, it’s common to set environment variables from the command line (ENV_VAR=example npm start) or with the popular dotenv package. Then, in your server-side code, you can access environment variables using process.env.ENV_VAR.

Things work slightly differently with Qwik.

Qwik can target different JavaScript runtimes (not just Node), and accessing environment variables via process.env is a Node-specific concept. To make things more runtime-agnostic, Qwik provides access to environment variables through a RequestEvent object which is available as the second parameter to the route action handler function.

import { routeAction$ } from '@builder.io/qwik-city';

export const useAction = routeAction$((param, requestEvent) => {
  const envVariableValue = requestEvent.env.get('ENV_VARIABLE_NAME')
  console.log(envVariableValue)
  return {}
})

So that’s how we access environment variables, but how do we set them?

Unfortunately, for production environments, setting environment variables will differ depending on the platform. For a standard server VPS, you can still set them with the terminal as you would in Node (ENV_VAR=example npm start).

In development, we can alternatively create a local.env file containing our environment variables, and they will be automatically assigned for us. This is convenient since we spend a lot more time starting the development environment, and it means we can provide the appropriate API keys only to the people who need them.

So after you create a local.env file, you can assign the OPENAI_API_KEY variable to your API key.

OPENAI_API_KEY="your-api-key"

(You may need to restart your dev server)

Then we can access the environment variable through the RequestEvent parameter. With that, we can replace the hard-coded value in our fetch request’s Authorization header with the variable using Template Literals.

export const usePromptAction = routeAction$(async (formData, requestEvent) => {
  const OPENAI_API_KEY = requestEvent.env.get('OPENAI_API_KEY')

  const prompt = formData.prompt
  const body = {
    model: 'gpt-3.5-turbo',
    messages: [{ role: 'user', content: prompt }]
  }

  const response = await fetch('https://api.openai.com/v1/chat/completions', {
    method: 'post',
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${OPENAI_API_KEY}`,
    },
    body: JSON.stringify(body)
  })
  const data = await response.json()

  return data.choices[0].message.content
})

For more details on environment variables in Qwik, see their documentation.

Recap

When a user submits the form, the default behavior is intercepted by Qwik’s optimizer which lazy loads the event handler.
The event handler uses JavaScript to create an HTTP request containing the form data to send to the server to be handled by the route’s action.
The route’s action handler will have access to the form data in the first parameter and can access environment variables from the second parameter (a RequestEvent object).
Inside the route’s action handler, we can construct and send the HTTP request to OpenAI using the data we got from the form, and the API keys we pulled from the environment variables.
With the OpenAI response, we can prepare the data to send back to the client.
The client receives the response from the action and can update the page accordingly.

Here’s what my final component looks like, including some Tailwind classes and a slightly different template.

import { component$ } from "@builder.io/qwik";
import { routeAction$, Form } from "@builder.io/qwik-city";

export const usePromptAction = routeAction$(async (formData, requestEvent) => {
  const OPENAI_API_KEY = requestEvent.env.get('OPENAI_API_KEY')

  const prompt = formData.prompt
  const body = {
    model: 'gpt-3.5-turbo',
    messages: [{ role: 'user', content: prompt }]
  }

  const response = await fetch('https://api.openai.com/v1/chat/completions', {
    method: 'post',
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${OPENAI_API_KEY}`,
    },
    body: JSON.stringify(body)
  })
  const data = await response.json()

  return data.choices[0].message.content
})

export default component$(() => {
  const action = usePromptAction()

  return (
    <main class="max-w-4xl mx-auto p-4">
      <h1 class="text-4xl">Hi 👋</h1>

      <Form action={action} class="grid gap-4">
        <div>
          <label for="prompt">Prompt</label>
          <textarea name="prompt" id="prompt">
            Tell me a joke
          </textarea>
        </div>

        <div>
          <button type="submit" aria-disabled={action.isRunning}>
            {action.isRunning ? 'One sec...' : 'Tell me'}
          </button>
        </div>
      </Form>

      {action.value && (
        <article class="mt-4 border border-2 rounded-lg p-4 bg-[canvas]">
          <p>{action.value}</p>
        </article>
      )}
    </main>
  );
});

Conclusion

All right! We’ve gone from a script that uses AI to get mediocre jokes to a full-blown application that securely makes HTTP requests to a backend that uses AI to get mediocre jokes and sends them back to the frontend to put those mediocre jokes on a page.

You should feel pretty good about yourself.

But not too good, because there’s still room to improve.

In our application, we are sending a request and getting an AI response, but we are waiting for the entirety of the body of that response to be generated before showing it to the users. And these AI responses can take a while to complete.

If you’ve used AI chat tools in the past, you may be familiar with the experience where it looks like it’s typing the responses to you, one word at a time, as they’re being generated. This doesn’t speed up the total request time, but it does get some information back to the user much sooner and feels like a faster experience.

In the next post, we’ll learn how to build that same feature using HTTP streams, which are fascinating and powerful but also can be kind of confusing. So I’m going to dedicate an entire post just to that.

I hope you’re enjoying this series and plan to stick around. In the meantime, have fun generating some mediocre jokes.

Thank you so much for reading. If you liked this article, and want to support me, the best ways to do so are to share it, sign up for my newsletter, and follow me on Twitter.

Originally published on austingil.com.

AI for Web Devs: Project Introduction & Setup

Austin Gil — Fri, 12 Jan 2024 15:39:00 +0000

If you’re anything like me, you’ve noticed the massive boom in AI technology. It promises to disrupt not just software engineering but every industry.

THEY’RE COMING FOR US!!!

Just kidding ;P

I’ve been bettering my understanding of what these tools are and how they work, and decided to create a tutorial series for web developers to learn how to incorporate AI technology into web apps.

In this series, we’ll learn how to integrate OpenAI‘s AI services into an application built with Qwik, a JavaScript framework focused on the concept of resumability (this will be relevant to understand later).

Here’s what the series outline looks like:

We’ll get into the specifics of OpenAI and Qwik where it makes sense, but I will mostly focus on general-purpose knowledge, tooling, and implementations that should apply to whatever framework or toolchain you are using. We’ll be working as closely to fundamentals as we can, and I’ll point out which parts are unique to this app.

Here’s a little sneak preview.

I thought it would be cool to build an app that takes two opponents and uses AI to determine who would win in a hypothetical fight. It provides some explanation and the option to create an AI-generated image. Sometimes the results come out a little wonky, but that’s what makes it fun.

I hope you’re excited to get started because in this first post we are mostly going to work on…

Boilerplate 😒

Prerequisites

Before we start building anything, we have to cover a couple of prerequisites. Qwik is a JavaScript framework, so we will have to have Node.js (and NPM) installed. You can download the most recent version, but anything above version v16.8 should work. I’ll be using version 20.

Next, we’ll also need an OpenAI account to have access to their API.

At the end of the series, we will deploy our applications to a VPS (Virtual Private Server). The steps we follow should be the same regardless of what provider you choose. I’ll be using Akamai’s cloud computing services (formerly Linode). New users can go to linode.com/austingil and get $100 in free credits to get started with Akamai.

Setting Up the Qwik App

Assuming we have the prerequisites out of the way, we can open a command line terminal and run the command: npm create qwik@latest. This will run the Qwik CLI that will help us bootstrap our application.

It will ask you a series of configuration questions, then generate the project for you. Here’s what my answers looked like:

If everything worked, open up the project and start exploring.

Inside the project folder, you’ll notice some important files and folders:

/src: contains all application business logic.
/src/components: contains reusable components to build our app with.
/src/routes: responsible for Qwik’s file-based routing. Each folder represents a route (can be a page or API endpoint). To make a page, drop a index.{jsx|tsx} file in the route’s folder.
/src/root.tsx: this file exports the root component responsible for generating the HTML document root.

Start Development

Qwik uses Vite as a bundler, which is convenient because Vite has a built-in development server. It supports running our application locally, and updating the browser when files changes.

To start the development server, we can open our project in a terminal and execute the command npm run dev. With the dev server running, you can open the browser and head to http://localhost:5173 and you should see a very basic app.

Any time we make changes to our app, we should see those changes reflected almost immediately in the browser.

Add Styling

This project won’t focus too much on styling, so this section is totally optional if you want to do your own thing. To keep things simple, I’ll use Tailwind.

The Qwik CLI makes it easy to add the necessary changes, by executing the terminal command, npm run qwik add. This will prompt you with several available Qwik plugins to choose from.

You can use your arrow keys to move down to the Tailwind plugin and press Enter. Then it will show you the changes it will make to your codebase and ask for confirmation. As long as it looks good, you can hit Enter, once again.

For my projects, I also like to have a consistent theme, so I keep a file in my GitHub to copy and paste styles from. Obviously, if you want your own theme, you can ignore this step, but if you want your project to look as amazing as mine, copy the styles from this file on GitHub into the /src/global.css file. You can replace the old styles, but leave the Tailwind directives in place.

Prepare Homepage

The last thing we’ll do today to get the project in a good starting point is make some changes to the homepage. This means making changes to /src/routes/index.tsx.

By default, this file starts out with some very basic text, and an example for modifying the HTML <head> by exporting a head variable. The changes I want to make include:

Removing the head export.
Removing all text except the <h1>. Feel free to add your own page title text.
Adding some Tailwind classes to center the content and make the <h1> larger.
Wrapping the content with a <main> tag to make it more semantic.
Adding Tailwind classes to the <main> tag to add some padding and center the contents.

These are all minor changes that aren’t strictly necessary, but I think it will provide a nice starting point for building out our app in the next post.

Here’s what the file looks like after my changes.

import { component$ } from "@builder.io/qwik";

export default component$(() => {
  return (
    <main class="max-w-4xl mx-auto p-4">
      <h1 class="text-6xl">Hi 👋</h1>
    </main>
  );
});

And in the browser, it looks like this:

Conclusion

That’s all we’ll cover today. Again, this post was mostly focused on getting the boilerplate stuff out of the way so that the next post can be dedicated to integrating OpenAI’s API into our project.

With that in mind, I encourage you to take a moment to think about some AI app ideas that you might want to build. There will be a lot of flexibility for you to put your own spin on things.

I’m excited to see what you come up with, and if you would like to explore the code in more detail, I’ll post it on my GitHub account at github.com/austingil/versus.

Thank you so much for reading. If you liked this article, and want to support me, the best ways to do so are to share it, sign up for my newsletter, and follow me on Twitter.

Originally published on austingil.com.

File Upload Security and Malware Protection

Austin Gil — Fri, 26 May 2023 15:43:58 +0000

Today we’re going to be wrapping up this series on file uploads for the web. If you’ve been following along, you should now be familiar with enabling file uploads on the front end and the back end. We’ve covered architectural decisions to reduce cost on where we host our files and improve the delivery performance. So I thought we would wrap up the series today by covering security as it relates to file uploads.

In case you’d like to go back and revisit any earlier blogs in the series, here’s a list of what we’ve covered so far:

Introduction

Anytime I discuss the topic of security, I like to consult the experts at OWASP.org. Conveniently, they have a File Upload Cheat Sheet, which outlines several attack vectors related to file uploads and steps to mitigate them.

Today we’ll walk through this cheat sheet and how to implement some of their recommendations into an existing application.

For a bit of background, the application has a frontend with a form that has a single file input that uploads that file to a backend.

The backend is powered by Nuxt.js‘ Event Handler API, which receives an incoming request as an “event” object, detects whether it’s a multipart/form-data request (always true for file uploads), and passes the underlying Node.js request object (or IncomingMessage) to this custom function called parseMultipartNodeRequest.

import formidable from 'formidable';

/* global defineEventHandler, getRequestHeaders, readBody */

/**
 * @see https://nuxt.com/docs/guide/concepts/server-engine
 * @see https://github.com/unjs/h3
 */
export default defineEventHandler(async (event) => {
  let body;
  const headers = getRequestHeaders(event);

  if (headers['content-type']?.includes('multipart/form-data')) {
    body = await parseMultipartNodeRequest(event.node.req);
  } else {
    body = await readBody(event);
  }
  console.log(body);

  return { ok: true };
});

All the code we’ll be focusing on today will live within this parseMultipartNodeRequest function. And since it works with the Node.js primitives, everything we do should work in any Node environment, regardless of whether you’re using Nuxt or Next or any other sort of framework or library.

Inside parseMultipartNodeRequest we:

Create a new Promise
Instantiate a multipart/form-data parser using a library called formidable
Parse the incoming Node request object
The parser writes files to their storage location
The parser provides information about the fields and the files in the request

Once it’s done parsing, we resolve parseMultipartNodeRequest‘s Promise with the fields and files.

/**
 * @param {import('http').IncomingMessage} req
 */
function parseMultipartNodeRequest(req) {
  return new Promise((resolve, reject) => {
    const form = formidable({
      multiples: true,
    });
    form.parse(req, (error, fields, files) => {
      if (error) {
        reject(error);
        return;
      }
      resolve({ ...fields, ...files });
    });
  });
}

That’s what we’re starting with today, but if you want a better understanding of the low-level concepts for handling multipart/form-data requests in Node, check out, “Handling File Uploads on the Backend in Node.js (& Nuxt).” It covers low level topics like chunks, streams, and buffers, then shows how to use a library instead of writing one from scratch.

Securing Uploads

With our app set up and running, we can start to implement some of the recommendations from OWASP’s cheat sheet.

Extension Validation

With this technique, we check the uploading file name extensions and only allow files with the allowed extension types into our system.

Fortunately, this is pretty easy to implement with formidable. When we initialize the library, we can pass a filter configuration option which should be a function that has access to a file object parameter that provides some details about the file, including the original file name. The function must return a boolean that tells formidable whether to allow writing it to the storage location or not.

const form = formidable({
  // other config options
  filter(file) {
    // filter logic here
  }
});

We could check file.originalFileName against a regular expression that tests whether a string ends with one of the allowed file extensions. For any upload that doesn’t pass the test, we can return false to tell formidable to skip that file and for everything else, we can return true to tell formidable to write the file to the system.

const form = formidable({
  // other config options
  filter(file) {
    const originalFilename = file.originalFilename ?? '';
    // Enforce file ends with allowed extension
    const allowedExtensions = /\.(jpe?g|png|gif|avif|webp|svg|txt)$/i;
    if (!allowedExtensions.test(originalFilename)) {
      return false;
    }
    return true;
  }
});

Filename Sanitization

Filename sanitization is a good way to protect against file names that may be too long or include characters that are not acceptable for the operating system.

The recommendation is to generate a new filename for any upload. Some options may be a random string generator, a UUID, or some sort of hash.

Once again, formidable makes this easy for us by providing a filename configuration option. And once again it should be a function that provides details about the file, but this time it expects a string.

const form = formidable({
  // other config options
  filename(file) {
    // return some random string
  },
});

We can actually skip this step because formidable’s default behavior is to generate a random hash for every upload. So we’re already following best practices just by using the default settings.

Upload and Download Limits

Next, we’ll tackle upload limits. This protects our application from running out of storage, limits how much we pay for storage, and limits how much data could be transferred if those files get downloaded, which may also affect how much we have to pay.

Once again, we get some basic protection just by using formidable because it sets a default value of 200 megabytes as the maximum file upload size.

If we want, we could override that value with a custom maxFileSize configuration option. For example, we could set it to 10 megabytes like this:

const form = formidable({
  // other config options
  maxFileSize: 1024 * 1024 * 10,
});

The right value to choose is highly subjective based on your application needs. For example, an application that accepts high-definition video files will need a much higher limit than one that expects only PDFs.

You’ll want to choose the lowest conservative value without being so low that it hinders normal users.

File Storage Location

It’s important to be intentional about where uploaded files get stored. The top recommendation is to store uploaded files in a completely different location than where your application server is running.

That way, if malware does get into the system, it will still be quarantined without access to the running application. This can prevent access to sensitive user information, environment variables, and more.

In one of my previous posts, “Stream File Uploads to S3 Object Storage and Reduce Costs,” I showed how to stream file uploads to an object storage provider. So it’s not only more cost-effective, but it’s also more secure.

But if storing files on a different host isn’t an option, the next best thing we can do is make sure that uploaded files do not end up in the root folder on the application server.

Again, formidable handles this by default. It stores any uploaded files in the operating system’s temp folder. That’s good for security, but if you want to access those files later on, the temp folder is probably not the best place to store them.

Fortunately, there’s another formidable configuration setting called uploadDir to explicitly set the upload location. It can be either a relative path or an absolute path.

So, for example, I may want to store files in a folder called “/uploads” inside my project folder. This folder must already exist, and if I want to use a relative path, it must be relative to the application runtime (usually the project root). That being the case, I can set the config option like this:

const form = formidable({
  // other config options
  uploadDir: './uploads',
});

Content-Type Validation

Content-Type validation is important to ensure that the uploaded files match a given list of allowed MIME-types. It’s similar to extension validation, but it’s important to also check a file’s MIME-type because it’s easy for an attacker to simply rename a file to include a file extension that’s in our allowed list.

Looking back at formidable’s filter function, we’ll see that it also provides us with the file’s MIME-type. So we could add some logic enforces the file MIME-type matches our allow list.

We could modify our old function to also filter out any upload that is not an image.

const form = formidable({
  // other config options
  filter(file) {
    const originalFilename = file.originalFilename ?? '';
    // Enforce file ends with allowed extension
    const allowedExtensions = /\.(jpe?g|png|gif|avif|webp|svg|txt)$/i;
    if (!allowedExtensions.test(originalFilename)) {
      return false;
    }
    const mimetype = file.mimetype ?? '';
    // Enforce file uses allowed mimetype
    return Boolean(mimetype && (mimetype.includes('image')));
  }
});

Now, this would be great in theory, but the reality is that formidable actually generates the file’s MIME-type information based on the file extension.

That makes it no more useful than our extension validation. It’s unfortunate, but it also makes sense and is likely to remain the case.

formidable’s filter function is designed to prevent files from being written to disk. It runs as it’s parsing uploads. But the only reliable way to know a file’s MIME-type is by checking the file’s contents. And you can only do that after the file has already been written to the disk.

So we technically haven’t solved this issue yet, but checking file contents actually brings us to the next issue, file content validation.

Intermission

Before we get into that, let’s check the current functionality. I can upload several files, including three JPEGs and one text file (note that one of the JPEGs is quite large).

When I upload this list of files, I’ll get a failed request with a status code of 500. The server console reports the error is because the maximum allowed file size was exceeded.

This is good.

We’ve prevented a file from being uploaded into our system that exceeds the maximum file limit size (we should probably do a better job of handling errors on the backend, but that’s a job for another day).

Now, what happens when we upload all those files except the big one?

No error.

And looking in the “uploads” folder, we’ll see that despite uploading three files, only two were saved. The .txt file did not get past our file extension filter.

We’ll also notice that the names of the two saved files are random hash values. Once again, that’s thanks to formidable default behavior.

Now there’s just one problem. One of those two successful uploads came from the “bad-dog.jpeg” file I selected. That file was actually a copy of the “bad-dog.txt” that I renamed. And THAT file actually contains malware 😱😱😱

We can prove it by running one of the most popular Linux antivirus tools on the uploads folder, ClamScan. Yes, ClamScan is a real thing. Yes, that’s its real name. No, I don’t know why they called it that. Yes, I know what it sounds like.

(Side note: The file I used was created for testing malware software. So it’s harmless, but it’s designed to trigger malware scanners. But that meant I had to get around browser warnings, virus scanner warnings, firewall blockers, AND angry emails from our IT department just to get a copy. So you better learn something.)

OK, now let’s talk about file content validation.

File Content Validation

File content validation is a fancy way of saying, “scan the file for malware”, and it’s one of the more important security steps you can take when accepting file uploads.

We used ClamScan above, so now you might be thinking, “Aha, why don’t I just scan the files as formidable parses them?”

Similar to MIME-type checking, malware scanning can only happen after the file has already been written to disc. Additionally, scanning file contents can take a long time. Far longer than is appropriate in a request-response cycle. You wouldn’t want to keep the user waiting that long.

So we have two potential problems:

By the time we can start scanning a file for malware, it’s already on our server.
We can’t wait for scans to finish before responding to user’s upload requests.

Bummer…

Malware Scanning Architecture

Running a malware scan on every single upload request is probably not an option, but there are solutions. Remember that the goal is to protect our application from malicious uploads as well as to protect our users from malicious downloads.

Instead of scanning uploads during the request-response cycle, we could accept all uploaded files, store them in a safe location, and add a record in a database containing the file’s metadata, storage location, and a flag to track whether the file has been scanned.

Next, we could schedule a background process that locates and scans all the records in the database for unscanned files. If it finds any malware, it could remove it, quarantine it, and/or notify us. For all the clean files, it can update their respective database records to mark them as scanned.

Lastly, there are considerations to make for the front end. We’ll likely want to show any previously uploaded files, but we have to be careful about providing access to potentially dangerous ones. Here are a couple different options:

After an upload, only show the file information to the user that uploaded it, letting them know that it won’t be available to others until after it’s been scanned. You may even email them when it’s complete.
After an upload, show the file to every user, but do not provide a way to download the file until after it has been scanned. Include some messaging to tell users the file is pending a scan, but they can still see the file’s metadata.

Which option is right for you really depends on your application use case. And of course, these examples assume your application already has a database and the ability to schedule background tasks.

It’s also worth mentioning here that one of the OWASP recommendations was to limit file upload capabilities to authenticated users. This makes it easier to track and prevent abuse.

Unfortunately, databases, user accounts, and background tasks all require more time than I have to cover in today’s article, but I hope these concepts give you more ideas on how you can improve your upload security methods.

Block Malware at the Edge

Before we finish up today, there’s one more thing that I want to mention. If you’re an Akamai customer, you actually have access to a malware protection feature as part of the web application firewall products. I got to play around with it briefly and want to show it off because it’s super cool.

I have an application up and running at uploader.austingil.com. It’s already integrated with Akamai’s Ion CDN, so it was easy to also set it up with a security configuration that includes IP/Geo Firewall, Denial of Service protection, WAF, and Malware Protection.

I configured the Malware Protection policy to just deny any request containing malware or a content type mismatch.

Now, if I go to my application and try to upload a file that has known malware, I’ll see almost immediately the response is rejected with a 403 status code.

To be clear, that’s logic I didn’t actually write into my application. That’s happening thanks to Akamai’s malware protection, and I really like this product for a number of reasons.

It’s convenient and easy to set up and modify from within the Akamai UI.
I love that I don’t have to modify my application to integrate the product.
It does its job well and I don’t have to manage maintenance on it.
Last, but not least, the files are scanned on Akamai’s edge servers, which means it’s not only faster, but it also keeps blocked malware from ever even reaching my servers. This is probably my favorite feature.

Due to time and resource restrictions, I think Malware Protection can only scan files up to a certain size, so it won’t work for everything, but it’s a great addition for blocking some files from even getting into your system.

Closing Thoughts

It’s important to remember that there is no one-and-done solution when it comes to security. Each of the steps we covered have their own pros and cons, and it’s generally a good idea to add multiple layers of security to your application.

Okay, that’s going to wrap up this series on file uploads for the web. If you haven’t yet, consider reading some of the other articles.

Please let me know if you found this useful, or if you have ideas on other series you’d like me to cover. I’d love to hear from you.

Thank you so much for reading. If you liked this article, and want to support me, the best ways to do so are to share it, sign up for my newsletter, and follow me on Twitter.

Originally published on austingil.com.

DEV Community: Austin Gil

The C̶a̶k̶e̶ User Location is a Lie!!!

Location, Location, Location

Getting User Location

Getting User Location From the User

Getting User Location From the Device

Getting User Location From Their IP Address

Getting User Location From Edge Compute

This is Why We Can’t Have Nice Things!

Can’t trust the user

Can’t trust the device

Can’t trust the IP address

Can’t trust edge compute

What Can We Do About It?

Content Translation

Weather App

Store Locator

Regional Pricing

Cookie Banners

Closing

I Deployed My Own Cute Lil’ Private Internet (a.k.a. VPC)

Background

Demo

Evidence

Code

1. Configure Terraform Provider

2. Set Up VPC and VPC Subnet

3. Set Up Application Servers

4. Set Up Database Servers

Closing

Advanced Architecture for AI Application (AKA AAAA!)

Basic Application Architecture

Add Database Architecture

Bring in Edge Compute

Cache Those Images

Voilà

AI for Web Devs: Deploying Your AI App to Production

Setup Runtime Adapter

Push Changes to Git Repo

Prepare Production Server

Upload, Build, & Run App

The Missing Steps

Hell yeah! We did it!

AI for Web Devs: Addressing Bugs, Security, & Reliability

Dealing With Bad HTTP Requests

Dealing With Bad User Input

Dealing With Injection Attacks

Conclusion

AI for Web Devs: AI Image Generation

Create A Dialog Component

Generate AI Images with OpenAI

Provide AI Images with Art Direction

Request an AI-Generated Image

Show the Image in the Dialog

Closing

AI for Web Devs: Prompt Engineering

Start Adapting the UI

Side-Quest: global.d.ts

Adjust the Backend

Begin Prompt Engineering

Character Building

Understanding Tokens

Reducing Tokens

Introducing LangChain

Identifying the Winner

Review

AI for Web Devs: What Are Neural Networks, LLMs, & GPTs?

Prerequisites

Refactor Server Logic

Refactor Client Logic

Enable Streaming on the Server

Enable Streaming on the Client

Stream OpenAI Response

Review

Conclusion

AI for Web Devs: Faster Responses with HTTP Streaming

Prerequisites

Refactor Server Logic

Refactor Client Logic

Enable Streaming on the Server

Side-Quest: `global.d.ts`