DEV Community: Will Dady

Getting to the meat and potatoes of serverless recipe parsing with Amazon Bedrock

Will Dady — Sat, 20 Jul 2024 05:28:25 +0000

If you're like me you've probably visited a few recipe websites in your time and had the unfortunate experience of having to scroll through the author's life story before getting to the actual recipe. I recently stumbled across an interesting website which is able to take a URL and extract just the recipe without all the surrounding fluff. This got me thinking... could I build something similar on AWS with serverless technologies and generative AI?

Lately I have been experimenting with various large language models on Amazon Bedrock. I have been particularly interested in the latest offering from Anthropic with it's Claude 3 family of models. My idea was to see if I could use Claude to extract a recipe from a web page and return it as a structured JSON object.

Crafting the prompt

I hypothesised that Anthropic's Hiaku model would make light work of finding a recipe in a body of text assuming it is given a well crafted prompt. Luckily, Anthropic's prompt engineering documentation provides some excellent guidance on how best to craft a prompt to work with their family of LLMs.

I began with some low-fi manual testing directly in the AWS Bedrock Chat Playground. I figured my eventual solution would scrape HTML from a target website and inject it into the prompt but in the meantime I'd need to do this manually. This is as simple as navigating to a recipe in my browser, viewing the page source and copying the raw HTML. I then paste the raw HTML into my prompt with some surrounding instructional text.

The Amazon Bedrock Chat Playground showing a well-formed JSON response from Claude

After some experimentation in the Bedrock console I was able to get Claude to consistently parse a recipe from a body of text and return it as structured JSON. Pretty cool!

I ended up settling on the following prompt template:

Read the HTML fragment contained inside the following <document></document> XML tags and extract the recipe.

<document>
<%- document %>
</document>

If you are able to successfully find a recipe in the document output your result as a JSON object with the following fields:

title: A string containing the receipe title
description: A string containing a summary of what the recipe makes
ingredients: An array of strings where each is a single ingredient
steps: An array of strings where each describes a step required to complete the recipe

<example>
{
  "title": "Pasta Aglio E Oilio",
  "description": "Pasta Aglio e Olio is a classic Italian dish known for its simplicity and flavor. Translating to 'pasta with garlic and oil' it's a traditional recipe originating from the region of Naples. The dish features spaghetti cooked al dente and then tossed with minced garlic, olive oil, red pepper flakes, and sometimes parsley. Despite its minimal ingredients, Pasta Aglio e Olio packs a punch of flavor, with the garlic-infused oil coating each strand of pasta for a deliciously satisfying meal. It's a go-to dish for a quick, easy, and tasty dinner option.",
  "ingredients": [
    "Half a Lemon",
    "Fresh Grated Parmesan",
    "2 - 3 Tbsp Butter",
    "1 Cup Chopped Italian Parsley",
    "1 Pinch Chili Flakes or Hot Pepper FlakesFresh",
    "Ground Pepper",
    "2 Tbsp Salt (for pasta water)",
    "3 Cloves Thin Sliced Fresh Garlic",
    "1 Cup Pasta Water",
    "1/4 Cup Olive Oil",
    "450g Spaghetti"
  ],
  "steps": [
    "Slice garlic cloves thin (do not chop) and remove stems from parsley. Chop parsley fine and set all this aside. Have all your ingredients handy and ready since you will need to get to them quickly.",
    "Bring water with 2 TBSP salt to a boil and add pasta. Cook for 10-12 minutes till al dente. Save 2 cups of the hot pasta water (for possibly creating more liquid) before draining.",
    "About 5 minutes before your pasta is ready, heat the olive oil in large pan on medium and add the sliced garlic. Toss gently and after about 30 seconds add the hot pepper flakes and salt and pepper. Cook and toss for about 30 seconds.",
    "Add parsley, cooked pasta, pasta water to your liking, butter and squeeze in the half lemon (watch for seeds). Stir gently and bring to gentle boil. Add more fresh pepper and salt to taste. Toss the mixture well to fully coat and turn off heat.",
    "Let mixture set about 30 seconds so liquid thickens. Serve hot and top with fresh grated parmesan."
  ]
}
</example>

If you are not able to find a recipe in the document respond with a JSON object with the following fields:

error: A string stating "No recipe found in document"
cause: A string describing the error

<example>
{
  "error": "No recipe found in document",
  "cause": "The document appears to be about Harley Davidson motocycles"
}
</example>

<example>
{
  "error": "No recipe found in document",
  "cause": "The document appears to be about cats"
}
</example>

Take note of the following:

I am using XML tags to delineate parts of the prompt as recommended in Anthropic's documentation - Use XML tags
The <document> tags is where you put the HTML you want Claude to read. The <%- document %> contained within is an ejs placeholder. More on this shortly.
I am using <example> tags to show how I want the JSON to be structured. I'm also telling Claude what to do if it is unable to find a recipe in the document.
I am explicitly asking for JSON output.

Building a solution with the AWS CDK

Now that I have a prompt which I am satisfied works, I set out to automate it.
I began by creating a new AWS CDK project using Typescript.

The bulk of the solution will be driven by one of my favourite serverless AWS services, Step Functions! Using the CDK I create a Step Functions state machine with it's type set to Express. We need to use the Express type as we will eventually expose the state machine via an API.

The state machine workflow performs these main tasks:

Scrape (and sanitise) the HTML from a web page
Generate the prompt by injecting the HTML into our prompt template
Invoke Amazon Bedrock with our finalised prompt
Handle the case where Claude was not able to find a recipe in the document

Step Functions workflow diagram

Scraping the web page

The first step of our state machine scrapes the contents of a web page containing our recipe. To do this I opted to create a simple Node.js based Lambda function using Typescript which takes a url parameter and uses fetch to... ahem... fetch the target web page.

At this point I could simply return the raw HTML to advance the state machine but that would be extremely wasteful as there is a lot of redundant markup which serves little value, not to mention the longer the prompt the more it costs. We can do better!

I opted to use cheerio, a nifty HTML parsing library to clean-up the HTML before returning it from the function.
Using cheerio I:

Extract the content of <body> tag
Delete elements which have little semantic meaning to Claude such as img, video, svg etc.
Delete all attributes from elements
Delete all <!-- > comments

I also do some final manipulation of the HTML string to remove excessive whitespace before returning the result.

Astute readers may note this is a far from ideal way of scraping a web page as it fails to consider dynamic content loaded after page load. My assumption is most recipe websites will deliver the complete recipe in the initial HTML sent from the server to aid SEO. That said, a headless Chromium + Puppeteer setup would likely be a better choice in a production environment.

Constructing the prompt

For the second step of our state machine I wrote this short Lambda function. It takes the sanitised HTML output of the previous step and combines it with our prompt template. To do this I use ejs to replace the <%- document %> placeholder in the prompt template with our sanitised HTML.

import * as ejs from 'ejs';

const PROMPT_TEMPLATE = `...`; // Omitted for brevity

interface Payload {
  input: string;
}

export const handler = async ({ input }: Payload) => {
  const output = ejs.render(PROMPT_TEMPLATE, { document: input });
  return { output };
};

Invoking Amazon Bedrock

Now we have our finalised prompt returned from the previous step we can invoke the model with it. What's nice is AWS Step Functions has a direct integration with Bedrock which means we can invoke it directly without first having to write another Lambda function. Here is what the BedrockInvokeModel task looks like as defined in the CDK app.

const model = bedrock.FoundationModel.fromFoundationModelId(
  this,
  'Model',
  new bedrock.FoundationModelIdentifier(
    'anthropic.claude-3-haiku-20240307-v1:0',
  ),
);

const aiTask = new tasks.BedrockInvokeModel(this, 'Invoke Model', {
  model,
  body: sfn.TaskInput.fromObject({
    anthropic_version: 'bedrock-2023-05-31',
    max_tokens: 1024,
    messages: [
      {
        role: 'user',
        content: sfn.JsonPath.stringAt('$.Payload.output'),
      },
      {
        role: 'assistant',
        content: '{',
      },
    ],
  }),
  resultSelector: {
    output: sfn.JsonPath.stringToJson(
      sfn.JsonPath.format(
        '{}{}',
        '{',
        sfn.JsonPath.stringAt('$.Body.content[0].text'),
      ),
    ),
  },
});

This takes the prompt I output from the previous step ($.Payload.output) and invokes the Anthropic Claude 3 Haiku model as it's input.

You might be wondering what that second 'assistant' message is. That is a message pre-fill which gives Claude a starting point on how to respond to the 'user' input. As instructed in our prompt template we always want Claude to respond with JSON. This combined with the prompt entered in the 'user' message helps guide Claude on how to respond. You can read more about how this works on Anthropic's website - Control output format (JSON mode).

An interesting quirk of pre-filling the opening JSON brace is that it is not included in the message response we receive from Bedrock. You'll notice in the resultSelector I am using sfn.JsonPath.format, which is one of Step Function's intrinsic functions, to prepend the missing opening brace. The resulting string is then converted to JSON with the sfn.JsonPath.stringToJson function.

Exposing via API Gateway

The last piece of the puzzle is to expose our state machine via Amazon API Gateway. Within the same CDK app I instantiate a new RestApi which proxies requests directly to my state machine.

const api = new apigateway.RestApi(this, 'StepFunctionsRestApi');
api.root.addProxy({
  defaultIntegration:
    apigateway.StepFunctionsIntegration.startExecution(stateMachine),
});

Once deployed the CDK CLI will output the URL of the newly created Rest API. Copy the URL so we can test it in our web browser.

Test it!

At this point all that's left to do is test it out on some web pages. To do this all you need to do is navigate to a website containing a recipe and paste your API URL in-front of the url in your browser's address bar.

For example, my Rest API is available at https://c7jdzx7r36.execute-api.ap-southeast-2.amazonaws.com/prod/. If I want to extract the recipe from https://www.recipetineats.com/caramel-slice/ I simply append the latter to the former e.g.

https://c7jdzx7r36.execute-api.ap-southeast-2.amazonaws.com/prod/https://www.recipetineats.com/caramel-slice/

After a few seconds, voila!... the recipe extracted from the page as JSON!

{
  "output": {
    "title": "Caramel Slice",
    "description": "This is a Caramel Slice that works as promised – the creamy caramel sets perfectly and will never be runny, the chocolate won't crack when cutting it and the caramel won't ooze out. It's an easy recipe with no thermometer required.",
    "ingredients": [
      "1 cup flour, plain/all purpose",
      "1/2 cup brown sugar, loosely packed",
      "1/2 cup desiccated coconut (US: sweetened finely shredded coconut)",
      "125g / 4.5oz  unsalted butter, melted",
      "125g / 4.5oz unsalted butter, roughly chopped",
      "1/2 cup (80g) brown sugar, loosely packed",
      "1 tsp vanilla extract (or essence)",
      "395g / 14oz sweetened condensed milk (1 can, 300ml)",
      "200g / 7oz dark or milk melting chocolate (US: semi-sweet chocolate chips)",
      "1 tbsp vegetable oil"
    ],
    "steps": [
      "Preheat oven to 180°C/350°F (fan 160°C)",
      "Grease and line a 28x 18cm (lamington pan) / 7\" x 11\" rectangle pan with baking/parchment paper (Note 2). Have overhang for ease of removal.",
      "Mix together Base ingredients and press into a pan (I use an egg flip)",
      "Bake for 15 minutes until the surface is golden. Cool in fridge if you have time (Note 3).",
      "Lower oven to oven to 160°C/320°F (fan 140°C)",
      "Place butter, sugar and vanilla in a saucepan over medium low heat. When the butter is melted, whisk to combine with sugar, then just leave it until it comes to a simmer.",
      "When bubbles appear, add condensed milk. Whisk constantly for 5 minutes (Note 4), until you start getting some big slow bubbles on the base.",
      "Once bubbles start appearing, whisk for 1 minute, then pour onto Base. Tilt pan to spread evenly.",
      "Bake for 12 minutes. Don't worry if you get splotchy brown bits (this happens with ovens that don't distribute heat evenly).",
      "Cool on counter for 20 minutes then refrigerate 30 minutes - bottom of pan should be warm but surface cool (not cold) to touch. (Note 5)",
      "Place chocolate and oil in a microwave proof bowl. Microwave in 30 second bursts, stirring in between, until chocolate is fully melted (takes me 4 x 30 sec).",
      "Pour over caramel, spread with spatula. Then gently shake pan to make the surface completely flat.",
      "Refrigerate 1 hour or until set. Remove from fridge and leave out for 5 minutes to take chill out of chocolate slightly. Then cut into bars or squares to serve!"
    ]
  }
}

Here is an example of what is output when you provide a URL to something which doesn't contain a recipe.

https://c7jdzx7r36.execute-api.ap-southeast-2.amazonaws.com/prod/https://news.ycombinator.com/

{
  "error": "No recipe found in document",
  "cause": "The document appears to be a Hacker News article listing and does not contain any recipes."
}

Closing thoughts

I had a lot of fun building this and hopefully this post highlights how combining serverless technologies with generative AI can be used for novel outcomes. This is very much a toy/experimental project and is not intended for serious production use. To make this safer for use in production a number of additional features would need to be added such as improved error handing, authentication at the API layer and caching to name a few.

If you'd like to try it out yourself you can find the complete CDK app here https://github.com/willdady/recipe-extractor-cdk

Photo by Syd Wachs on Unsplash

Photo mosaics with Rust

Will Dady — Sat, 10 Aug 2019 23:24:36 +0000

When I was a kid I remember seeing a poster of Yoda from Star Wars which was made up from various screenshots from the movie. I remember being facinated how several tiny images could be arranged to create a larger image and that how the colour of each small image contributed to the larger picture overall.

The idea of arranging small coloured tiles to create images has been around since 300 BC and artwork created in such a way is called a Mosaic which Wikipedia describes as:

a piece of art or image made from the assembling of small pieces of colored glass, stone, or other materials.

While the technique of arranging coloured tiles to form an image has been around for hundreds of years, photo mosaics are a relatively new take on the art style.

Photo mosaics

The first mainstream use of a photo mosaic I recall was on one of the official posters for the 1998 film staring Jim Carrey, The Truman Show.

A photo mosaic poster for The Truman Show (1998)

You can see in the following close-up of the poster, specifically Jim Carrey's right eye, the image is made up of various scenes from the film. Interestingly each tile is tinted rather than solely relying on the actualy colour in the frame. It's a forgivable workaround considering there probably isn't that many fleshy-beige coloured scenes in the movie to sample from.

A close-up of the poster for The Truman Show (1998)

Writing a photo-mosaic generator

As I've been learning the Rust programming language lately I thought creating a photo-mosaic generator could be a fun project. Rust is a relatively new systems programming language akin to C/C++ in terms of performance. What makes Rust unique is the way it manages memory compared to other low-level languages.

I knew a generator like this would require a large set of tile images to sample from. The original plan was for it to only create mosaics from emoji which is how it got it's name emosaic (emoji + mosaic = emosaic) but as I progressed it made more sense to keep in generic and let the user provide their own pool of tile images.

Similar to my previous (and first) Rust project swirlr, I used dependencies from crates.io, Rust's equivalent to NPM for Node.js. The main libraries I used are image, for loading an manipulating images and sampling pixels, and clap a command line argument parser.

Emosaic works by taking a directory of images used as tiles and a source image.

emosaic /path/to/tiles/ source.png

The output file will be saved to the current directory as output.png though a custom output path can be provided via the -o option. Output is always in PNG format.

The app works by first reading all images in the tiles directory. Each image's pixels are looped over to find the average colour of each image. I found it necessary to exclude any tiles where more than 50% of it's pixels are completely transparent as these didn't look good in the final output. The image's path and average colour is then added to a Tile struct. Each Tile struct was then added to a TileSet struct which represented all available tiles to choose from in the next step.

Next the source image is loaded and for each pixel in the source image we call the TileSet's closest_tile(&self, rgba: Rgba<u8>) -> Tile method. This method works by naively looping over each tile and comparing the distance between colours. We say "distance" as we calculate how close one colour is to another by treating each as a point in 3D space RGB -> XYZ (alpha is ignored). The resulting Tile is then added to a Map keyed by the pixel colour that way subsequent lookups can simply check if the Tile for a given colour has already been found, this is a particularly useful optimization as lossy formats like JPEG will repeat colours often.

Once the tile is found it's copied to the same row/column coordinates in the output image factoring in the desired tile size. The tile size can be configured as the optional command line option -t which defaults to 16. For example a 100x100 source image yields an output image of 1600x1600 (100 x 16 = 1600) so it's important the source image is small.

Further improvements

Im pretty happy with how this turned out and it's super fast thanks to the low-level nature of Rust. For example, the Marilyn Monroe image above is generated from a directory of 2625 tiles, each with dimensions of 64x64 and a source image with with dimensions of 100x100. The command completes in about 2.9 seconds on a 2017 Macbook Pro.

100x100 source image

Some further optimisations could be made such as caching the TileSet to disk that way the program doesn't need to read potentially thousands of tile images every time the app is run. Also at the time of writing it does not provided an option to 'tint' the output image like the Truman Show example above.

Overall I'm really liking Rust. It has a very steep learning curve but it's rewarding once the borrow checker stops complaining. It's nice to feel confident that if the app compiles it will run safely. I find small projects like this are a great way to get familiar with a new programming language.

Feel free to checkout the complete source code for emosaic on github.

willdady / emosaic

Mosaic image generator written in Rust!

emosaic

Mosaic generator written in Rust!

Building

To build make sure you have rust installed.

cargo build --release

Once compiled, the binary can be found at target/release/emosaic in the repository root.

Usage

The command expects a path to a directory containing square 'tile' images and a source image.

emosaic /path/to/tile/images/ source.png

Modes

The strategy used to generate the mosaic is controlled by the -m, --mode option.

1to1 (Default)

For each pixel in the source image a tile with the nearest matching average color will be emitted.

Assuming a source image with dimensions 100x100 and default tile size of 16 the output image will be 1600x1600.

4to1

For every 2x2 pixels one tile will be emitted. Tiles are divided into 2x2 segments and the average colour of each segment is stored. The tile with the nearest average color in each segment to the target pixels will be chosen. This mode…

View on GitHub

Creating a remote BitTorrent box with Terraform, Transmission and Docker on AWS

Will Dady — Sun, 14 Jul 2019 12:47:12 +0000

My head has very much been in the DevOps space lately as I've been more and more hands on with AWS. I got thinking about an idea I had some time ago... What would be involved in running an ephemeral BitTorrent client on AWS with downloads automatically saved to S3?

My ultimate goal was to be able to spin up and tear down a remote BitTorrent client all from the command line. I knew I wanted to run Transmission as I had used it in the past as it has a web interface for adding and removing torrents and monitoring downloads.

VPN

As I'm hosting this in the cloud I really wanted the BitTorrent traffic to route via a VPN rather than directly to the host instance. When I first started experimenting with this idea I begain by spinning up a t2.micro instance via the AWS console, connecting via SSH and installing OpenVPN directly on a host. I quickly discovered the hard way than starting OpenVPN while connected to the host over SSH wouldn't work as I guess by default it takes over the host's network traffic, immediately dropping my SSH connection and bricking the server until it's rebooted via the AWS console.

This lead me to Docker. Having OpenVPN containerised felt like a more sensible solution as it would allow me to SSH into the host instance to tinker with the setup. I figured someone must have already created an OpenVPN docker image and after some googling I found that github user haugene had created an aptly named image, docker-transmission-openvpn, which bundles both OpenVPN and Transmission together. Bonus!

Syncing completed downloads to S3

Once downloads complete Transmission copies files to the /data/completed directory. I needed to figure out a way to automatically copy completed downloads to S3. For this I decided to write my own utility in Go which would simply watch the contents of a directory for new files and upload them to an S3 bucket. This utility is called go-watch-s3 and I have a dedicated blog post about this here.

Both the Transmission container and the container running go-watch-s3 share a bind-mounted directory so when Transmission wrote files to it's /data/completed directory go-watch-s3 would see the new files and upload them. Pretty neat!

Provisioning the instance with Terraform

The provisioning is handled by a Terraform configuration which sets up our infrastructure. It creates the EC2 instance, a security group so we can access the Transmission UI, and the appropriate IAM role, policy and instance profile so the instance is able to write files to our existing S3 bucket.

Installing software on the instance is handled as part of a user data script which runs when the instance first launches and is defined as part of the aws_instance resource type in Terraform. This script is actually relatively simple as it just installs Docker, pulls the above mentioned images and runs them. I had considered using Docker Compose as it's usually a nicer experience when orchestrating multiple containers but went ahead with vanilla docker run commands instead.

Finally, all that's left to do is to apply the configuration with terraform apply to create our resources on AWS. Once the configuration is applied the url of the Transmission web UI will be output however you will need to wait a few minutes for the startup script to do it's thing. Once the UI is available torrents can be added with completed downloads auto-magically copied to S3!

Once you're finished, destroying the infrastructure you just created can be torn down with terraform destroy.

Stating the obvious

In summary I feel like this was a fun 'toy' project though I don't personally feel the need to improve it further. The itch has been scratched if you will. To see the complete Terraform configuration check out terraform-transmission-aws on github.

While there is nothing illegal about BitTorrent from a pure technology standpoint, pirating content may well fast-track you to a reprimand from AWS or worse your account closed.