DEV Community: Ed Miller

Meet Stache Forcache, a Movember-themed AI character created using Meta AI Studio

Ed Miller — Tue, 31 Dec 2024 01:27:52 +0000

Note: AI Studio is currently only available in the United States.

We’re a little over a week into Movember. On the 1st of the month, I wrote a blog on LinkedIn about how I will Mo My Own Wayby building a men’s health chatbot. For my first exploration, I used Meta AI Studio. AI Studio enables you to quickly create AI characters and share them across Messenger, Instagram and WhatsApp.

In less than an hour using AI Studio, I was able to create and customize an AI character: a hipster, Australian barber with a passion for men’s health and Movember. Before I walk through how I did this, let me first introduce you to Stache Forcache:

AI Character: Stache Forcache

Getting started with AI Studio

To get create your own AI character, head over to https://aistudio.instagram.com/ (you can also do this in the apps, but I’m using a web browser on a laptop). You should see something like this:

AI Studio front page

If you have already developed AI characters, you can find them under Your AI characters . You can use the Discover search box to find other people’s characters. For now, let’s get started by clicking Create an AI which will let you create a custom AI character or get started from examples.

Create an AI character

Creating an AI character

You start by describing what your AI character is and what is does. At this stage, we can provide a high level description, we’ll get into the details later. Cycle through the examples for ideas. You can modify one of the examples or write your own under “Custom AI character”. Once you are happy with your description, click Next.

Add details about your AI character

AI Studio will generate a name, tagline and some options for an avatar. You can edit these now or wait until later. Once you’re ready, click Create AI. This will create your character with prefilled details and take you to AI Studio’s main editing page.

Edit details page

Example Chat Panel

On the right side of the page, you will see an example chat where you can talk to your AI while your are adding information.

Test chat panel

You can also edit the character’s avatar, name and tagline by clicking the respective pencil icon. The name and tagline are simple text fields. Editing the avatar will pop up a dialog box.

Stache’s avatar

Click the pencil in the description field to update the description and generate new images. Pick on that you like and click Save . You can be quite detailed in the description. Here’s what I have for Stache (some of this was suggested by AI Studio and some modified by me):

A warm, medium shot of the hipster barber character standing in front of a vintage, wooden barber chair, with a friendly smile on his face. He has a bushy, handlebar mustache and no beard. He is wearing a crisp, white apron with a black mustache embroidered on the chest. His sleeves are rolled up, and he’s holding a straight razor in one hand, while the other is gently stroking the back of the chair. The background is a warm, golden color, with subtle, muted tones of green and brown, evoking a sense of a classic, Australian barbershop. The overall style is reminiscent of a 35mm film still, with a slight, nostalgic flair.

Refining your character’s knowledge

On the left side of the page is where you can really customize your character’s behavior. There are two tabs, the first of which is Knowledge.

What does your AI character do?

Here you will describe what your character does and who it is. Add some details about what your character knows about and give it a personality. Here’s what I have for Stache:

Description of Stache

Full text:

Stache is a hipster barber with a bushy mustache and a passion for men's health. He's always ready to share his expertise and advice on a wide range of men's health concerns, especially mental health, prostate cancer and testicular cancer. With a friendly and approachable demeanor, Stache makes it easy for men to open up about their health concerns and get the support they need. Whether you're looking for advice on how to improve your mental health or seeking guidance on how to talk to your friends about health issues, Stache is here to help.

Instructions

In this section you can define how your character behaves. You can have up to a dozen instructions for your character. It’s best if you write the instruction in third person. For example:

Stache is knowledgeable about men’s health issues, especially prostate cancer, testicular cancer and mental health.

To help promote Movember, I included:

Stache participates in the Movember movement and encourages other to get involved.

Since Stache is Australian (Movember strarted in Australia), I have included:

Stache speaks with a friendly, Aussie accent and uses colloquialisms like “mate” and “g’day” to make users feel at ease.

Example dialogue

To really bring your characters voice to life, you can add example dialog. You can use this to amplify personality traits, specify formatting for responses, etc. I haven’t added any example dialog for Stache yet.

Editing introduction and capabilities

In the Introduction and Capabilities tab, you can define how your AI starts conversations and enable specific capabilities.

How does your AI character greet new people?

Here you define your AI character’s welcome message. This is your characters first impression. Stache always starts with

G’day mate! What’s on your mind? Need some advice on men’s health or just want to chat?

Prompts for people to start the conversation

These are suggestions to get the conversation started. You can use these to indicate what the AI is capable of or nudge users in the right direction.

Conversation prompts

What your AI character can do

Here you enable additional capabilities for your AI character. There are three capabilities enabled by default:

Image generation: generate images when asked by the user.
Long-term memory: remembers previous conversations.
Reels sharing: shares reels when specifically asked by user.

There are two optional capabilities:

Dynamic image generation: can generate images without being asked.
Search: can search the internet and share links.

For Stache, I have search turned on, to hopefully get access to the latest information on Movember and men’s health topics. I have dynamic image generation turned off.

AI character capabilities

Publishing your AI character

Once you’re happy with your AI character, you can publish. First, select the audience: everyone, followers, close friends, or only me. Under discoverability, you can set in which apps your AI character will be visible. You can also choose if you want your AI character to appear on your Instagram profile.

Publish your AI character

Once you’re ready, click Publish . For sharing publicly, your AI will be reviewed to ensure it meets the AI Studio policies.

Sharing and insights

When your AI character is ready, you can find it on your AI Studio dashboard under Your AI characters.

AI Studio characters

You can see I have two public AI characters: Stache Forcacheand Andino (an Andean bear that likes to chat about the environment) as well as a private one: GrAIzer (a brown bear in Alaska). Select a character to start a chat. Click the pencil to resume editing a character. Under the °°° menu, you can Copy share link , See insights , and Delete AI.

For example, the share link for Stache Forcache is https://aistudio.instagram.com/ai/8961958923866535/?utm_source=share

Under See insights, you can see statistics like total chats, average message sends, total messages, and positive feedback (as a percentage). I don’t have any data for my AI character so far.

Depending on your discoverability settings, you may also be able to find your AI character in Messenger, Instagram and WhatsApp. Here’s an example of Stache in each app:

Stache Forcache in Messenger, Instagram, and WhatsApp

That’s it! Anyone can create a custom AI character using AI Studio. For additional information, check out the AI Studio Handbook(or click Creation guide in AI Studio).

If you want to learn more about Movember or want to chat about men’s health, give Stache a try. You can also search for Stache Forcache in Messenger, Instagram and WhatsApp.

Give AI Studio a try and let me know what you create!

Originally published at https://www.linkedin.com.

Meet Stache Forcache, a Movember-themed AI created using Amazon PartyRock

Ed Miller — Wed, 27 Nov 2024 01:21:09 +0000

We're nearing the end of Movember. On the 1st of the month, I wrote a blog about how I will Mo My Own Way by building a men's health chatbot. For my first exploration, I used Meta AI Studio (read about it here). In this post, I will describe building a similar chatbot using Amazon PartyRock Playground.

PartyRock is an Amazon playground for building AI-generated apps. It's powered by Amazon Bedrock, so it has powerful generative AI foundation models, but it also uses GenAI to help you create your app without any heavy lifting. Geat started by heading to https://partyrock.aws/.

Get started by clicking Generate app.You will get a popup where you can define what your app will do in plain text.

For example, I have started with:

Stache is a hipster barber with a bushy mustache and a passion for men's health. He's always ready to share his expertise and advice on a wide range of men's health concerns, especially mental health, prostate cancer and testicular cancer. With a friendly and approachable demeanor, Stache makes it easy for men to open up about their health concerns and get the support they need. Whether you're looking for advice on how to improve your mental health or seeking guidance on how to talk to your friends about health issues, Stache is here to help.

PartyRock will generate an app, including a set of widgets with various functionality:

Click Edit to modify widget locations, layout and content:

For example, I removed a few of the widgets, leaving only Chat with Stache. For a chat widget, you can set the prompt, labels and model. For the Prompt, I describe how I want Stache to talk and defined what he knows:

The Labels tab include the widget title, some placeholder text and the initial message:

The Model tab lets you select a model:

I selected Claude 3.5 Sonnet. You can also set model Temperature and Top P, which affect the randomness and creativity. Since I am looking for a more factual chatbot, I have set both to 0.

Once you have everything set up, you can click Leave edit and try it out:

By default, all apps are private. If you want to share, click Share and set the sharing level (public, shared or private):

If you have an especially interesting interaction with your app, you can click Snapshot to save the session. You can find all your snapshots in the Snapshots section.

Here you can get a URL to share or delete the snapshot. For example, you can see the snapshot from a chat with Stache here:

https://partyrock.aws/u/bluevalhalla/b2ofzFpv1/Stache's-Health-Cache/snapshot/bKOgCILud

That's all it takes to create an AI chatbot app using PartyRock. Give it a try and drop a comment below to let me know what you created.

If you linked this post, please consider donating to my Movember fundraising page here:
https://movember.com/m/bluevalhalla

Let's Mo!

Intro to Llama on Graviton

Ed Miller — Wed, 28 Aug 2024 20:19:35 +0000

Note: Banner image generated by AI.

Are you curious about how to supercharge your application with AI while cutting costs? Discover how running Large Language Models on AWS Graviton can offer you the necessary performance at a fraction of the price.

It has been less than two years since ChatGPT changed the virtual face of AI. Since then, large language models (LLMs) have been all the rage. Adding a chatbot in your application may dramatically increase user interaction, but LLMs require complicated and costly infrastructure. Or do they?

After watching the “Generative AI Inference using AWS Graviton Processors” session from the AWS AI Infrastructure Day, I was inspired to share how you can run an LLM using the same Graviton processors as the rest of your application.

In this post, we will:

Set up a Graviton instance.
Following the steps (with some modifications) in "Deploy a Large Language Model (LLM) chatbot on Arm servers” from the Arm Developer Hub to:
- Download and compile llama.cpp
- Download a Meta Llama 3.1 model using huggingface-cli
- Re-quantize the model using llama-quantize to optimize it for the target Graviton platform
- Run the model using llama-cli
- Evaluate performance
Compare different instances of Graviton and discuss the pros and cons of each
Point to resources for getting started

Subsequent posts will dive deeper into application use cases, costs, and sustainability.

Set up a Graviton instance

First, let’s focus on the Graviton3-based r7g.16xlarge. This is a common instance type these days. I’ll be running it in us-west-2. Using the console, navigate to EC2 Instances and select “Launch instances”. There are only a few fields necessary for a quick test:

Name: this is up to you; I have called mine ed-blog-r7g-16xl
Application and OS Images
- AMI: I am using Ubuntu Server 24.04 LTS (the default if you select Ubuntu)
- Architecture: Choose 64-bit (Arm)
Instance type: r7g.16xlarge
Key pair: Select an existing one or create a new one
Configure storage: I’m bumping this up to 32 GiB to make sure I have room for the code and Meta Llama models.

You can leave defaults for the rest, just click “Launch instance” after the Summary.

Once the instance has started, you can connect using your favorite method. For simplicity, I will use the EC2 Instance Connect method, which will provide a terminal in your browser window:

Build and Run Meta Llama 3.1

To build and run Meta Llama 3.1, we will follow the steps (with some modifications) in "Deploy a Large Language Model (LLM) chatbot on Arm servers” from the Arm Developer Hub to:

Download and compile llama.cpp

First, we install any prerequisites:

sudo apt update
sudo apt install make cmake -y
sudo apt install gcc g++ -y
sudo apt install build-essential -y

Then we clone llama.cpp and build it (the -j$(nproc) flag will use all available vCPU cores to speed up compilation):

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j$(nproc)

Finally, we can test it using the help flag:

./llama-cli -h

Download Meta Llama 3.1

Next, we’ll set up a virtual environment for Python packages:

sudo apt install python-is-python3 python3-pip python3-venv -y
python -m venv venv
source venv/bin/activate

Now install HuggingFace Hub and use it to download a 4-bit quantized version of Meta Llama 3.1:

pip install huggingface_hub
huggingface-cli download cognitivecomputations/dolphin-2.9.4-llama3.1-8b-gguf dolphin-2.9.4-llama3.1-8b-Q4_0.gguf --local-dir . --local-dir-use-symlinks False

Re-quantize the model

The model we downloaded is already 4-bit quantized (half-byte per weight). This gives us a 4x improvement in model size compared with the original bfloat16 (2-byte per weight). However, the width of the Scalable Vector Extension (SVE) is different for Graviton3 (2x256-bit SVE) and Graviton4 (4x128-bit SVE2). Graviton2 does not have SVE but will use 2x128-bit Arm Neon technology. To maximize the throughput for each generation, you should re-quantize the model with the following block layouts:

Graviton2: 4x4 (Q4_0_4_4)
Graviton3: 8x8 (Q4_0_8_8)
Graviton4: 4x8 (Q4_0_4_8)

For the Graviton3 instance, we will re-quantize the model using llama-quantize as follows:

./llama-quantize --allow-requantize dolphin-2.9.4-llama3.1-8b-Q4_0.gguf dolphin-2.9.4-llama3.1-8b-Q4_0_8_8.gguf Q4_0_8_8

Run the model

Finally, we can run the model using llama-cli. There are a few arguments we will use:

Model (-m): The optimized model for Graviton3, dolphin-2.9.4-llama3.1-8b-Q4_0_8_8.gguf
Prompt (-p): As a test prompt, we’ll use “Building a visually appealing website can be done in ten simple steps”
Response length (-n): We’ll ask for 512 characters
Thread count (-t): We want to use all 64 of the vCPUs

Here’s the command:

./llama-cli -m dolphin-2.9.4-llama3.1-8b-Q4_0_8_8.gguf -p "Building a visually appealing website can be done in ten simple steps:" -n 512 -t 64

When you run the command, you should see several parameters print out followed by the generated text (starting with the prompt) and finally performance statistics:

Evaluate Performance

The two lines highlighted above are the prompt evaluation time and the text generation time. These are two of the key metrics for user experience with LLMs. The prompt evaluation time relates to how long it takes the LLM to process the prompt and start to respond. The text generation time is how long it takes to generate the output. In both cases, the metric can be viewed in terms of tokens per second (T/s). For our run we see:

Evaluation: 278.2 T/s
Generation: 47.7 T/s

If you run the standard Q4_0 quantization with everything else the same, as with this command:

./llama-cli -m dolphin-2.9.4-llama3.1-8b-Q4_0.gguf -p "Building a visually appealing website can be done in ten simple steps:" -n 512 -t 64

You will see a decrease in performance:

Evaluation: 164.6 T/s
Generation: 28.1 T/s

Using the correct quantization format (Q4_0_8_8, in this case) you get close to 70% improvement on evaluation and generation!

When you are done with your tests, don’t forget to stop the instance!

Comparing Graviton-based instances

Using the process above, we can run the same model on similarly equipped Graviton2 and Graviton4-based instances. Using the optimum quantization format for each, we can see an increase in performance from generation to generation:

Generation	Instance	Quant	Eval (T/s)	Gen (T/s)
Graviton2	r6g.16xlarge	Q4_0_4_4	175.4	25.1
Graviton3	r7g.16xlarge	Q4_0_8_8	278.2	42.7
Graviton4	r8g.16xlarge	Q4_0_4_8	341.8	65.6

The performance differences are due to vectorization extensions, caching, clock speed, and memory bandwidth. You may see some variation at lower vCPU/thread counts and when using different instance types: general purpose (M), compute optimized (C), etc. Graviton4 also has more cores per chip, with instances available up to 192 vCPUs!

Determining which instances meet your needs depends on your application. For interactive applications, you may want low evaluation latency and a text generation speed of more than 10 Tok/s. Any of the 64 vCPU instances can easily meet the generation requirement, but you may need to consider the expected size of prompts to determine evaluation latency. Graviton2 performance shows that serverless solutions using AWS Lambda may be possible, especially for non-time critical applications.

Get Started!

As you can see, running Meta Llama models on AWS Graviton is straightforward. This is an easy way to test out models for your own applications. In many cases, Graviton may be the most cost-effective way of integrating LLMs with your application. I’ll explore this further in the coming months.

In the meantime, here are some resources to help you get started:

Deploy a Large Language Model (LLM) chatbot on Arm servers

Have fun!

AWS ML Heroes in 15: Amazon Rekognition for Wildlife Conservation

Ed Miller — Mon, 13 Nov 2023 00:54:44 +0000

On August 4 I presented an AWS Heroes in 15 session, "Amazon Rekognition for Wildlife Conservation". Here's the abstract:

In this webinar you'll learn how we used Amazon Rekognition as a bear detector to quickly implement the first stage in our Bearcam Companion machine learning pipeline. After collecting data for a season and engaging the Bearcam viewers to help label the data, we trained an improved bear detection model using Amazon Rekognition Custom Labels."
Learning Objectives:

Objective 1: Learn how we integrated Amazon Rekognition with an Amplify application.

Objective 2: Learn how we use a serverless Lambda function with Amazon Rekognition to automate bear detection.

Objective 3: Learn how we engaged bearcam viewers and Amazon Rekognition Custom Labels to improve the bear detector.

Check it out here:

You can also read a related blog here.

Amazon Rekognition Custom Labels for Bears

Ed Miller — Tue, 25 Jul 2023 01:18:24 +0000

In previous blogs about the Bearcam Companion application, I wrote about using Amazon Rekognition as a bear detector and automating it with a Lambda function. Rekognition saved me a lot of time in getting the application running. It can detect bears (and nearly 300 other objects), but it may not be a perfect fit for your application.

As you can see in the image above, sometimes Rekognition mis-labels bears as dogs, kangaroos, people, etc. I expect “bears” are not a big portion of the training data, especially in all the various poses we see on the bearcam. These mislabels need to be fixed before users can identify the bears in the application.

I don’t really want to spend so much time manually adjusting labels. For most machine learning, the next step would be to fine tune your model. You can essentially fine tune Amazon Rekognition by using Custom Labels. You can do this to make it better at detecting specific objects (like bears) or train it to detect new objects like your product or logo. It really depends on your application needs.

Custom Labels

There are 6 steps for Amazon Rekognition Custom Labels:

Create project
Create dataset
Label images
Train model
Evaluate
Use model

Step 1: Create project

You can create a Custom Labels project in the AWS console:

Step 2: Create dataset

Next, create a dataset. You can pre-split the dataset into train and test, or you can let Custom Labels do it for you.

There are several ways to import the images, in my case I imported them as a manifest file. The manifest can come from SageMaker GroundTruth or you can create your own. Since I’ve been collecting and updating image labels in Bearcam Companion, I generated my own manifest using a python script to extract data from DynamoDB Object and Image tables with links to images on S3. I uploaded the manifest file to S3 and provided the link in the form:

I had pre-split dataset: 80% for train and 20% for test. This resulted in 2486 training images and 605 test images. Many images have multiple bears, which increases the total number of labels. You can view these in the console:

Step 3: Label images

My manifest file already included labels from Bearcam Companion. If your data is not labeled, you can add labels directly in Custom Labels interface. You can also use GroundTruth for larger labeling jobs and farm them out to a broader workforce. I did find some errors in my dataset
like in the image below where the bear on the left is missing a bounding box. I adjusted labels in Custom Labels directly:

Step 4: Train model

When the dataset is ready, you can start training. This may take several hours. When completed, the model is ready to run. But first, let’s look at the model performance:

Step 5: Evaluate

Clicking on the model link will bring you to the evaluation page. Here you will find per label performance, such as F1, precision, recall and the assumed threshold used for evaluation. In my model, there’s only one label: bears, but you may have others.

View the test results to see performance on each image in test set. You can quickly filter on errors to see where your model may need improvement.

Here is a false negative example where the model missed one of the bears (correct in green; incorrect in red):

Here is a false positive example where the model labeled branches in the water as bear:

You may even find errors in the test set, like this image where the boxes in red should actually be correct.

You can fix the labels and retrain the model

Step 6: Use model

Once you are happy with the model, you can start (and stop) it in the console:

Example code for the model is provided. You can choose how many instances of your model you need. This will increase the throughput. Keep in mind, you are charged by instance-hours, that is, how many instances you have running for how many hours.

Conclusion

In this post, we learned how to create a specialized detector using Amazon Rekognition Custom Labels. We did this all in the console (other than creating our dataset). The process is simple and cost effective for many applications.

It turns out Custom Labels may not be good fit for Bearcam Companion. We only need 12 inference per hour, and I'm not sure if we can spin Custom Labels up and down frequently enough to keep the costs down. For our low usage, we prefer an on-demand model for bear detection. So one of the next projects is to train a bear detector using Amazon SageMaker.

We'll save that for another time.

For more on Amazon Rekognition and Custom Labels, check out my AWS ML Heroes in 15: Amazon Rekognition for Wildlife Conservation talk on August 4, 2023.

AWS Heroes in 15: Bear Conservation with ML, Serverless and Citizen Science

Ed Miller — Sun, 14 May 2023 18:40:02 +0000

On April 7 I presented an AWS Heroes in 15 session, "Bear Conservation with ML, Serverless and Citizen Science". In the session I covered the following

Developing a web application using AWS Amplify Studio
Using serverless Lambda functions with Amazon Rekognition to automate a web application
Leveraging Amazon Rekognition and Amazon SageMaker to engage Explore.org viewers for human-in-the-loop machine learning

Check it out here:

Interview: Using Machine Learning to Build Bear Identification Technology

Ed Miller — Sun, 14 May 2023 18:29:02 +0000

I spoke with Linda Haviv about the BearID Project at AWS re:Invent in 2022. See the full interview here:

2022: A Year in Review

Ed Miller — Wed, 04 Jan 2023 07:52:49 +0000

2022 started off no different from other years (well, at least compared with years in the COVID-era). However, things changed in early March when I was accepted into the AWS Community Builders program.

AWS Community Builder

The AWS Community Builders program offers technical resources, education, and networking opportunities to AWS technical enthusiasts and emerging thought leaders who are passionate about sharing knowledge and connecting with the technical community.

I was interested in the AWS Community Builder program as it related to my role at Arm, where I lead many of our technical engagements with Amazon (including AWS). Arm and AWS have a deep relationship around the Graviton EC2 family, which is comprised on Arm compute cores. However, I am more involved in IoT and Machine Learning, so I ended up in the Machine Learning community.

I planned to use this opportunity to share my experience in IoT and Machine Learning while expanding my knowledge of AWS services. I set up a new blog on dev.to and wrote my first post describing my goals. More on my involvement with the AWS Community Builders soon…

Multi-species BearID

In April, our second paper from the BearID Project, Multispecies facial detection for individual identification of wildlife: a case study across ursids, was published in Mammalian Biology. For this paper we collaborated with Russ Van Horn of San Diego Zoo Wildlife Alliance to train a multispecies bear face detector using images of bears living under human care. We were then able to use this multispecies detector to build a full pipeline for identifying Andean Bears.

AWS Summit San Francisco

Also in April, I attended the AWS Summit in San Francisco. Although I have been to many AWS events, this was my first as an AWS Community Builder. The AWS CB team hosted a great reception one of the evenings where I had a chance to meet my fellow community builders and the AWS team.

BearID at the Edge

In May, I completed an experimental edge deployment of BearID models trained in Azure Custom Vision using Azure Percept. The blog post on the Microsoft IoT Blog, Wildlife Monitoring and Conservation with Azure Percept, details the training and deployment.

Amazon re:MARS

In June I attended the Amazon re:MARS conference in Las Vegas. This was my second time attending this event which focuses on Machine Learning, Automation, Robotics and Space. I attended a lot of great sessions, including a workshop combining robotics and augmented reality (see the photo of the Jetbot and Magic Leap above). I managed to connect with other AWS Community Builders and a couple AWS Heroes, including Peter Hanssens:

// Detect dark theme var iframe = document.getElementById('tweet-1539455887517130753-106'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1539455887517130753&theme=dark" }

Bearcam Companion

From April through September, I was working on an AWS-based web application, the Bearcam Companion. I documented the development from start to finish in the following 8-part blog series:

Part 1: provide the background and define the M❤️P
Part 2: use Amplify Studio to define the data model
Part 3: develop the frontend with Amplify and React
Part 4: add authentication and improve the UX
Part 5: version control with GitHub and ML with Rekognition
Part 6: storage with S3
Part 7: Lamdas and Lanbda Layers
Part 8: Deployment

You can see a demo of the application here:

In August, the Bearcam Companion blogs got some ❤️ from the AWS Developers account on Twitter:

// Detect dark theme var iframe = document.getElementById('tweet-1555643305207160833-910'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1555643305207160833&theme=dark" }

IoT Builders Live

In September, I represented Arm as the guest on the first ever AWS IoT Builders live stream. I had a great time talking about IoT development with hosts Dan Gross and Nenad Ilic:

Imagine

The Edge Impulse Imagine conference was also in September. I attended the first day, which was live at the Computer History Museum in Mountain View, CA. The event featured presentation and demos of the latest innovations in edge machine learning. During the keynote, Edge Impulse announced “the ultimate AI for nature camera” which they are developing with Conservation X Labs and the Arribada Initiative. There was also a great Impact Panel on conservation and AI ethics, moderated by WILDLAB.NET’s Stephanie O’Donnell (there was even a BearID cameo in her intro slides):

// Detect dark theme var iframe = document.getElementById('tweet-1575182474346319872-707'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1575182474346319872&theme=dark" }

October Accolades

October brought more accolades for BearID, starting with an article in The Daily Beast about the project and Bearcam Companion application. The article corresponded with Fat Bear Week. I was then interviewed on the AWS Twitch channel during a DeepRacer live stream [Edit: the stream is no longer available]. The Bearcam Companion blogs were promoted by the AWS Machine Learning account on LinkedIn. Finally, I was recognized as a runner up in the AWS Amplify x Hashnode Hackathon for the Bearcam Companion as detailed in Amplify-ing Bears.

Arm DevSummit

Also in October, I published a blog titled Accelerate IoT Development with Arm Virtual Hardware on AWS, leading up to the Arm DevSummit. Note: you can still watch content on-demand!

AWS Machine Learning Hero

In one of the top highlights of the year, I was selected as an AWS Machine Learning Hero in the final cohort of 2022. This came as a surprise, as I had only been selected to the AWS Community Builders in February.

I believe my work on the Bearcam Companion, the blogs and interviews, and most importantly, the amazing support from the AWS Machine Learning channel team, is what put me on the fast track to becoming an AWS Machine Learning Hero. But what is an AWS Machine Learning Hero? According to the AWS Heroes webpage:

AWS Machine Learning Heroes are developers and academics who are proficient with deep learning frameworks and are passionate enthusiasts of emerging Amazon ML technologies. They enjoy helping developers of all machine learning proficiencies learn and apply ML, at speed and scale, through Hero blog posts, videos, and technical sessions, as well as direct engagement.

You can read about myself and my 6 fellow AWS Hero inductees on this blog post. You can also find my card (shown above) here.

AWS re:Invent

The last major activities for 2022 took place in Las Vegas at AWS re:Invent. This was my 4th re:Invent, but the first as an AWS Hero. I must say, being a Hero at re:Invent comes with some serious perks. From the Heroes dinner to special lounge access and front-row seats for a shout-out from Werner Vogels during his keynote, it was quite the experience.

I had also been selected as a PeerTalk Expert. PeerTalk is a new onsite networking program for AWS re:Invent attendees. From the AWS Events app, you can connect with other attendees and set up face to face meetings. As an expert, I was available for connections and meetings and helped with promotion. I had a few great meetings through this tool.

My main function at re:Invent was representing my employer, Arm. Our team manned a small booth providing information about Graviton and our IoT solutions. I helped set up a Software Defined Camera demo which showed the same containerized software stack consisting of a vision based ML model communicating to AWS IoT Core through Greengrass running on 2 different physical boards and Arm Virtual Hardware.

Photo by Reed Hinkel

As the culmination of my Bearcam Companion work for this year, I presented a DevChat in the Developer Lounge in the re:Invent Expo. The presentation covered background on the BearID Project, the Explore.org bearcams and the application I built using various AWS services. Sadly the presentation wasn’t recorded (perhaps I will record something similar and post it at a later date). The following day I was interviewed on the AWS Build On Live stream by the amazing Linda Haviv. This stream is still available (for now):

Migrating (Back) to Medium

I plan to utilize Medium as my primary blog moving forward. I have been posting and crossposting on Medium for years, so you can find many of my posts and follow me at bluevalhalla.medium.com. I will cross post on my dev.to blog when it is relevant.

Looking Ahead to 2023

Overall, 2022 was a big year for me, but I’m already looking forward to next year. The BearID Project team will be hard at work extending our application to work with trail camera video clips (look for a new paper in the coming year). I will continue to extend the Bearcam Companion by using the 2022 data to build models to automatically identify the bears on camera in 2023 (with corrections from the bearcam community!). At Arm I will continue to focus on IoT and Machine Learning at the edge and hope to publish more content utilizing AWS services (who knows, maybe I’ll even tie it in with BearID).

Wishing you all a happy and prosperous 2023!

Follow me at bluevalhalla.medium.com!

Accelerate IoT Development with Arm Virtual Hardware on AWS

Ed Miller — Thu, 13 Oct 2022 17:20:20 +0000

For more than a decade, the Internet of Things, or IoT, has been growing in both magnitude and complexity. The complexity derives from the requirements of an IoT device including connectivity, security, cloud service clients, over-the-air update and, increasingly, machine learning. Embedded developers building IoT devices face challenges with developing and testing these applications at scale. DevOps teams, now responsible for managing these device and service integrations at scale, face challenges with incorporating IoT devices into cloud-native flows like Continuous Integration and Continuous Delivery (CI/CD) and Infrastructure as Code (IaC).

The use of “board farms” can address some of these issues. However, they cause significant setup and maintenance costs. Testability can also be compromised due to limited access to peripherals and public cloud services. A new paradigm for IoT software development and testing is needed, and Arm Virtual Hardware is the solution.

What is Arm Virtual Hardware?

Arm Virtual Hardware (AVH) is a family of functionally accurate representations of Arm-based processors, systems, and third-party hardware. AVH enables embedded and IoT developers to build and test software using modern agile software practices without the need for hardware.

There are three main classes of AVH:

AVH for Cortex-M Processors: Software models of individual Cortex processors provided in containers, along with relevant development tools, which run in the cloud. An Amazon Machine Image (AMI) is available on AWS Marketplace and can be run on various Elastic Compute Cloud (EC2) instances. You can find it here.
AVH for Corstone: Software models of Corstone subsystems which are available in the same AMI as listed above.
AVH for 3rd Party Hardwar: Partial or complete reference platforms, including CPUs, sensors and connectivity modules. They are available as a Software as a Service (SaaS) solution from Arm. Under the hood, they leverage the Arm instruction set architecture of EC2 instances powered by Graviton. You can sign up for the private beta program here.

Use Cases

One valuable scenario for AVH is developing software before silicon is available. For example, the Corstone-310 combines the latest Cortex-M85 microcontroller with the Ethos-U55 neural processing unit. You can develop and test tinyML applications with this AVH while waiting for the first silicon to land.

Another use case for AVH is CI/CD. A great example of this is AVH integration with GitHub Actions. You can set up your repository to build and run your test suite on AVH after every merge. Here’s a video that shows how to get started:

You can find the example here. There are integrations available for other leading CI/CD solutions such as CircleCI, GitLab and Jenkins. For MLOps, have a look at this guide from Qeexo.

Arm Virtual Hardware and AWS IoT

As noted, all classes of Arm Virtual Hardware are available in the AWS cloud. However, the real value of AVH comes to light when developing with AWS software and services.

For Cortex-M based systems, you can develop software leveraging FreeRTOS and AWS IoT services. The AVH Corstone-300 is qualified for AWS IoT Core and is listed in the AWS Partner Device Catalog. For an in-depth workshop from AWS, check out Develop AWS IoT projects on Arm Virtual Hardware with FreeRTOS and CMSIS packs. This workshop explores more advanced AWS IoT concepts like Device Shadows and Device Jobs.

For Cortex-A based systems running Linux, try one of the AVH 3rd Party Hardware platforms in the private beta. You can easily get AWS IoT Greengrass running on the virtual Raspberry Pi 4. If you are working with machine learning, check out the Machine Learning Operations with AWS IoT Greengrass v2 and Amazon SageMaker Edge Manager. You can run much of this workshop on the AVH i.MX8M Arm Cortex Complex.

Call to Action

Join the upcoming AVH workshops and masterclasses at Arm DevSummit and ask experts your questions:

For further information and resources on Arm Virtual Hardware, visit (https://www.arm.com/products/development-tools/simulation/virtual-hardware).

Bearcam Companion: Demo Video

Ed Miller — Sat, 01 Oct 2022 02:58:29 +0000

I have created a demo video for the Bearcam Companion web application:

Bearcam Companion: Hosting with Amplify and GitHub

Ed Miller — Sun, 04 Sep 2022 06:29:10 +0000

The Bearcam Companion application was pretty much ready to go after my last post on Lambdas. The final step (at least for a minimum viable product) was to publish the site. Not surprisingly, I chose to use AWS Amplify Hosting.

AWS Amplify Hosting

There is an overview on how to use Amplify to set up hosting for your site in the AWS Amplify User Guide. I used the Amplify Console to get started.

Connect to GitHub

Since I already have my code in a GitHub repository (as described here), decided to use a continuous integration flow. In the setup, I selected GitHub and connected Amplify to my repository. I only have one branch and one back end (staging) so far, so I selected those. The build settings were automatically detected for both the front end and back end, so I confirmed them. After clicking Save and Deploy, the build process started.

Auto-build Process

The build process goes through 4 steps:

Provision - Sets up the build environment on a default host.
Build - Clones the repo, deploys the backend and builds the front end.
Deploy - Deploys artifacts to a managed hosting environment.
Verify - Screen shots of the application are rendered.

The first build failed because I didn't have aws-amplify in the GitHub repo. I ran

npm install aws-amplify

and committed the change. Once I pushed it to GitHub, the Amplify build process started automatically.

This time the build was successful, and I was able to view the application at the provided URL:

https://master.d7ijzylsc7qfm.amplifyapp.com/

The build history, and all the details for each step, are available in the console.

Every time I push commits to the main repo, the above process kicks off, ending with a fresh new deployment of the site.

Custom URL

The default URL is not very user friendly. I want the web application to appear as part of the BearID Project (http://bearid.org/). Specifically, I want to set it up at app.bearid.org.

To achieve this, I followed the Set up custom domains section of the Amplify User Guide. Essentially, you need to add some DNS records wherever your domain is managed. In my case, I had to add 2 CNAME records. The first record maps the subdomain (app) to the default Amplify app url above. The second record points to the AWS Certificate Manager (ACM) validation server, which enables TLS, and therefore, https. Finally I set up a subdomain forward so that all accesses to my URL go to the Amplify application.

Once I completed the setup, I had to wait a few hours for all the records to propagate and for the SLL configuration to complete:

Once the Domain Activation step above completed, I am able to view the Bearcam Companion at https://app.bearid.org/.

Conclusion

In this post I discussed hosting my application with Amplify, connecting it to my GitHub repo for automatic deployment and setting up a custom URL. Once again, Amplify made the process extremely easy.

If you find yourself watching the Brooks Falls Brown Bears cam on Explore.org and you are interested in learning who is who, sign up and log in to the Bearcam Companion. If you already know the bear of Brooks River, log in and help us label the images!

Bearcam Companion: My First Lambda

Ed Miller — Tue, 23 Aug 2022 02:34:30 +0000

I have been making progress on the Bearcam Companion web application. I have implemented most of the main React frontend components with the associated Amplify backends. However, some of the functionality which I had implemented in the UI should really be automated. This calls for one of the staples of serverless, AWS Lambda.

AWS Lambda

What is AWS Lambda? Here's what the AWS Lambda page says:

AWS Lambda is a serverless, event-driven compute service that lets you run code for virtually any type of application or backend service without provisioning or managing servers. You can trigger Lambda from over 200 AWS services and software as a service (SaaS) applications, and only pay for what you use.

Creating a Lambda with Amplify CLI

The first thing I wanted to automate was running the object detection machine learning models on every new image. In a previous post, I described how I accomplished this using Amazon Rekognition from the UI. In my most recent post, I described how I upload images to S3 and update the Images table. Now I want to use the Images table update to trigger a Lambda to run Rekognition on the image and save the object detection results to the Objects table.

I created the Lambda using the Amplify CLI to add a function:

amplify add function

There are numerous options for setting up the Lambda, so read the documentation carefully. For my needs, here are some key settings:

Function Name: bcOnImagesFindObjects
Runtime: NodeJS
Function template: CRUD function for Amazon DynamoDB, since I will be reading from the Images table and saving the Rekognition results in the Objects table
Resource access: GraphQL endpoints for Images and Objects
Trigger: DynamoDB Lambda Trigger for Images

Developing the Lambda

After creation, the function template appears in your project under:

amplify/backend/function/<function-name>/src/index.js

The template provides a basic structure from which to build. The trigger data comes in an event stream (multiple events can be batched for efficiency). The first thing I did was to parse the event records. I only care about INSERT events. From those I pull out the S3 information for the image. Here's my parseRecords() function:

function parseRecords (records) {
  var inserts = [];
  records.forEach(record => {
    if (record.eventName === "INSERT")
    {
      // get image info
      const imageS3obj = record.dynamodb.NewImage.file.M
      const insert = {
        imageID: record.dynamodb.NewImage.id.S,
        Bucket: imageS3obj.bucket.S,
        Region: imageS3obj.region.S,
        Key: "public/" + imageS3obj.key.S
      }
    }
  });
  return (inserts);
}

Next I loop through the images, calling processLabel() which sends the image to Rekognition for object detection using rekognition.detectLabels:

async function processImage(imageInfo) {
  const params = {
    Image: {
      S3Object: {
        Bucket: imageInfo.Bucket,
        Name: imageInfo.Key
      },
    },
    MinConfidence: MinimumConfidence
  }
  return await rekognition.detectLabels(params).promise();
}

For each result, I call parseDetections() to pull out the relevant bounding box information from the JSON response:

function parseDetections(detections) {
  var boxes = [];
  const labels = detections.Labels;
  labels.forEach(object => {
    object.Instances.forEach(instance => {
      var bb = instance.BoundingBox;
      const box = {
        Name: object.Name,
        Confidence: instance.Confidence,
        Width: bb.Width,
        Height: bb.Height,
        Left: bb.Left,
        Top: bb.Top
      }
      boxes.push(box);
    })
  })
  return (boxes);
}

Finally, I save each box to the Objects table by using fetch() to POST the data to the appropriate GraphQL endpoint. The main handler looks like this:

exports.handler = async function(event, context, callback) {
  try { // Parse DynamoDB Images Records
    const inserts = parseRecords(event.Records);
    for (insert of inserts) {
      // Call Rekognition on every new image
      let detections = await processImage(insert);
      const boxes = parseDetections(detections);
      for (box of boxes) {
        // Save each bounding box to Objects
        const options = getFetchOptions(box, insert.imageID);
        response = await fetch(GRAPHQL_ENDPOINT, options);
        body = await response.json();
        if (body.errors) {
          console.log("GraphQL error", body);
        } else {
          console.log("GraphQL success")
        }
      }
    }
  } catch (err) {
    callback(err.message);
  }
  return { status: "complete" };
}

Once complete, you can deploy the Lambda with amplify push. Of course it didn't work at first!

Testing the Lambda Locally

There are multiple ways to debug Lambdas. You can start testing locally, using amplify mock function. The mock capability will run the Lambda locally and feed it with event data from a JSON file. I was able to capture a DynamoDB stream event from CloudWatch, which I used as my test JSON.

One of my main problems, and not for the first time, had to do with asynchronous functions. I still don't have I still have some problems with awaits and promises, etc. I am mainly using await inside of async functions, but sometimes I find no data is coming back because I have somehow returned from the function before the data arrived.

Another problem I encountered was writing data directly to DynamoDB. This works, but it doesn't fill in all the automatic fields that Amplify had created. Instead, use the GraphQL endpoints to write through AppSync.

Testing the Lambda in the Console

One of the first problems I ran into when I did an amplify push to deploy the Lambda was a missing module. The following line was failing:

const fetch = require('node-fetch');

Not surprisingly, node-fetch is not part of the standard NodeJS runtime. Somehow I needed to include this package. I could either go to the src directory of the Lambda function and install the package there, or I could use a Lambda Layer. I chose the latter. More on that in a bit.

Once the Lambda is loading properly, you can test and modify code in the Lambda console:

You can test with pre-defined event JSON files, much as you can with amplify mock:

From this console, you can also access various monitor logs:

From the monitors logs you can jump to the CloudWatch LogStream:

Lambda Layers

Lambda Layers provide a means to share common libraries across multiple Lambdas. Here's a diagram from the Amplify docs on layers.

With amplify, you add a Lambda Layer much like you add a Lambda

amplify add function

Once I have the layer, I can add packages with the appropriate package manager, in my case, npm for NodeJS:

npm i node-fetch

When I'm done setting up the Lambda Layer, I need to update the Lambda function to have it use the layer:

amplify update function

When I am done with everything, I can deploy the updates function and new layer with amplify push.

I still had an error related to JavaScript versions. I had to downgrade node-fetch from 3.x to 2.x. Once I did, I redeployed the Lambda Layer and updated the Lambda function to use the new version. I can see the trigger and layer information in the Lambda function overview:

Conclusion

In this post I described

Creating a Lambda function triggered by a change in a DynamoDB table
Testing the Lambda function locally and in the console
Implementing a Lambda Layer for common libraries

Overall, Amplify continues to impress by making it easy to deploy backend functionality. I was able to deploy a serverless function using a Lambda written in the same language as my frontend code. I still have some challenges with asynchronous functions, but that's more to do with my own inexperience with NodeJS/JavaScript.

Next time I will write about publishing my shiny new website. Follow along here and on Twitter (bluevalhalla).