DEV Community: Dominik Kundel

Five ways to use Generative AI in JavaScript

Dominik Kundel — Sun, 28 Jul 2024 19:09:11 +0000

Machine Learning and AI development is traditionally dominated by Python and because of that the ecosystem of tutorials, libraries, and examples is primarily dominated by Python. However, with the rise of the AI Engineer concept, we are seeing more full-stack web developers begin to work on AI, and with it the demand for JavaScript/Typescript compatible tooling rises. In fact in February 2024, Jared Palmer from Vercel even claimed that "The AI engineer of the future is a TypeScript engineer".

In this blog post, we'll go through five ways how you as a JavaScript developer can use different generative AI tools without brushing up on your Python skills.

Cloud APIs

If you are just getting started and especially if you are planning to use a Large Language Model (LLM) like OpenAI's GPT models or Anthropic's Claude models using their APIs directly can be an excellent start.

Interacting with the model is just one fetch call away.



fetch("https://api.openai.com/v1/chat/completions", {
 body: JSON.stringify({
    "model": "gpt-4o-mini",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "Who won the world series in 2020?"
      },
      {
        "role": "assistant",
        "content": "The Los Angeles Dodgers won the World Series in 2020."
      },
      {
        "role": "user",
        "content": "Where was it played?"
      }
 ]
  }),
 headers: {
 Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
    "Content-Type": "application/json"
  },
 method: "POST"
})

In fact, OpenAI's "Chat Completions" API has become the de-facto standard for a lot of other model providers. Providers like Groq or Together.ai provide OpenAI compatibility meaning you just need to change the URL to switch to a different provider to choose a different model.

If you are looking to use other models there are also providers like Replicate that specialize in hosting open-source models with a consistent REST API and even expose APIs to fine-tune some models on their platform.

Docker

While Cloud APIs are great to get started, sometimes you don't want to rely on a cloud-hosted provider for your use case. For example, maybe you want to explicitly run a model like Llama 3 8B directly on your local machine or you want to use an open-source library like Unstructured.io that's written in Python and use it within your JavaScript project without paying for the hosted API.

Some projects provide for that reason Docker container that will expose an HTTP API when run. For example, you can start the Unstructured API Docker container by running:



docker run -p 8000:8000 -d --rm --name unstructured-api downloads.unstructured.io/unstructured-io/unstructured-api:latest 
--port 8000 --host 0.0.0.0

Once the container is running you now have a localhost version of the Unstructured API that you can use to chunk documents to later store in your vector database.



const form = new FormData();

const buffer = // e.g. `fs.readFileSync('./fileLocation');
const fileName = 'test.txt';

form.append('file', buffer, {
 contentType: 'text/plain',
 name: 'file',
 filename: fileName,
});

const response = await fetch('http://localhost:8000/general/v0/general', { 
 method: 'POST',
 body: form,
 headers: {
 Accept: "application/json",
    "Content-Type": "multipart/form-data"
  },
})
```

Similarly, you could use [`llama.cpp` docker container](https://github.com/ggerganov/llama.cpp/blob/master/docs/docker.md) or [Ollama](https://ollama.com) to run local APIs for your LLM models such as Llama 3.

If you are working with an ML team that trained their own model or you want to host any model off [Huggingface](https://huggingface.co) and use the same Docker container approach, you can also check out [`cog` by Replicate](https://github.com/replicate/cog). It wraps Docker and is specifically designed for creating Docker containers for ML models.

All of this works great if you have a relatively small surface area of tasks you want to perform and the composability is limited. 

## JavaScript-native libraries

Now this one might be the most obvious option but the best option remains picking a library or tool that was natively written in JavaScript or TypeScript and fortunately, this ecosystem continues to grow.

Most cloud API model providers offer a JavaScript native SDK incl. [OpenAI](https://www.npmjs.com/package/openai), [Anthropic](https://npm.im/@anthropic-ai/sdk), and [Google](https://npm.im/@google/generative-ai).

Additionally, two of the most popular open-source LLM frameworks [Langchain](https://www.langchain.com) and [LlamaIndex](https://www.llamaindex.ai) provide TypeScript versions of their frameworks. Vercel also offers the [`ai` SDK](https://sdk.vercel.ai) that is built from the ground up with a stronger focus on bringing together LLMs and the front-end experiences they power. Even though the documentation is heavily focused on Vercel's own Next.js framework, the SDK also works with other frameworks.

```js
import { openai } from '@ai-sdk/openai';
import { generateText } from 'ai';

const { text } = await generateText({
 model: openai('gpt-4o'),
 prompt: 'Write a vegetarian lasagna recipe for 4 people.',
});
```

However, since most of these ultimately wrap other tools and frameworks as integrations, you are still often left with more limited functionality than their Python counterparts. For example, the Python version of Langchain has 18 different document transformer integrations while the JavaScript one has 5. 

## Native LLM APIs

Now this one is still more forward-looking. Google Chrome recently released an [experimental set of APIs](https://developer.chrome.com/docs/ai/built-in) into the [Chrome Dev](https://www.google.com/chrome/dev/?extra=devchannel) and [Chrome Canary](https://www.google.com/chrome/canary/) channels that expose access to a locally run [Gemini Nano model](https://blog.google/technology/ai/google-gemini-ai).

```js
const session = await window.ai.createTextSession();
await session.prompt("Translate the following to German: Hello how are you?")
// " Hallo, wie gehts"
```

Since the model is so small compared to state-of-the-art models including smaller ones like [GPT-4o mini](https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/) or [Llama 3.1 8B](https://ai.meta.com/blog/meta-llama-3-1/), you will likely have a harder time prompting this reliably though. With the pace of model development, this will likely change quickly though.

While this API is still experimental and only spearheaded by Chrome, the trend of local LLMs might change this quickly as more companies get interested. Mozilla, for example, recently announced that they are focused on moving "local AI" forward incl. creating a new [dedicated accelerator program](https://blog.mozilla.org/en/mozilla/mozilla-builders-accelerator/) and Apple is already using local models for their new Apple Intelligence feature.

If you want to give the `window.ai` API a shot, check out [Google's explainer repository](https://github.com/explainers-by-googlers/prompt-api) as well as the [`chrome-ai` package for Vercel's `ai` SDK](https://github.com/jeasonstudio/chrome-ai) to get started.

## Pythonia

One interesting approach to using Python tools in JavaScript is [`pythonia`](https://www.npmjs.com/package/pythonia). It's one half of the [`JSPyBridge` project](https://github.com/extremeheat/JSPyBridge) that creates an interface to call JavaScript from Python and Python from JavaScript by facilitating the interprocess communication so that you can write code in the language of your choice. 

It uses inter-process communication (IPC) and [JavaScript Proxies](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Proxy) to enable you to almost use identical code when calling a Python library in JavaScript than in Python and then actually executing it in Python.

For example, here's a code snippet taken from the [getting started guide of the Python library `haystack-ai`](https://haystack.deepset.ai/overview/quick-start):

```python
from haystack import Pipeline, PredefinedPipeline

pipeline = Pipeline.from_template(PredefinedPipeline.CHAT_WITH_WEBSITE)
result = pipeline.run({
    "fetcher": {"urls": ["https://haystack.deepset.ai/overview/quick-start"]},
    "prompt": {"query": "Which components do I need for a RAG pipeline?"}}
)
print(result["llm"]["replies"][0])
```

By using the `pythonia` npm package we can write the same equivalent code:

```js
import { python } from "pythonia";

const haystack = await python("haystack");
const { Pipeline, PredefinedPipeline } = await haystack;

const template = await PredefinedPipeline("chat_with_website");
const pipeline = await Pipeline.from_template(template);
const result = await pipeline.run({
 fetcher: { urls: ["https://haystack.deepset.ai/overview/quick-start"] },
 prompt: { query: "Which components do I need for a RAG pipeline?" },
});

console.log((await result.valueOf()).llm.replies[0]);
python.exit();
```

You might notice that this code is slightly longer and heavily uses `await`. That's because of the IPC communication. `pythonia` does a lot of optimizations behind the scenes to effectively communicate between the channels. For example, the actual data is not being sent back from Python to Node.js unless you call `valueOf()`. However, outside of that the code is very similar and is using native Python libraries.

### Performance of `pythoia`

One concern for you might be performance and while it would be slower than entirely running in Python, the actual performance might surprise you. If you want to use a Python library, like [RAGatoille](https://github.com/bclavie/RAGatouille), but the rest of your system is written in JavaScript, really the only alternative to `pythonia` is exposing the library through an HTTP API and using `fetch` to bridge the systems. 

If we run a benchmark where we use the `haystack-ai` code snippet from above and run it both using `pythonia` and expose it using [FastAPI](https://fastapi.tiangolo.com), both requests are slow because of their calls to OpenAI but `pythonia` actually slightly wins the race.

<figure>
![Benchmark screenshot where pythonia is 13% faster across 10 runs](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/24iqajuncyl2zbsh382z.png)
<figcaption>Using `pythonia` results in an average of 13% faster results but both take longer than 1.2s per run</figcaption>
</figure>


Overall while there is a performance hit of using `pythonia` over using only native Python, given the long-running nature of most generative AI calls, the overhead becomes relatively negligible especially when compared to making local HTTP requests.

## Conclusion

While more and more JavaScript developers are getting into the Generative AI space, we still have ways to go to catch up to an ecosystem that has the breadth of the Python space. Cloud APIs, running local Docker containers, and bridging projects such as `pythonia` are great options to tap into this space without moving all of your logic into Python. Ultimately it's up to us though to either grow the space of available AI JavaScript tools by contributing to existing open-source projects or even starting new ones if you want to maintain a project. In the meantime, AI tools such as [GitHub Copilot](https://github.com/features/copilot), [Cursor](https://www.cursor.com), or [Codeium](https://codeium.com) can help you with writing some Python code.

Creating your own image generation model using JavaScript

Dominik Kundel — Fri, 23 Feb 2024 08:00:00 +0000

Similar to a lot of other developers I've been experimenting in the last year a lot with different generative AI solutions. But I'm not a machine learning engineer, and I'm not even a Python developer. I'm a web developer who loves writing JavaScript & TypeScript from front-end over back-end to coffee machines.

So even just using open-source models like Stability AI's Stable Diffusion XL (SDXL) has been intimidating to me, let alone figuring out how to "fine-tune" such a model for my own needs. Just looking at the model's Hugging Face page — you can think of Hugging Face like the GitHub of sharing machine learning models — brings up all sorts of questions. What's CUDA? What does torch do? And most importantly:

how do I make it generate images of my corgi?

In this blog post we'll explore how you can create your own image generation model on top of SDXL without needing any specific hardware and just using JavaScript and APIs.

How to create your own image generation model

Creating an image generation model is extremely hard and requires a lot of data and the results can be mixed. For that reason various companies are working on their proprietary models like Adobe's Firefly or OpenAI's DALL·E 3. The problem with these models is that they are more of the "you get what you get" variety, meaning while you might be able to influence them with some example images they are largely static.

That's where open-source models come into play as base models for our project.

Choosing a base model

There are a multitude of open-source text-to-image generation models. Stability AI's Stable Diffusion XL (SDXL) is probably the most popular one among them though. It's both versatile, produces high quality output and is licensed in a way that you can use it for commercial use cases, unlike some alternative models.

The other reason why we want to use SDXL is that it not only allows for fine-tuning but it allows fine-tuning through a technique called LoRA.

What is fine-tuning? What does LoRA mean?

Prompt: "A film still of a TOK dog, big smile, dressed in a lab coat in front of a blackboard, warm color grading, 70mm, bokeh, cinematic, anamorphic, beautiful" — View generation details

The core part of a model are its weights. The weights are essentially the brain of the model and so if we want to teach a model something new we need to update those weights. This updating of the weights is done through a process called fine-tuning.

The problem is that the weights are a massive matrix though and updating all the relevant weights can be intensive work and therefore costly. That's where LoRA comes into place. LoRA stands for Low-Rank Adaptation.

With LoRA rather than changing the whole matrix of weights, an "adapter" gets created that is essentially a smaller matrix that sits on top of the original model and manipulates the relevant weights. The output of this process is also significantly smaller since it sits on top of the original weights rather than representing the whole set of weights. This means you could have more easily a multitude of fine-tuned models on top of the same base model without wasting storage.

So now we know what LoRA is but how do we apply it? A quick Google you'll find lots of Python based tutorials that are often heavy in machine learning terms and on top of that require you to have NVIDIA hardware that's compatible with NVIDIA's CUDA tooling. So what if you don't have that or don't have time to learn all the Python concepts?

Introducing Replicate

That's where Replicate comes in. They're goal is "making machine learning accessible to every software developer." There are some other offerings that provide ways for you to fine-tune your model but for this blog post we'll use Replicate.

If you don't have a Replicate account yet, start by signing up for a Replicate account.

Once you signed up, create an API key and add it as an environment variable named REPLICATE_API_TOKEN.

If you have an existing Node.js project already you can install the Replicate Node.js library using:

npm install replicate

Alternatively, we'll create a new project using the create-replicate CLI tool:

npx create-replicate sdxl-fine-tune --model stability-ai/sdxl

This will both create a new project and also run an example image generation of SDXL and your output should look similar to this:

View generation details

And you'll find the code that generated this image in sdxl-fine-tune/index.js.

If you are creating your own project you can replicate the same generation using this code:

import Replicate from 'replicate'

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN
})
const model = 'stability-ai/sdxl:39ed52f2a78e934b3ba6e2a89f5b1c712de7dfea535525255b1aa35c5565e08b'
const input = {
  width: 768,
  height: 768,
  prompt: 'An astronaut riding a rainbow unicorn, cinematic, dramatic',
  refine: 'expert_ensemble_refiner',
  scheduler: 'K_EULER',
  lora_scale: 0.6,
  num_outputs: 1,
  guidance_scale: 7.5,
  apply_watermark: false,
  high_noise_frac: 0.8,
  negative_prompt: '',
  prompt_strength: 0.8,
  num_inference_steps: 25,
}

console.log({ model, input })
console.log('Running...')
const output = await replicate.run(model, { input })
console.log('Done!', output)

Now that we were able to generate our first image using SDXL, it's time to create our own model.

Gathering your data

Before we can start the process of fine-tuning we'll need some data, or rather some images, that we can train the model on using LoRA.

For my project I used 21 images of various sizes but with my subject being very clear in the picture and captured from different angles.

Once you have collected all the images of what you want to train the model on(20-30 is a good amount to have for a pet for example), put them in a folder and ZIP up the folder so we can start our fine-tuning.

Starting your fine-tuning

Now that you have your files together let's do the fine-tuning! With Replicate there are two ways we can do this.

Using the website

The first option and maybe the easiest to try things out is using the website interface. You can upload the ZIP file you created directly there and generate your model.

Using the JavaScript

The alternative way is using Replicate's Trainings API for example through the Node.js client library.

When using the API we first need to create a new model that will act as our target for our training model. You can create both private and public models. I'll be creating a private one. Take note of your username and model name.

In order to kick off a training using the API you'll need to host your ZIP file somewhere publicly like an S3 bucket or any static file hosting.

Once you have uploaded your ZIP file it's time to set up the code. Create a new file fine-tune.js and add the following code:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

const training = await replicate.trainings.create(
  "stability-ai",
  "sdxl",
  "39ed52f2a78e934b3ba6e2a89f5b1c712de7dfea535525255b1aa35c5565e08b",
  {
    destination: "<your-username>/<your-model-name>",
    input: {
      input_images: "https://<path-to-your-training-data>",
      mask_target_prompts: "photo of a dog",
    },
  }
);

console.dir(training);

Make sure you replace <your-username>, <your-model-name> and <path-to-your-training-data> according to your respective values.

Since I'm fine-tuning the model for pictures of my dog, I added a parameter of mask_target_prompts that essentially tells the training process to focus in my input pictures on the photo of a dog. If you just train the model on a specific style of picture you can for example omit this parameter. There are a variety of parameters that you can pass into the training. Check the model for documentation on what parameters are available.

Once you are ready start your training by running:

node fine-tuning.js

The output will include a link to your training process or alternatively you can find the training run within the Trainings section of your Replicate dashboard.

On this page you'll be able to watch the logs live as the training is going on.

Testing your new model

Once your training run is complete it's time to test our new model. Update your index.js file to point to your own model.

You'll need both your username and model as well as the version of your model that you can find either on the training page or in the Versions tab of your model page.

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

const output = await replicate.run(
  "<your-username>/<your-model-name>:<your-version>",
  {
    input: {
      seed: 1234,
      width: 1024,
      height: 1024,
      prompt: "A TOK dog sitting in a throne in the style of a renaissance painting",
      refine: "expert_ensemble_refiner",
      scheduler: "K_EULER",
      lora_scale: 0.8,
      num_outputs: 1,
      guidance_scale: 7.5,
      apply_watermark: true,
      high_noise_frac: 0.95,
      negative_prompt: "",
      prompt_strength: 0.8,
      num_inference_steps: 50
    }
  }
);
console.log(output);

When looking at the code you'll notice that the prompt says:

A TOK dog sitting in a throne in the style of a renaissance painting

But what is the TOK? It's basically a special "keyword" or token that represents the concept that you fine-tuned into the model. So when we say A TOK dog we are basically saying we want a dog that looks like the dog we trained the model on (in this case my corgi). The dog in this promopt is not stricly required and the prompt could also be:

A TOK sitting in a throne in the style of a renaissance painting

But using TOK dog or even TOK corgi helps nudge the model into the right direction.

If you really don't like TOK you could pass during the training process the token_string parameter in with a different token but TOK seems to be the established way to do this.

With that we have everything ready to generate some corgi images. Let's run:

node index.js

And here we go! My corgi on a throne 🎉

Prompt: "A TOK dog sitting in a throne in the style of a renaissance painting" — View generation details

Getting ready for production

As you start using your new model you might notice that it's taking a bit of time to execute and sometimes might even get queued for a while. That's because by default your model runs on shared infrastructure until you are ready to "Deploy" your model.

When you are ready and happy with your model and want to use it in production, head over to the version of your model that you want to deploy and press the "Deploy" button.

From here you can pick how many instances and what infrastructure you'll want it to run on. Here you are primarily trading off cost, scale and execution time.

Once you deployed your model you have to update your code once more to point against your specific deployment by changing the call from replicate.run to:

let prediction = await replicate.deployments.predictions.create(
  "<your-username>",
  "<your-deployment-name>>",
  {
    input: {
      prompt: "A TOK dog sitting in a throne in the style of a renaissance painting"
    }
  }
);
prediction = await replicate.wait(prediction);
console.log(prediction.output);

And with that you are ready to create all the pictures your heart desires.

Running it locally

If you have the right CUDA compatible NVIDIA hardware on your local computer you can also use Replicate's cog tool to run the model you trained locally instead:

cog predict r8.im/<your-username>/<your-model-name>@sha256:<your-version> \
  -i 'seed=1234' \
  -i 'width=1024' \
  -i 'height=1024' \
  -i 'prompt="A TOK dog sitting in a throne in the style of a renaissance painting"' \
  -i 'refine="expert_ensemble_refiner"' \
  -i 'scheduler="K_EULER"' \
  -i 'lora_scale=0.8' \
  -i 'num_outputs=1' \
  -i 'guidance_scale=7.5' \
  -i 'apply_watermark=true' \
  -i 'high_noise_frac=0.95' \
  -i 'negative_prompt=""' \
  -i 'prompt_strength=0.8' \
  -i 'num_inference_steps=50'

Under the hood this will leverage Docker to set up the environment to run your predictions.

Conclusion

Stability AI's SDXL model is an extremely powerful open-source text-to-image generation model and there is a variety of different guides and ways to fine-tune SDXL for your own purposes. But with Replicate we were able to train the model without having to understand how LoRA works in detail, deal with having the right infrastructure or having to understand how to use Python libraries like PyTorch.

If you are hooked like me, Replicate offers a variety of other models to fine-tune directly on their platform incl. Llama 2 and Mistral models.

Prompt: "A TOK corgi with sunglasses in a pool, cinematic, anamorphic, 70mm" — View generation details

Compete For Glory In An IoT Tug-Of-War!

Dominik Kundel — Thu, 23 Jul 2020 12:13:46 +0000

NomNom the Roomba was born in July 2020. A series 585 Roomba, NomNom never goes anywhere without its external microcontroller and Twilio Super SIM. Its favorite activities are wearing fancy hats and eating confetti.

Compete for glory in an IoT tug-of-war! Direct NomNom the Roomba to vacuum up your team’s confetti first. The winning team goes down in history as NomNom’s Most Favorite CodeLanders.

If you want to learn more about how we built this, check out the GitHub repository of the project.

dkundel / remote-controlled-roomba

Code of the Break Activity at CodeLand 2020 that allowed participants to remote control a Roomba

Join us live on Twitch this Thursday!

Dominik Kundel — Thu, 07 May 2020 02:17:49 +0000

The Twilio Hackathon is over but the Twilio Office Hours on Twitch are not!

Announcing: The Twilio Hackathon Winners!

Jess Lee for The DEV Team ・ May 5 '20

#twiliohackathon #opensource #showdev

The only thing that changes is the name. Instead of joining us for Twilio Office Hours, we welcome you to join us for Remote Participants every Thursday 4pm PST on Twitch.

New Title Same Show

The title change but the hosts will remain! Phil Nash and I are excited to hang out with you between for two hours answering your questions about Twilio, coding in general and jumping into the code editor to build fun things!

What's on this week?

For this week we will look into the winners of the hackathon and also cover some of the fun hacks that were submitted but didn't make it into the top 14 submissions.

We'll also dive into using Cloudflare workers to host single page applications on Twilio Functions.

Other than that we welcome you to join us and help us decide what we talk about 😊 or maybe drop by and show us your awesome projects!

When is this happening?

The next stream is on:

Thursday May 7, 2020 at 4pm PST
Thursday May 7, 2020 at 7pm EST
Friday May 8, 2020 at 9am AEST

Those are all the same times but Phil is in Australia and therefore a day ahead 😊

We can't wait for you to join us at twitch.tv/twilio

Twilio Office Hours with Phil & Dom

Dominik Kundel — Tue, 28 Apr 2020 20:27:10 +0000

The #twiliohackathon with DEV is over in a few days and we hope your projects are going well. We can't wait to see all the awesome things you built this month.

To wrap up the hackathon we'll host another edition of our weekly office hours on Twitch! Over there Phil and I will be looking at some of the already submitted apps (no we won't judge them), answer any last questions and additionally we'll have a special guest with Thor from Stripe showing us how he combined Twilio and Stripe to create a mobile payment experience.

The Hackathon office hours will be hosted on the Twilio Twitch channel at:

19:00 EDT Thursday 30th April
16:00 PDT Thursday 30th April
09:00 AEST Friday 1st May

(They're all the same time, Phil is just a day ahead in Australia!)

We'll see you on Twitch right here: https://www.twitch.tv/twilio

Twilio Office Hours with Phil & Dom

Dominik Kundel — Thu, 23 Apr 2020 21:58:35 +0000

We hope your #twiliohackathon projects are going well but we also know that everyone hits a stumbling block at one point and we at Twilio want to make sure you have all the help you might need.

One way you can seek our help every day is by reaching out to the Twilio Hackathon Help Thread or by requesting access to the DEV connect live chat.

But if you prefer to see the faces of the people answering your questions, you can also jump onto our weekly office hours on Twitch! Over there Phil and I will be chatting hackathons, answering your questions, bounding around ideas and checking out some of the projects people are working on. And if we have some time, we'll be jumping into some live coding ourselves.

The Hackathon office hours will be hosted on the Twilio Twitch channel at:

19:00 EDT Thursday 23th April
16:00 PDT Thursday 23th April
09:00 AEST Friday 24th April

(They're all the same time, Phil is just a day ahead in Australia!)

We'll see you on Twitch right here: https://www.twitch.tv/twilio

Twilio Office Hours with Phil & Dom

Dominik Kundel — Wed, 15 Apr 2020 17:47:59 +0000

We hope your #twiliohackathon projects are going well but we also know that everyone hits a stumbling block at one point and we at Twilio want to make sure you have all the help you might need.

One way you can seek our help every day is by reaching out to the Twilio Hackathon Help Thread or by requesting access to the DEV connect live chat.

The Hackathon office hours will be hosted on the Twilio Twitch channel at:

19:00 EDT Thursday 16th April
16:00 PDT Thursday 16th April
09:00 AEST Friday 17th April

(They're all the same time, Phil is just a day ahead in Australia!)

We'll see you on Twitch right here: https://www.twitch.tv/twilio

Learn How to Contribute to Open Source

Dominik Kundel — Mon, 21 Oct 2019 16:52:41 +0000

Contributing to an open-source project can often be intimidating. Especially if it is your first contribution. At one point we've all been there. But contributing to open source can also be fun and rewarding once you get through it. We decided to try and help you with your first steps into the world of open source and hopefully make it less intimidating and more fun!

As a result we are launching today two new projects! The Open Pixel Art project and a brand new Twilio Quest mission focused on taking you step-by-step through your quest into the world of open source.

Open Pixel Art - A Collaborative Art Project

One problem doing your first contribution to open source is finding the right project to get started. While there are some great projects that welcome new contributors, we wanted to create something that makes it easy to identify what to contribute.

We often talk at Twilio about how Code is Creative. And we love pixel art. In fact at SIGNAL 2017 we created a giant wall of pixels for people to be creative and eventually hack. Inspired by our love for pixel art, we created the Open Pixel Art project with the aim to teach people how to contribute to open source while collaborating on an art project.

The project is based on a few basic principles.

Every user on GitHub can contribute exactly one pixel with a color of their choice.
Pixels can never be removed.
Pixels can be layered. (Your pixel will live on in the project even if it might be layered on the canvas. Who knows, maybe we go 3D at one point?)

To contribute a pixel to the project, you'll have to create a pull request where you add your pixel with its coordinates to the _data/pixels.json file of the project.

{
  "data": [
    {"y": 1, "x": 3, "color": "#F22F46", "username": "twilio-labs"},
+    { "y": 1, "x": 4, "color": "#FFFF00", "username": "dkundel"},
    {"y": 2, "x": 9, "color": "#F22F46", "username": "twilio"},
    ...
  ]
}

Once you create your pull request for the project on GitHub, a few bots will kick off to validate your pull request. We wanted to make the experience of creating a pull request as gratifying for you as possible. That means if you only contributed a pixel change and all your checks passed, a bot will automatically merge your pull request.

After your pull request has been merged by the Mergify bot, Netlify will automatically deploy the changes and you should be able to see your contributions a few minutes later on open-pixel-art.com.

If you want to learn more about the project and how to contribute, make sure to head over to the project on GitHub and check out the contributing guide! But if you want to be guided step-by-step through your journey into open source, don't worry! We created a special TwilioQuest mission just for you!

Embark on Your Personal Open Source Quest

The Open Pixel Art project might take away the burden of finding the right project to contribute to. But if you've never contributed to an open-source project or are new to things like git and GitHub, it might still be intimidating to you.

For that reason we created a special mission in TwilioQuest, our game aiming to teach you various Twilio and coding related concepts in a fun and engaging way. The mission will guide you through contributing to the Open Pixel Art project from signing up to GitHub, over what a fork or pull request is, all the way to getting your pull request merged.

Throughout your quest, we'll make sure to help you to have your development environment set up, validate that your fork is working, award you experience points and you can equip your TwilioQuest character!

And after your mission you'll hopefully be equipped to explore the wide world of open source and contribute to more projects.

We Can't Wait to See What You Contribute!

Whether this is your first or hundredth open-source contribution, we are excited to be part of your journey. And remember, a contribution doesn't always have to be about code, some of the most valuable contributions to a project are often not code related.

If you have any questions about the Open Pixel Art project or as you embark on your open-source quest, feel free to create a GitHub issue or reach out to me directly.

Twitter: @dkundel
Email: dkundel@twilio.com
GitHub: dkundel

Open Source Tips from Maintainers for Maintainers

Dominik Kundel — Fri, 18 Oct 2019 19:36:48 +0000

Over the last few years open source has become increasingly more popular. With initiatives such as Hacktoberfest or 24 Pull Requests there are plenty of initiatives to drive more contributions in open source. One thing that people often tend to forget though are the pains and challenges that maintainers often face, especially if your project is rapidly growing or you are new to maintaining a project. And when you search the internet, the resources for maintainers are often limited.

From Maintainers for Maintainers

This year for Hacktoberfest, Twilio decided to kick off a series of blog posts from various maintainers of large open-source projects and communities to share learnings and tips with the community and other maintainers.

Throughout October, we are publishing one maintainer tip blog post every Wednesday. You can find them all by heading over to the Twilio Blog but we wanted to give you a quick summary of each of these blog posts and we'll update this list as they are being published.

Aligning with Intention in Open Source

In this blog post Evelyn Masso is sharing one of her key learnings during her Processing Foundation fellowship working on the p5.js project. Contributing is great but how do you make sure your contributions are in line with the project’s goals?

Read Evelyn's post on the Twilio blog

10 lessons I learned from maintaining an open source community for 4 years

Five years ago Linda Peng started the open source community CodeBuddies to find remote running partners for code. In this blog post she's sharing 10 of her key lessons learned during the last years working on CodeBuddies.

Read Linda's post on the Twilio blog

Designing and Enforcing Code of Conducts

Having a Code of Conduct in your open source project or community is becoming the de-facto standard with various projects providing easy to use templates. In this blog post, Kat Marchán shares learnings from their time founding communities like lgbtq.technology and WeAllJS and maintaining various open-source projects. In this post, you’ll learn why having a clear enforcing process is key to great communities.

Read Kat's post on the Twilio blog

How to Hacktoberfest: Tips and Tricks for Maintaining your Repo and your Mental Health

Hacktoberfest can be a wonderful and hectic time depending on how many contributions your projects gets during this time. In this blog post Carly Vanderwert from the Twilio Developer Education team is sharing her top tips and tricks she learned from maintaining the SendGrid docs project.

Read Carly's post on the Twilio blog

More to Come

We hope that this series of blog posts can be helpful for other maintainers but ultimately we want to hear from you what you would like to learn more about! So feel free to share your feedback below this post or by sending me an email to dkundel@twilio.com.

Creating a virtual photo booth using Twilio, WhatsApp & Cloudinary

Dominik Kundel — Fri, 06 Sep 2019 19:04:21 +0000

If you've ever been to Berlin, chances are high that you've seen multiple photo booths in bars and on the street. Berlin and photo booths just belong together. This year for JSConf EU we wanted to build our own virtual photo booth, powered by Twilio for WhatsApp. In this post, we'll look at how we built the digital part of the photo booth using Twilio and Cloudinary.

Let's build a purely digital version of this photo booth where when a person sends in an image, we'll perform some manipulations on that picture and send it back.

Prerequisites

Before we get started, make sure you have the following:

A Twilio account (sign up for free)
The WhatsApp Sandbox Channel installed (learn how to activate your WhatsApp sandbox)
A free Cloudinary account

If you prefer to view the next steps as a video to follow along, make sure to check out this video on YouTube:

Setting up Your WhatsApp Number

When a person sends a picture to our WhatsApp number (or Sandbox for that matter), Twilio will trigger a webhook request that we'll have to receive. We could use any HTTP server for this, but in our case we'll use a Twilio Function. This allows us to host a serverless HTTP endpoint using Node.js.

Inside the Twilio Console, head to the Functions section and create a new Twilio Function with the "Hello SMS" template.

Once you created your new Function, give it a name like "WhatsApp Photobooth" and a path such as /photo. Make sure to save the changes and copy the path to your Twilio Function.

In a new tab, head over to the WhatsApp sandbox configuration screen and set the URL for "When a message comes in" to the path of your Twilio Function. Scroll all the way to the bottom to save the changes.

You can test if everything is properly set up by texting any message to your WhatsApp sandbox. The response should always be "Hello World".

Receiving the Picture

Now that we have our Twilio Function wired up, let's see how we can retrieve an image when we sent one.

When Twilio makes the HTTP request to your webhook, it will send a set of info around your message as part of that request. In Twilio Functions we can access these via the event object that is passed to our Function.

Specifically in our case the event.MediaUrl0 is of interest because it will contain the URL of our image if one has been sent with the message. We might also have event.MediaUrl1 to event.MediaUrl9 available if more images have been sent but in our case we'll focus on the first one.

For now, we'll just log the URL, modify your Twilio Function accordingly and hit save:

exports.handler = function(context, event, callback) {
    console.log(event.MediaUrl0);
    let twiml = new Twilio.twiml.MessagingResponse();
    twiml.message("Hello World");
    callback(null, twiml);
};

Afterwards, send in a picture to the sandbox and check the logs at the bottom of your Function. You should see a URL logged. Press the "copy" button to copy the URL and open it in another tab and you can see the image you've sent.

Now that we can access the sent image. Let's see how we can modify it using Cloudinary.

Manipulating the Picture

Cloudinary is a platform for image management and manipulation. It's great for our use case because we'll want to do some image manipulation on our picture. We'll also need a URL that we can give Twilio to send the resulting image. Cloudinary combines both of these in one platform.

Start by installing the cloudinary SDK as a dependency and setting your Cloudinary credentials. Go to the Configure page of your Twilio Functions and in the Environment Variables section, add your CLOUDINARY_URL from the Cloudinary Dashboard.

Afterwards, scroll to the Dependencies section and add cloudinary with the version * to install the latest version.

When you are done with both, hit save to apply the changes.

Now that we have the Cloudinary SDK all set-up, it's time to work on the image manipulation. In our case we'll want to do the following to the picture:

Change the aspect ratio to a 3x4 picture.
Adjust the height to 1600px
Apply an overlay on top. In our case, it's going to be this overlay

Before we can start with the image manipulation, you'll have to upload the overlay to your Cloudinary media library. You can either use your own in the size 1200px width by 1600px height or download one here.

Upload it to the Cloudinary Media library and give it a name of overlay-photo.

Cloudinary uses the concept of transformations to manipulate an image and comes with a wide variety of options. It also supports eager transforms. That means we can pass a set of transformations to it during image upload to tell it which transformations it should have readily cached. You can learn about the various available transforms in the Cloudinary docs.

In our situation the transformations convert to the following in JavaScript:

const cloudinaryTransforms = {
  transformation: [
        { aspect_ratio: 0.75, crop: "crop" },
        { height: 1600, crop: "scale" },
        {
          overlay: "overlay-photo",
          flags: "relative",
          height: "1.0",
          width: "1.0"
        }
  ]
};

Since the Cloudinary API is callback based, we'll use the built-inbuilt in promisify method to enable the use of async/await with this API.

During the upload, we'll also have to assign the photo a public ID which acts as a name. We'll use the ID of your sent message (MessageSid) for this one as it's a good unique identifier.

As a result of the upload, we'll be able to retrieve the URL of the eager transform. We can then use that to set the media on our response Messaging TwiML.

Update your Twilio Function to contain the following:

const cloudinary = require("cloudinary").v2;
const { promisify } = require("util");
const uploadImage = promisify(cloudinary.uploader.upload);
const cloudinaryTransforms = {
  transformation: [
        { aspect_ratio: 0.75, crop: "crop" },
        { height: 1600, crop: "scale" },
        {
          overlay: "overlay-photo",
          flags: "relative",
          height: "1.0",
          width: "1.0"
        }
  ]
};

exports.handler = async function(context, event, callback) {
  let twiml = new Twilio.twiml.MessagingResponse();

  if (!event.MediaUrl0) {
    twiml.message('Please send an image');
    callback(null, twiml);
    return;
  }

  const res = await uploadImage(event.MediaUrl0, {
        public_id: event.MessageSid,
        eager: [cloudinaryTransforms]
  });
  const url = res.eager[0].secure_url;

  twiml.message("Thanks").media(url);
  callback(null, twiml);
};

Press Save to deploy your changes.

Now that you have the changes deployed, text a picture into your WhatsApp sandbox number and you should get as a response with the transformed picture.

What's Next?

Congratulations, you've built your own digital photo booth powered by Twilio, WhatsApp, and Cloudinary. But your journey with Twilio doesn't have to be over. Why don't you check out:

And if you have any questions or want to show me what you built, feel free to contact me:

Twitter: @dkundel
Email: dkundel@twilio.com
GitHub: dkundel
dkundel.com

SMS Forwarding and Responding Using Twilio and JavaScript

Dominik Kundel — Wed, 20 Mar 2019 20:07:21 +0000

There are a variety of reasons why you sometimes don't want to give out your phone number to people but still want to receive SMS and be able to reply. Maybe you are trying to sell something on the internet, maybe you want to set up a way for people to report code of conduct violations at an event, or maybe you just don't trust the newly met person that asked for your number. In the past I showed how you can quickly set up a message forwarding with practically no coding using TwiML Bins. However, it doesn't let you respond using that number. Let's take a look at how we can change that using Twilio Functions.

Creating a Twilio Function

Before we get started, make sure you have a Twilio account. Sign up here for free: www.twilio.com/try-twilio

Once you are logged in, head to the Functions section of the Runtime part of the Twilio Console. Create a new Twilio Function and choose the "Hello SMS" template.

Change the name of the Twilio Function to something that gives you a hint on what it does. I'll name mine "My SMS Forwarder" and give it the path /forward-sms.

Right now our Function is just an auto-responder returning "Hello World". In order to actually forward SMS similar to our previous blog post, update the code the following way:

exports.handler = function(context, event, callback) {
    const MY_NUMBER = 'YOUR_NUMBER_HERE';

    let twiml = new Twilio.twiml.MessagingResponse();
    twiml.message({ to: MY_NUMBER }, `${event.From}: ${event.Body}`);
    callback(null, twiml);
};

Make sure to replace YOUR_NUMBER_HERE with the phone number you want the incoming SMS forwarded to.

In Twilio Functions we can access webhook data such as the From number or the message Body using event.. The new twiml.message line will now return the TwiML needed to forward an SMS to the specified number, containing the number of the person who sent it and the original message body.

Save the code by clicking the "Save" button. Head to the Phone Numbers section in the Twilio Console and pick the number that you want to use for SMS forwarding or buy a new one.

Once you are in the configuration screen, scroll to the bottom of the page, and under 'A message comes in' select the value "Function", and then the name of your Function. In my case that's "My SMS Forwarder".

Afterwards click "Save" and grab your phone to text any message to it. You should see a reply with your phone number and the message you sent:

If you want to see if it properly works, ask a friend or colleague to text your Twilio number and you should see their message. Alternatively get a second Twilio number and use the API Explorer's "Message Create" functionality to send an SMS to your forwarding number.

Responding to Messages

Now that we have the message forwarding solved, how do we reply to those messages? If you want to respond with your actual number, that's easy, copy the number from the SMS and write them from your phone. If you want to keep on using your masked number it's not quite that easy.

Right now if you reply to any SMS it will trigger the same SMS webhook and basically just echo to you. We'll modify this behavior by adding the following logic to this:

Check if the message came from our own number:

Came from someone else -> Forward the SMS like we previously did
Message is from us -> Parse message to receive intended recipient and forward the message

In order to find out the intended recipient, we'll establish a certain pattern that all of our responses have to follow. It will be the same way that we are currently forwarding messages:

RECIPIENT_NUMBER: MESSAGE_BODY.

Update your Twilio Function code to apply this logic:

exports.handler = function(context, event, callback) {
    const MY_NUMBER = 'YOUR_NUMBER_HERE';

    let twiml = new Twilio.twiml.MessagingResponse();
    if (event.From === MY_NUMBER) {
        const separatorPosition = event.Body.indexOf(':');
        if (separatorPosition < 1) {
            twiml.message('You need to specify a recipient number and a ":" before the message.');
        } else {
            const recipientNumber = event.Body.substr(0, separatorPosition).trim();
            const messageBody = event.Body.substr(separatorPosition + 1).trim();
            twiml.message({ to: recipientNumber }, messageBody);
        }
    } else {
        twiml.message({ to: MY_NUMBER }, `${event.From}: ${event.Body}`);    
    }
    callback(null, twiml);
};

Click save and wait until the updated Function is deployed (there will be a green box saying the deployment was successful).

Grab your phone and try to send an SMS to a friend or your other Twilio number. The SMS has to be in the format +RECEIPIENT_NUMBER: message.

The other person should receive only the text you sent. If you sent it to another Twilio number, check your messaging logs instead.

You can also try sending a text that doesn't fit the format and you should receive a reply that the message is not appropriately formatted.

Congratulations!

That's it. You are all set up with an SMS forwarding number that you can use whenever you don't want to give out your normal phone number. But this is really just the start. With Twilio Functions you have access to the npm ecosystem. So why not hook up a Google Spreadsheet or the API of your preferred contacts host to look up names instead of having to write down the phone number to respond. Or create a list of blocked phone numbers. Or maybe you have a completely different idea, I would love to hear what you come up with.

Feel free to reach out to me:

Email: dkundel@twilio.com
Twitter: @dkundel
GitHub: dkundel
Web: dkundel.com

Building an Intelligent Coffee Order System with Twilio Autopilot

Dominik Kundel — Mon, 18 Mar 2019 18:39:27 +0000

Wouldn't it be great if you could save daily a few, maybe awkward, interactions with other humans and let bots take care of it instead? However, often these bots are not too intelligent when it comes to interacting with you. In this blog post we'll learn how we can build a smarter SMS bot in just a few minutes.

In a previous blog post we talked about how we love to serve coffee at conferences and other events "Twilio-Style" by allowing attendees to order their coffee via SMS. If you haven't read the blog post, make sure to check out Serving Coffee with Twilio Programmable SMS and React. Unfortunately we had a fairly rudimentary way of handling the orders. Until now.

Let's see how we can upgrade this existing project using Twilio Autopilot to be truly intelligent.

If you would prefer to see how this works by watching a video, here is our tutorial on YouTube:

Setup

Before we get started, make sure you have your setup ready. If you've performed the steps in our previous blog post, you are all set. If you haven't, make sure to check out "Serving Coffee with Twilio Programmable SMS and React". Alternatively follow the setup instructions in the README of the Barista Lite GitHub project.

Creating your Personal Coffee Assistant

In order to get started with Twilio Autopilot, we'll have to create a new "Assistant". For this head to the Autopilot section of the Twilio Console and create a new Assistant. You can give the assistant any name. I'll name mine "CoffeeBot" for now.

Once created let's make sure we wire it up to our phone number to start testing the out-of-the-box experience. Click on "Channels" on the left side and choose "Programmable Messaging".

You should see a URL there. Copy it and go to your phone number in the Twilio Console Phone Numbers section and update the "When a message comes in" webhook to the URL you copied and make sure to press Save.

Once saved, take your phone and send any message to your number. You should see as a reply "This is your new Task".

This message comes from the default task that has been created when you created your assistant. But what is a Task?

What are Tasks, Samples, Fields, Models, ...???

When you start working with Autopilot and you haven't worked with any Natural Language Processing before, there might be lots of new terms that might be confusing for you. So let's try to clear these up a bit more.

Tasks: These are different units of work that you want to perform. Autopilot allows you to perform different "actions" inside a task. Those could be things like "say" to communicate a message to the user, "handoff" to forward the communication to a human, "redirect" to ping a webhook to decide what to do next, "collect" to gather a bunch of data, or many more.
Samples: In order to let Autopilot understand when to trigger which task we need to provide it with sample statements and map these against the existing tasks. The more samples you have per task, the more intelligently your bot will be able to route your user to the right task.
Fields: Sometimes your samples are not fully static. For example a sample like "I would like to have one espresso" has some important pieces of info that you want to extract that might vary from user to user. Say the quantity and the type of coffee are actually fields in this sample. We'll later look at how we can work with them and the different available Field Types.
Models: Every time you modify these parts of your Autopilot assistant, you'll have to rebuild a new model. You can imagine a model as a giant "smart" decision tree that is the outcome of all the info you gave your assistant. The more info you give it, the more complex and smarter your model gets.

Creating your First Task

In order to see your existing tasks and create new ones, click on the "Task Builder" section of your assistant. You'll see one existing task there already called "hello_world". It also has some labels attached to it to signalize that it has been configured as the Fallback, Initiation and OnFailure task. If you want to understand what these all mean or change one of them, click on the "Defaults" tab in the Tasks view.

Instead of creating a task from scratch, let's start by modifying this one. Click on the task and you'll see a code editor pop up that contains the following JSON:

{
  "actions": [
    {
      "say": "This is your new Task"
    }
  ]
}

Inside it you can see it shows the exact text we received earlier. Let's modify it to have a more friendly greeting:

{
  "actions": [
    {
      "say": "Hi there! Thanks for reaching out to your friendly Coffee Bot. Please let me know what you would like to order."
    },
    {
      "listen": true
    }
  ]
}

As you can see we changed the text behind the "say" key but also added a second action called "listen" to tell Autopilot that it should keep the session open. After modifying the JSON, hit save.

Next let's add some samples to trigger this task. Go to the "Natural Language Router" section and click on the expand button next to the text field. This way we can add multiple samples at once by adding them on different lines. Feel free to add whatever sample sentences you want or copy the following:

hey there
hi
what's up
hello

Select the hello_world task next to it and press the "+" button to add them:

After doing all these changes we need to create a new model. Go to the "Build Models" tab and create a new Model Build with a name like "v0.0.1".

Once your model's status changes to "Completed" your bot is ready to be tested.

For this text anything you want to your number and you should see an updated message.

Creating a Dynamic Task

Alright, now that we have a static task let's bring in some more dynamic experiences using Fields. As previously mentioned, Fields allow you to add placeholders inside your samples that will later be automatically extracted by Autopilot so that we can work with them.

Before we can work with the fields though we'll need to create a new task and something that will handle the field values later. For this let's first create a Twilio Function that will log the value of a field and return a say action.

Go to the Functions section of Twilio Runtime and create a new "Blank" function. Give it a name like "Log Field Value" and add a path like /log-value. Change the code of your Function to the following:

exports.handler = function (context, event, callback) {
  console.log(event.Field_Quantity_Value)
  const response = {
    actions: [
      {
        say: "Thank you for your order."
      }
    ]
  }
  callback(null, response);
};

Afterwards go back into your CoffeeBot Assistant and create a new task in Task Builder section. Give it a name like new_order and add the following code:

{
  "actions": [
    {
      "redirect": "https://<your_runtime>.twil.io/log-value"
    }
  ]
}

Make sure to update the URL with the link to your Twilio Function. Afterwards click save to make sure your changes are not lost.

Next we need to add Fields to this task. Click on the "Modify" link next to our task and in the dialog click the "View Fields" link. In there you'll be able to add new Fields. Create a new Field with the name Quantity. Autopilot comes with a variety of predefined data types for Fields. In our case we care about the Quantity number. So go ahead and choose "Twilio.NUMBER" as the type of this field.

Afterwards close the dialog again and switch to Samples as we'll have to add new samples for our task. Expand the input field and place the following values into it:

I would like to have {Quantity} coffee please.
{Quantity} coffee.
Could you make me {Quantity} coffee please?
Can I have {Quantity} coffee?

The {Quantity} tells Autopilot that this is a placeholder for the Field "Quantity". Associate the samples with the new_order task and add them by hitting the "+" button.

After creating the samples, go to the Build Models section again and trigger a new model build. Once that's finished, go back to your Twilio Function to see the logs in the bottom of the Function and take out your phone.

Text it something like "Ten coffee please" and check the logs of your Function. You should see it log "10".

That's because we are logging the parameter Field_Quantity_Value. Autopilot will automatically pass every captured Field as Field_<FIELD_NAME>_Value to the webhook. You can also see that it turned Ten into 10. That's because we told Autopilot that this Field is of type number. It will now handle both numbers as digits or words. Try it by texting "Can I have 15 coffee?"

Adding Custom Field Types

While the built-in Field Types serve a variety of use cases, there will be the situation where you'll want to have your own Field Type. In our CoffeeBot case this would be, for example, the different types of coffee that we serve.

To create a new Custom Field Type go into the Natural Language Router part of your bot and click on the Manage Fields tab. Press the "Create your first Field Type" button and give it a name like coffee_type. Once created click on the name in the list of Field Types and press the Plus button to add new examples. This is where you'll want to add possible values for this type. In our case this would be any valid Coffee type. You can enter one value per line for ease of use.

espresso
americano
latte
mocha
tea
coffee
flat white

Note that the values that you provide here will not be the only valid ones. Autopilot will learn and try to match other words to this field type as well.

After you created them, let's add CoffeeType as a valid Field for our new_order task. Go back into the View Fields part of your new_order Task and add a new Field of name CoffeeType and data type coffee_type.

Now before we build our new model, we need to update our samples to use the new field. Go ahead and delete the old samples for the new_order and instead create new samples using the following value:

I would like to have {Quantity} {CoffeeType} please.
{Quantity} {CoffeeType}.
{CoffeeType}
Could you make me {Quantity} {CoffeeType} please?
Can I have {Quantity} {CoffeeType}?

Once saved, go into Model Builds and build a new Model by incrementing the version in your build name.

Your model will now be able to understand both quantity and different coffee types.

Connecting our Ordering System

To actually be able to create new orders we now need to change the action that we are triggering to our original Twilio Function URL and update our Twilio Function.

Go to your "Barista Create Order" Twilio Function from the previous blog post and update it accordingly:

exports.handler = function (context, event, callback) {
  const ORDER_LIST = 'orders';
  const SERVICE_SID = context.SYNC_SERVICE_SID || 'enter Sync Service Sid';

  const orderType = event.Field_CoffeeType_Value;
  const orderQuantity = event.Field_Quantity_Value || 1;
  const order = `${orderQuantity}x ${orderType}`;

  const orderData = {
    order: order,
    phoneNumber: event.UserIdentifier,
    status: 'open'
  };

  // Create a sync list item for the order
  const twilioClient = context.getTwilioClient();
  twilioClient.sync
    .services(SERVICE_SID)
    .syncLists(ORDER_LIST)
    .syncListItems.create({ data: orderData })
    .then(x => {
      callback(null, {
        actions: [
          {
            say: `Thank you for ordering ${order}`
          }
        ]
      });
    })
    .catch(err => callback(err));
};

This will read the right fields as well as fall back to a quantity of one if no quantity could be detected. Additionally we are using the event.UserIdentifier here to get the phone number since this will be the phone number for SMS and voice bots. Note that this won't work for Alexa or Google Home for example.

Save your Function changes and copy the URL to your Function. Go back into your Autopilot Task and modify the new_order task and update the URL for the redirect to your Function URL.

Make sure you rebuild your model one more time to catch all changes.

If you haven't yet opened your terminal do that now and start your React interface from the previous blog post by running:

npm start

You should see in your browser the following screen:

Let's test if everything is working by texting something like "Can I have two latte please?". Note that we are spelling it lowercase and are altering our sentence from the example sentences.

You should get a confirmation of your order of "2x Latte" and it should appear in the browser UI. Click "Finish Order" or "Cancel Order" and you should receive the respective confirmation via SMS.

What's Next?

This is just the beginning of what you can do with Twilio Autopilot. If you want to learn more, I suggest you check out the Collect action that allows you to do form-filling with ease. You could use it for example to ask the customer for additional information like their name or if they want to have soy milk added to their coffee. Or check out how you can use the Handoff functionality to connect either to a Twilio Flex agent or directly to the phone of your barista for any questions the bot can't answer.

And if you want to import/export your bot to another account for example, you should check out the autopilot-cli that will help you with building your bot.

If you have any questions or if you want to show me what cool thing you built with Autopilot or just in general, feel free to reach out to me: