DEV Community: Dmitry Kozhedubov

Unsplash chatbot for Discord, Pt. 2: more ways to bring pictures to Discord

Dmitry Kozhedubov — Fri, 04 Feb 2022 20:11:25 +0000

In the previous post we've built a very basic Discord bot that can search Unsplash images for user specified query and output results to back a channel. In this article we'll expand on that functionality and allow Discord users to schedule a random picture for a later time.

Just AI Conversational Platform (JAICP)

The JAICP part of our bot needs two changes: implement /random endpoint of the Unsplash API and add two new states to the bot itself - one to process user's request to schedule a picture for later and another to fulfill it.

Unsplash API client

JS client is located in the script/functions.js file and here's the code that implements the /random endpoint:

var UnsplashAPI = {
    // ...

    random: function () {
        var picture = $http.get("https://api.unsplash.com/photos/random", {
            dataType: "json",
            headers: {
                "Authorization": "Client-ID //replace with your access key"
            }
        });
        return picture.data;    
    }
};

This is simpler than search because it doesn't require any request parameters and returns only one image object.

Scheduling intent

Now we need to define an intent to schedule a picture for later and a slot with time for when to post. Go to CAILA -> Intents and create a new intent, I called it RandomPicForLater. Unlike our previous intent, Search, this one will have a slot.

Slots are similar to query parameters in HTTP GET request and slot filling is a task that conversational systems perform to gather slot values from a user.

Our RandomPicForLater intent will have one slot called reminderTime and will be of type @duckling.time. Duckling is a library that extracts entities from text, and it is one of the tools used in JAICP for this purpose. Entity types in Duckling are called dimensions and there's a number of them built in, among them is Time which suits us perfectly since we need to ask users when they want us to schedule a post for and then parse a text input into a datetime object.

User's intent might be expressed as "schedule unsplash pic" or "schedule a random picture". In return, we might ask something like "When do you want me to schedule it?" to get the posting time. Fill these values in the corresponding fields ⬇️

Fulfillment

Back to the editor, add the following code to main.sc:

    ...
    state: ScheduleRandom
        intent!: /RandomPicForLater
        script:
            $session.reminderTime = $parseTree["_reminderTime"];
            var event = $pushgate.createEvent(
                $session.reminderTime.value,
                "scheduleEvent"
            );
            $session.reminderId = event.id;
            $temp.reminderTime = moment($session.reminderTime.value).locale("en").calendar();
        a: Very well, your random Unsplash picture will arrive at {{$temp.reminderTime}}.

This a new state ScheduleRandom which is triggered by RandomPicForLater intent.

What happens in the script block is interesting, because we first retrieve that reminderTime slot value and then make use of JAICP's Pushgate API that allows you to create outbound communications, as in define custom events, handle them, send outbound messages and even have bots notify your systems via webhooks. Here specifically we schedule a new scheduleEvent at user requested time and then handle it in the next state ⬇️

    state: Remind
        event!: scheduleEvent
        script:
            var picture = UnsplashAPI.random();

            $response.replies = $response.replies || [];
            var content = [];
            log("picture_desc= " + picture.urls.description);
            log("picture_url= " + picture.urls.small);
            content.push({
                "title": picture.description || "No description",
                "image": picture.urls.small,
                "url": picture.links.html,
                "btnText": "View on Unsplash"
            });

            var reply = {
                "type": "carousel",
                "text": "Your scheduled random picture",
                "content": content
            };
            $response.replies.push(reply);

Notice that Remind state is triggered not by an intent or a pattern match, but by scheduleEvent. The handler then does two things:

get a random picture from Unsplash API client
build a reply of type carousel, similar to what we did in Part 1, but with just a single item

The chatbot is now fully functional which you can verify by trying it in the test widget:

Extending Discord adapter

The only problem now is that Discord adapter to Chat API only works in request-response fashion and doesn't actively listen for messages incoming for the chatbot server. Let's fix that.

JAICP's Chat API provides an events endpoint which clients can use to fetch server-initiated events. Every time a new Discord user starts a conversation with the bot, we'll start a very minimalistic loop that will periodically try to fetch server events that occurred after the last known response ⬇️

const startPollingLoopForUser = async function (userId, channel) {
  setInterval(async () => {
    const endpoint = `https://app.jaicp.com/chatapi/${process.env.JAICP_CHAT_API_KEY}/events`;
    const eventsResponse = await axios.get(endpoint, {
      params: {
        clientId: userId,
        ts: lastJaicpMessageTimestamps[userId],
      },
    });
    eventsResponse.data.events.forEach((event) => {
      if (event.type === "botResponse") {
        const ts = Date.parse(event.event.timestamp);
        if (
          event.event.questionId !== lastQuestionIds[userId] &&
          ts > lastJaicpMessageTimestamps[userId]
        ) {
          lastJaicpMessageTimestamps[userId] = ts;
          lastQuestionIds[userId] = event.event.questionId;
          event.event.data.replies.forEach((reply) => {
            processReply(channel, reply);
          });
        }
      }
    });
  }, POLLING_INTERVAL_MS);
};

Here, we check for events of type botResponse and then perform some basic deduplication to ensure messages aren't sent to Discord more than once.

Going back to the main request-response handler, we now need to update event timestamp and question history and start a polling from above for a given user.

lastJaicpMessageTimestamps[message.author.id] = Date.parse(
        response.data.timestamp
      );
lastQuestionIds[message.author.id] = response.data.questionId;
if (!pollingLoops.hasOwnProperty(message.author.id)) {
    pollingLoops[message.author.id] = true;
    startPollingLoopForUser(message.author.id, message.channel);
}

Please note that for the purpose of this article I'm using the very basic data structures and don't persist data between adapter restarts, so this code is by no means production ready, but still give you a decent foundation to build a fully fledged adapter for virtually any chat platform.

When you run a complete example and test it in Discord, it should look something like this ⬇️

Conclusion

Expanding on Part 1, our Discord chat bot can now can send a random picture at requested time. Moreover, Discord-to-JAICP adapter now can handle both traditional request-response interchange and server initiated events.

As usual, complete source code is available on Github - adapter and chatbot (make sure to check out part-2 branch for both).

Cover photo by Volodymyr Hryshchenko on Unsplash.

Building an Unsplash chatbot for Discord

Dmitry Kozhedubov — Thu, 27 Jan 2022 14:36:28 +0000

Discord is a popular realtime chat application that has gamer-centric roots but has recently repositioned itself towards online communities in general. Unsplash has long become a de-facto source of gorgeous free to use pictures, powered by its amazing community of creators. It is only natural to marry the two and in this two posts series I'll show you how to build a chatbot that brings pictures from Unsplash to Discord chats in a couple of different ways.

Tools

In order to build the chatbot I'll obviously utilize Unsplash API which is really simple and easy to use.

For the chatbot logic I will use Just AI Conversational Platform (JAICP), which is an enterprise-grade platform that enables customers to design, develop, deploy and operate intelligent conversational AI assistants in a wide variety of text and voice channels (disclaimer: I work there). One benefit of using such platform is if you build a Discord bot and then decide to use it in Facebook Messenger, you can do that in a few clicks without changes to your code. We provide a pretty generous free tier, which means you could run a bot for your community for free or at a low fee.

Finally, because JAICP doesn't have a native Discord integration (yet) but has an API that allows for 3rd party integrations, I will build a Node.js adapter that translate messages from Discord to JAICP and vice versa.

Discord bot

The first step is to step a Discord application for your bot through the Developer Portal.

On the Applications page click New Application button in the top navigation bar. In the popup window enter something like "jaicp-discord-unsplash" and then click Create. You will be redirected to your application's page where you need to select Bot tab on the left hand side. Click Add Bot and then Yes, do it! - this will turn your app into a bot and provide you with settings, specifically bot token that we will use later on.

Remember to keep your bot token private and immediately regenerate it if you suspect it's been compromised as it gives the full access to your bot actions.

Just AI Conversational Platform (JAICP)

Next, we need to create a JAICP account. After you signed up using a method of your choosing, click Create Project button on the left hand side, choose Create from scratch option in the wizard, and finally specify a name, like "jaicp-discord-unsplash". For the purpose of this article I'll keep my bot's code in the built-in repository but alternatively you can choose to use an external Git provider like Github.

Looking around

You will not actually start from scratch but a Hello World example that will show a bit of a powerful JAICP DSL which is based on YAML and Javascript and provides many useful abstractions that allow you to build complex chatbots and voice assistants across different channels very quickly and efficiently.

The main scenario file is main.sc where we'll basically define bot's state machine (finite-state machine, FST), transitions between states in response to user's input and responses that the bot provides back to users.

Now let's create a bot that will activate on phrases like search Unsplash or find images on Unsplash, ask a user what they'd like to search for, perform an API request and finally return results.

Unsplash part

But before we develop any scenario code, let's write a simple Javascript client for Unsplash API.

var UnsplashAPI = {
    search: function (query) {
        var pictures = $http.get("https://api.unsplash.com/search/photos?page=1&per_page=3&query=" + query, {
            dataType: "json",
            query: {
                query: query
            },
            headers: {
                "Authorization": "Client-ID // replace with your Unsplash API key"
            },
            timeout: 10000
        });
        return pictures.data;
    }
}

This code defines a very simple function that performs a call to Unsplash API and returns the first 3 pictures that match a query in a JSON format. It uses JAICP's built-in $http service that allows you to integrate bots with external systems.

Intent recognition

Next, let's setup intent recognition for the bot, even if there's only one intent for now. Go to the Intents page under CAILA (Conversational AI Linguist Assistant, which is JAICP's NLU component) folder on the left hand side and click Create Intent at the top.

Name the intent search and under training phrases enter some that you would normally associate with searching for images on the internet - some examples include search unsplash, find images. Then you can test the model right there and verify that search intent is recognized correctly.

Scenario

Now that API and intents are sorted out, let's turn our attention to the bot scenario. Go back to main.sc and append the following code into it:

state: SearchUnsplash
    intent: /search
    go!: /SearchUnsplash/RequestQuery

    state: RequestQuery
        a: What should I search for?

        state: SearchPictures
            q: *
            script:
                var query = $request.query;
                var pictures = UnsplashAPI.search(query);

                $response.replies = $response.replies || [];
                var content = [];
                pictures.results.forEach(function (picture) {
                    content.push({
                        "title": picture.description,
                        "image": picture.urls.small,
                        "url": picture.links.html,
                        "btnText": "View on Unsplash"
                    });
                });

                var reply = {
                    "type": "carousel",
                    "text": "Unsplash search results for \"" + query + "\":",
                    "content": content
                };
                $response.replies.push(reply);
            go: /

There are a few things going on here. First, SearchUnsplash state of the state machine is defined, which is soft of an activation state for our bot. The bot will enter into it every time search intent is recognized from user's input. All it does is to redirect to the next (nested) state, /SearchUnsplash/RequestQuery, which will ask a user what they want for search for. Nesting here means, that once the root intent is recognized, subsequent user input will be processed (intent recognition, slot filling) in the context of that root intent.

Once a user enters a search query it will put the chatbot into SearchPictures state because it has a wildcard (*) matching - indeed, search query is an arbitrary sentence.

Finally we have our search query and can fulfill user's intent by putting a few lines of Javascript in the script block. Here we call an Unsplash API wrapper that we created earlier and transform results into a reply message of type Carousel, which is tailor-made for outputting lists of data, even though the end look-and-feel may vary based on a channel, whether it's Discord or Facebook or something else.

Setting up Chat API

The last thing we need to do in JAICP is to set up Chat API credentials for us to be able to communicate with external chat platform such as a Discord.

Go to Channels, click Connect channel under the Inbound section and select Chat API. Once you hit Save, you will be able to grab access token which is requires to the final step.

Putting it all together

One key component is still missing - as I just mentioned, JAICP currently doesn't a native Discord integration, but does have an extension point called Chat API, which allows developers to integrate conversational AI solution into a pretty much any chat platform.

I've created a really simple adapter in Node.js that listens for messages on Discord, interacts with JAICP via Chat API and then replies back to Discord. It makes use of an excellent Discord.js library and also axios for http requests.

const client = new Client({
    intents: [Intents.FLAGS.GUILDS, Intents.FLAGS.GUILD_MESSAGES],
  });
client.on("message", function (message) {
  if (message.author.bot) return;

  message.channel.sendTyping();

  axios
    .post(
      `https://app.jaicp.com/chatapi/${process.env.JAICP_CHAT_API_KEY}`,
      {
        query: message.content,
        clientId: message.author.id,
      },
      {
        headers: {
          "Content-Type": "application/json",
        },
      }
    )
    .then(function (response) {
      response.data.data.replies.forEach(async (reply) => {
        if (reply.type === "text") {
          message.channel.send(reply.text);
        } else if (reply.type === "carousel") {
          message.channel.send(reply.text);

          reply.content.forEach((item) => {
            let embed = new MessageEmbed()
              .setImage(item.image)
              .setDescription(item.title || "No description")
              .setURL(item.url);

            let actionRow = new MessageActionRow();
            let b = new MessageButton()
              .setLabel(item.btnText)
              .setStyle(5)
              .setURL(item.url);
            actionRow.addComponents(b);

            message.channel.send({ embeds: [embed], components: [actionRow] });
          });
        }
      });
    })
    .catch(function (error) {
      console.log(error);
    });
});
client.login(process.env.BOT_TOKEN);

We instantiate a websocket client that listens for Discord messages to come in, passes the text content to JAICP and transforms the replies back into Discord format. In particular, it uses embeds for images themselves and action rows/buttons to provide a link back to Unsplash - it's a nice thing to do even if Unsplash doesn't require attribution. Although JAICP can return many more than just two reply types, text and carousel are the only two we need to handle for this tutorial.

For the purpose of this article I just run it locally but you can obviously deploy it to something like Heroku.

Once you run it and try the bot in Discord it should look something like this:

You can see that I asked the bot to search for pictures of coffeeshops, and indeed I got what I wanted.

Conclusion

In this tutorial we saw how to add a conversational bot to Discord that can potentially do much more than just post pictures, even though pictures from Unsplash are typically gorgeous. This is actually a part 1 of the two piece series - in part 2 I'll show you how to set up a scheduled photo of the day post.

You can find the code both of the chatbot project and Discord adapter on Github.

Naturally, the cover photo of this post is from Unsplash Photo by Chuck Fortner.

Testing gRPC services - request collections and modern load testing

Dmitry Kozhedubov — Sat, 25 Sep 2021 20:03:36 +0000

A while ago I a post called Easy ways to (performance) test a gRPC service about bring workflows around gRPC services closer to what many people are used to with tools like cURL and Apache Bench. In this post I further explore how to streamline gRPC workflows and make them more familiar to those with REST API experience.

Collections

At this point anyone involved with APIs most likely worked with Postman. Over time Postman has become very powerful and covers a full range of API building and testing activities from mock servers to unit tests, but one thing that probably that heavily contributed to Postman's rise was the ability to create easily accessible request collections.

Of course Postman isn't the only tool in the space, and one of the wonderful alternatives is Insomnia. While Postman has become somewhat bloated over time, Insomnia pitches, among the other things, simplicity and elegance and in fact delivers on those. I've personally used for quite some time with REST APIs but only recently discovered that they've added gRPC support. Let's explore how it works.

gRPC request collections in Insomnia

The algorithm is actually pretty simple, you need to create a request and select gRPC as its type, then import a single .proto file or a folder containing one or more .proto files to accommodate imports. Imports will work across your entire collection, but for for a specific request you'll need to select a .proto file that contains the definition of a service you want to test.

From there you need to set an endpoint and request body in JSON format (Insomnia supports both unary and streaming calls) and hit Send. Here I'm using Greeter service from grpc-go repository ⬇️

It appears that gRPC support in Insomnia is indeed very young as there's no support yet for request headers or any response metadata other than status ⬇️

Good news is that Insomnia seems to be moving fast and is open source, so it's only a matter of time until this improvements are made by the core team or contributed from outside.

Modern load testing with k6

In part 1 we looked at ghz for load testing gRPC services, and now I want to cover k6, which claims to be a modern load testing tool built for developer happiness. After only a brief experience with it I can see why is that and why Grafana moved to acquire k6 earlier this year.

The boilerplate Greeter service performance test would look like this:

import grpc from 'k6/net/grpc';
import { check, sleep } from 'k6';

const client = new grpc.Client();
client.load(['../grpc-go/examples/helloworld/helloworld'], 'helloworld.proto');

export default () => {
  client.connect('localhost:50051', {
    plaintext: true
  });

  const data = { "name": "Dmitry"  };
  const response = client.invoke('helloworld.Greeter/SayHello', data);

  check(response, {
    'status is OK': (r) => r && r.status === grpc.StatusOK,
  });

  client.close();

  sleep(0.5);
};

The algorithm for the above code is really simple:

In the so-called init phase photo definitions are loaded
VU (virtual user) phase (default function) is where actual test behavior is defined
- Create gRPC channel and a stub
- Invoke the RPC
- Check if the response status is zero (OK)
- Close the connection and wait for 500ms before completing this specific iteration

To make this a little more interesting, let's add some thresholds that we expect our test run to meet. To do that, we need to export options object in the init phase ⬇️

export let options = {
  thresholds: {
    grpc_req_failed: ['rate<0.001'],   // http errors should be less than 0.1%
    grpc_req_duration: ['p(95)<10'], // 95% of requests should be below 10ms
  },
};

Here we require error rate of less than 0.1% and p95 response time to be under 10ms.
Finally, assuming the above test is saved to a file greeter.js to run it with 100 request concurrency and 10000 total requests, execute the following command:

k6 run -i 10000 -u 100 greeter.js

After a few seconds the output will look something like this ⬇️

          /\      |‾‾| /‾‾/   /‾‾/
     /\  /  \     |  |/  /   /  /
    /  \/    \    |     (   /   ‾‾\
   /          \   |  |\  \ |  (‾)  |
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: greeter.js
     output: -

  scenarios: (100.00%) 1 scenario, 100 max VUs, 10m30s max duration (incl. graceful stop):
           * default: 10000 iterations shared among 100 VUs (maxDuration: 10m0s, gracefulStop: 30s)


running (00m50.9s), 000/100 VUs, 10000 complete and 0 interrupted iterations
default ✓ [======================================] 100 VUs  00m50.9s/10m0s  10000/10000 shared iters

     ✓ status is OK

     checks...............: 100.00% ✓ 10000      ✗ 0
     data_received........: 1.4 MB  27 kB/s
     data_sent............: 2.1 MB  42 kB/s
   ✓ grpc_req_duration....: avg=1.93ms   min=172.66µs med=1.12ms  max=34.43ms  p(90)=4.33ms   p(95)=5.81ms
     iteration_duration...: avg=509.28ms min=500.64ms med=505.5ms max=621.77ms p(90)=518.03ms p(95)=525.32ms
     iterations...........: 10000   196.286414/s
     vus..................: 100     min=100      max=100
     vus_max..............: 100     min=100      max=100

Note that p95 response time threshold was met and none of the requests failed, therefore the test run was successful.

Limitations

During my brief introduction to k6 I found a couple of things that somewhat limit its usage at my work at this time:

It doesn't yet support streaming RPCs, which appears to be due to lack of event loops in k6 but is work in progress. Our RPCs are primarily bi-directional streaming however.
Creating new connection for every iteration is not best practice in gRPC and I occasionally ran into connection errors. This design decision was probably made to keep virtual users fully independent from each other, but real word gRPC applications would either reuse the same connection or maintain a relatively small pool of those.

Conclusion

As expected, gRPC tooling ecosystem is rapidly evolving, lowering the entry barrier into relatively complex technology. Both covered tools certainly have room for improvement, but I appreciated k6's superior developer experience for load testing and will look to integrate it in our teams' workflows even beyond gRPC.

Cover photo by Felix Fuchs on Unsplash

How to measure speech recognition latency

Dmitry Kozhedubov — Thu, 29 Jul 2021 12:09:02 +0000

One of the obvious applications of speech recognition is voice assistants. While there's an abundance of smart devices on the market, Amazon Echo family being probably the most famous of them, there is still plenty of cases where you might want to build your own solution. You would have to choose from a wide variety of commercial and open source offerings for speech recognition and NLP at the very least. In this article, I'll show how to measure speech recognition latency, i.e. determine how quickly it can return transcription required for your application to understand user's request.

Speech recognition for voice assistants

Typically voice assistants perform streaming speech recognition, meaning they send a continuous stream of digital audio to whatever speech-to-text backend you're using. That allows to minimize the time required to transcribe the speech, understand user's intent and finally perform the requested operation. Obviously, if you're evaluating different vendors or open source solutions for speech recognition, you'll likely want to measure their latency, because your users' experience will largely depend on how low it is.

But how do measure latency if there's a continuous stream of audio and no unambiguous response time like in the case of a REST API?

Measuring streaming speech recognition latency

Audio stream originates at some point in time 0 (zero) and sound is then being sent for processing in equal chunks, in this example 100ms each. If your recording sample rate is 16 kHz, 100ms of sound will be represented by 1600 samples. Assuming you want your voice assistant to turn on a smart bulb, command's representation in 100ms chunks of audio would look something like this (timings on the X axis are arbitrary):

Speech recognition will probably yield some interim results up until the point when the transcript matches some sort of definition of a command, be it a NLP pipeline or a simple regular expression. Aligned on the same time axis as the audio stream, results will look like on the image below. Difference between the end of command pronunciation and the moment matching transcript arrives is our latency.

Implementation

Let's put the algorithm to test and implement it in Python using Google Cloud Speech-to-text streaming API. The API client implementation itself is a slightly modified Google Cloud tutorial on streaming recognition, so I'll focus mainly on the sound handling logic and latency calculation.

We will attempt to transcribe an audio file so that the experiment would be repeatable but we'll emulate a realtime streaming. First, let's define a function that accepts the target file name and the pattern we expect transcript to match:



def match_command(
        self, filename: str, pattern: str, language_code: str
    ) -> Tuple[str, int]:
        info = sf.info(filename)
        client = StreamingClient(info.samplerate, language_code)
        for transcript in client.recognize(
            self.generate_request_stream(filename, info.samplerate)
        ):
            match = re.match(pattern, transcript)
            if match:
                self.closed = True
                return transcript, current_time_ms() - self.command_end_ts
        return None, None

Here we initialize the API client with sample rate and language code, feed audio request generator to it and then iterate recognition results. Next, let's look at the request generator itself.

Emulating realtime streaming

For starters, no user starts speaking the millisecond her microphone is turned on. So to be as close to real life as possible, we'll stream a little bit of silence at the beginning. It will be a random amount of time between 100ms (chunk size) and 2 seconds.



pre_silence = np.random.choice(
    [*range(chunk_size, samples_per_ms * 2000 + 1, chunk_size)], 1
)[0]
pre_silence_sent = 0
while pre_silence_sent <= pre_silence:
    yield np.zeros(chunk_size, dtype="int16").tobytes()
    pre_silence_sent += chunk_size
    time.sleep(0.1)

The next step is to stream the audio file itself, still emulating real time by calling time.sleep every time. I'm using an excellent library Soundfile to open the file, strip WAVE headers and read samples. I'm also capturing a timestamp of when a successive chunk is about to be sent - we will need the last timestamp to calculate latency in the end.



with sf.SoundFile(filename, mode="r") as wav:
    while wav.tell() < wav.frames:
        sound_for_recognition = wav.read(chunk_size, dtype="int16")
        self.command_end_ts = current_time_ms()

        yield sound_for_recognition.tobytes()
        time.sleep(0.1)

That would actually be a little bit inaccurate way to capture a moment when command pronunciation ended. The last chunk will most likely be shorter than 100ms worth of samples because no one alighs their speech to time scale 🙂 That means that actual speech has ended at least 100 - last_chunk_duration milliseconds ago. Let's account for that:



with sf.SoundFile(filename, mode="r") as wav:
    while wav.tell() < wav.frames:
        sound_for_recognition = wav.read(chunk_size, dtype="int16")

        if sound_for_recognition.shape[0] < chunk_size:
            last_chunk_size = sound_for_recognition.shape[0]
            sound_for_recognition = np.concatenate(
                (
                    sound_for_recognition,
                    np.zeros(
                        chunk_size - sound_for_recognition.shape[0],
                        dtype="int16",
                    ),
                )
            )
            self.command_end_ts = (
                current_time_ms()
                - (chunk_size - last_chunk_size) / samples_per_ms
            )
        else:
            self.command_end_ts = current_time_ms()

        yield sound_for_recognition.tobytes()
        time.sleep(0.1)

Now that we streamed all of our voice command recording, we could just end the request, however, that would force the speech-to-text backend to flush transcription hypotheses, which will result in artificially lower latency. To mitigate that, we'll stream chunks of silence until the moment when the matching transcript will naturally arrive. Speech-to-text backend will likely use voice activity detection (VAD) to determine that speech has ended and it's time to flush recognition results.



while not self.closed:
    yield np.zeros(chunk_size, dtype="int16").tobytes()
    time.sleep(0.1)

And that's it! Going back to match_command function, it contains a listening loop for results - as soon as matching transcript arrives, we capture that timestamp, subtract the timestamp of when command pronunciation ended, and that will be our latency.

Conclusion

Depending on your location and internet connection you may get different results, but I get latency of about 700ms for my reference files, which means that unless there's really complex and time-consuming post-processing, the hypothetical voice assistant would be able to provide response in under a second, which most users would consider to be pretty fast.

The complete code for this article is on Github. For now I've provided implementation only for Google Cloud Speech-to-text but will try to add more vendors for comparison over time.

Cover photo by Thomas Kolnowski on Unsplash

Find an audio within another audio in 10 lines of Python

Dmitry Kozhedubov — Wed, 21 Jul 2021 20:53:53 +0000

One of the fun (and sometimes handy) digital audio processing problems is to find an audio fragment within another, longer, audio recording. Turns out, a decent solution only takes about 10 lines of Python code.

What we essentially want is an offset of the short clip from the beginning of the longer recording. In order to do that, we need to measure similarity of two signals at various points of the longer signal - this is called cross-correlation and has applications beyond audio processing or digital signal processing in general. We will use well-known libraries such as NumPy that implement all the algorithms for us, we will basically need to just connect the plumbing, if you will.

Turn it up to 11

Well, actually the solution will take a bit more than 10 lines of Python, mostly because I'm generous with whitespace and want to actually build a handy CLI tool. We'll start with the main() implementation:



def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--find-offset-of', metavar='audio file', type=str, help='Find the offset of file')
    parser.add_argument('--within', metavar='audio file', type=str, help='Within file')
    parser.add_argument('--window', metavar='seconds', type=int, default=10, help='Only use first n seconds of a target audio')
    args = parser.parse_args()
    offset = find_offset(args.within, args.find_offset_of, args.window)
    print(f"Offset: {offset}s" )


if __name__ == '__main__':
    main()

Here we define the interface of our CLI tool that takes two audio files as arguments as well as the portion of target signal we want to use for calculations - up to 10 seconds is a reasonable default. Next step is to actually implement find_offset function.

First, we'll use a library called librosa to read both of our audio files, match sampling rate and convert them to raw samples in float32 format (almost) regardless of the original audio format. The last part is accomplished with ffmpeg which is used basically everywhere where AV processing is involved.



y_within, sr_within = librosa.load(within_file, sr=None)
y_find, _ = librosa.load(find_file, sr=sr_within)

Next line is where the actual magic happens - we perform cross-correlation of the parent signal and the target signal (or a window of it) using Fast Fourier Transform (FFT) method:



c = signal.correlate(y_within, y_find[:sr_within*window], mode='valid', method='fft')

What we get is an array of numbers, each of them representing similarity of the target audio to the longer recording at each point of it. If we plot it, it'll look something like this:

X axis represents indexes of parent audio samples and if it has a sampling rate of 16 kHz, every second is represented by 16000 samples. We can see a sharp peak - this is where our signals are the most similar.

The last thing we need to do is to find an index of a sample where similarity of two signals is the highest and divide it by the sampling rate - that'll be the offset (in seconds) that we want.



peak = np.argmax(c)

offset = round(peak / sr_within, 2)

Test drive

I'm a big heavy metal and Black Sabbath fan so I'll use an audio from their lesser known live DVD called Cross Purposes Live. I used the Mac tool called PullTube to download the video clip and ffmpeg to extract the audio and convert it to WAV at the 16 kHz sampling rate.
From that show I particularly like the song called Cross of Thorns and if I cut a clip of it and run the CLI tool we just programmed with it and the full recording, I'll get an offset of 3242.69 seconds which is precisely the moment the song starts in the Youtube video. Voilà!

You can find the full source code on Github.

--
Thanks to Frank Vessia @frankvex for making this cover image available freely on Unsplash 🙏
https://unsplash.com/photos/Z3lL4l49Ll4

How to (performance) test a gRPC service

Dmitry Kozhedubov — Wed, 20 Jan 2021 05:47:16 +0000

One of the common questions about gRPC services seems to be this - how do I quickly test my service, is there a Postman or something? Turns out, there is. Today I’ll talk about two tools that are indispensable for anyone who works with gRPC more or less regularly.

Refresher on gRPC

gRPC is a high performance RPC framework that uses Protocol Buffers (protobuf) for serialization. Serialized protobuf messages are not human readable, which introduces some difficulties with testing. Normally you'd generate client stub with a plugin for the language of your choice and then implement the client logic.

Can’t you just build a client?

Well, technically you can, but is this the most efficient way? You’d have to download dependencies, generate stubs, implement the client - quite a few steps when all you need do to in case of a JSON API is to fire up Postman or cURL and throw some JSON at it.

The latter was apparently the inspiration for the folks at FullStory that built a tool called, not surprisingly, grpcurl. This tool allows you to do many things you’d normally use cURL for, but with gRPC, including building requests and viewing responses in your terminal in plain JSON.

Show, not tell

Assuming you've got your Hello World server from the official Go quick start running, this is how you'd make a request to it ⬇️

grpcurl -plaintext -proto  helloworld/helloworld.proto \
-d '{ "name": "Dmitry" }' \
localhost:50051 helloworld.Greeter/SayHello

In the first line we tell grpcurl to use plain text connection as opposed to TLS and specify a .proto file that will be used to figure out request formats and method signatures. Next, we have request payload in a JSON document that follows the structure of the proto message. Finally, we specify the host, service and the RPC to call.

As you've probably guessed already, you'll see JSON in return as well. Pretty effortless too, right?

{
  "message": "Hello Dmitry"
}

A couple of things I'd personally want to be able to do with grpcurl in the future:

be able to use serialized protobuf messages instead of having to specify JSON request
save requests and create collections as you would do in Postman (looks like FullStory team is already going in this direction with grpcui project)

Load testing

Being able to test gRPC API with cURL like command is great, but what if you need to check how your service behaves under more serious load? Fortunately, there's a tool for that too. It's called ghz and it pretty much resembles some other benchmarking tools you're likely familiar with - ApacheBench (ab) and its more modern alternative hey.

This is how you'd throw a million requests at the above-mentioned Go Hello World server ⬇️

ghz -c 100 -n 1000000 --insecure \
  --proto helloworld/helloworld.proto \
  --call helloworld.Greeter.SayHello \
  -d '{"name":"Joe"}' \
  localhost:50051

Structure of the command is very similar to that of gprсurl, except we specify concurrency (100) and total number of requests (1M) as we would do with ApacheBench.

In response you'll get a nice summary with a classic average response time, requests per seconds and well as percentiles and a histogram:

Summary:
  Count:    1000000
  Total:    20.77 s
  Slowest:  18.05 ms
  Fastest:  0.15 ms
  Average:  2.00 ms
  Requests/sec: 48143.58

Response time histogram:
  0.145 [1] |
  1.935 [561604]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  3.725 [393260]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  5.516 [37030] |∎∎∎
  7.306 [5976]  |
  9.096 [1302]  |
  10.886 [557]  |
  12.677 [189]  |
  14.467 [50]   |
  16.257 [26]   |
  18.047 [5]    |

Latency distribution:
  10 % in 1.13 ms
  25 % in 1.44 ms
  50 % in 1.83 ms
  75 % in 2.34 ms
  90 % in 3.04 ms
  95 % in 3.63 ms
  99 % in 5.28 ms

Status code distribution:
  [OK]   1000000 responses

Some cool ghz features include:

ability to use serialized binary messages
JSON/TOML config files as opposed to command line arguments
support for streaming RPCs

Conclusion

Getting started with gRPC can be frustrating, especially from a tooling standpoint when coming from a REST API background. Fortunately, the ecosystem is growing very quickly and it's certainly possible to build a very efficient and reliable workflow for developing gRPC services.