DEV Community: Arafat Tehsin

ParkingGPT - Your AI Parking Copilot

Arafat Tehsin — Sat, 13 Jan 2024 23:41:33 +0000

About 10 months (March, 2023) ago, I got a mention in a tweet from one of my good friends and our local community leader, Simon Waight. It was about solving a complex parking problem with Computer Vision. I bookmarked this interesting challenge and made it as a part of my want-to-do list. After a month, I picked this up and started to think as how I would like to solve this problem. I faced several problems which I have mentioned later in this blog post but finally managed to create this small yet useful cross-platform app (or copilot or gpt or whatever you want to call it) for everyone living in Australia. While there's a big need of improvements and additional features, it is live and opensource. You may consider it as a new year gift from me.

Learning from my failures

As I mentioned earlier, it was just a tweet which triggered this idea. However, there were many hiccups in its development. Someone rightly said that success never comes easy. As the world is going through a rapid change of how we develop, use and interact with technology, picking up these challenges can be daunting, time-consuming and hard. Initially, as soon as I picked it up, I opted for the latest and finest technology of that time for Computer Vision. I chose Project Florence by Microsoft which is a large foundational model for Vision. The reason why I chose this is because I wanted to use the dense captions capability of this model.

However, after trying out different techniques, I failed as it couldn't do the proper OCR of the parking signs. Later on, I was suggested by one of the product team members to use Form Recognizer (which is now called Azure AI Document Intelligence). I tried with that too but couldn't get any success either. The problem I was getting with Form Recognizer was that the text was not correct all the time. It used to be messed up sometimes. To combat this problem, I decided to use a combo approach which means that I used Form Recognizer for OCR (with a bit of gibberish results) and then corrected them with GPT-3. This worked for a few signs but failed miserably for others. Basically, the success rate was 50%. I wanted something robust that would solve this problem for us without any training of a custom dataset.

Finally, the time came when I witnessed the launch of OpenAI's GPT-4 with Vision followed by the support for the same in Azure OpenAI Service. It felt like my wish came true with time, but then I had to wait because I had so many other things planned esp. the year end tasks and the family holidays. 🎈

Considering all of it, I kept a dedicated slot between my personal time to work on this as I really wanted to finish it off before my work begins in January 2024. So, here I am, I think I achieved what I planned. I created a ParkingGPT for all of my fellow Aussies to decide when they can park or not. It's very initial stage and currently I have just tested with the parking signs within New South Wales. However, if you are based in any other state then please feel free to give it a shot and let me know your feedback. 🙏

ParkingGPT

ParkingGPT enables you to decide whether you want to park or not, all using the power of Vision AI & LLM. This uses the newly released GPT-4 Turbo with Vision. With ParkingGPT, you can easily identify parking signs and make informed decisions about where to park. Whether you’re a driver looking for a convenient parking spot or a parking enforcement officer looking to enforce parking regulations, ParkingGPT is your AI parking copilot.

This app just has two main features. It either allows you to capture a photo of a parking sign from your camera or it lets you pick it up from the photo gallery.

Technical Implementation

It is quite straight forward as you can see from the repo. I have tried my best to follow the best practices for .NET MAUI and how you usually develop apps. There may be certain negative cases which I may have missed, such as writing an effective prompt of ignoring non-parking sign images but overall, it works well.

This app takes an image (either from the camera or from the gallery) and then converts it into base64 encoded string. This is then sent to OpenAI API's for processing and returned with the response. This app is good to work with both Azure OpenAI and OpenAI. All you need is the API endpoint key for it to run. The API endpoint key stays in your app's specific storage which means that you do not have to enter it again and again.

youtube.com

Some common FAQs

Is it published on a Google Play Store?

No, but you can request me for the .apk package by just commenting on this post and sideload it to your Android for now. If there's much interest, I can think of publishing it to the store.

Is it published on an Apple App Store?

No. I do not own a Macbook which restricts me from creating a package (or .ipa) for this app. The developer program subscription is too expensive for a hobbyist but I will see.

Does it work seamlessly with multiple parking signs?

I have tried 2 to 3 ones and it works fine. However, I am not sure how it would react to the more than 2 or 3 signs. I am still testing it.

Is it smart enough to make decision based upon the no parking signs and parking signs?

Not for now. I will see this for future.

If you are a developer then the code is there for you as opensource to see how I have done it. As a part of my mission to empower every developer with necessary applied AI skills, I have done this in my personal time. Please do not forget to star ⭐ the repo if you like and share it among your friends so others can get an idea on how to create small but useful utilities like ParkingGPT.

Until next time.

Best practices for building collaborative UX with Human-AI partnership

Arafat Tehsin — Thu, 30 Nov 2023 07:04:58 +0000

In today’s digital age, it is not easy for any business to work without the application of artificial intelligence (AI). Especially for those in retail or consumer market, AI has now become the integral aspect of their growth. If it is rightly (read: responsibly) used then it ensures the improvement in overall customer experience and helps drive sales. I recently completed a course by John Maeda, VP of Design and AI at Microsoft where he discusses a few essential principles for building remarkable user experience (UX) with the AI copilot stack in real-world applications. These principles focus on creating a collaborative user experience rather than a pretty UX.

Collaborative UX

Collaborative UX is a concept that recognises the importance of collaboration between humans and AI in the design and development process. When building AI copilots, it is crucial to understand that the user is the pilot and the AI is the copilot. The success of the copilot depends on the skills and expertise of the pilot (human). Collaborative UX aims to establish a strong partnership between humans and AI, ensuring that the AI copilot complement the human pilot rather than replacing them.

There are several collaborative UX patterns that help establish and maintain the pilot-copilot relationship. I have summarised a few of them below:

AI Notice Pattern

One of the collaborative UX design patterns that serves as a reminder to users that the information they are interacting with is generated by AI, is referred as AI Notice Pattern. This pattern encourages users to critically assess the AI’s output rather than blindly trusting it. It acknowledges the non-deterministic nature of AI, emphasising that while AI can offer valuable insights, it is not foolproof and may occasionally produce inaccurate or incomplete results. The presence of AI in the user experience is highlighted which fosters a collaborative relationship between humans and AI and empowers users to make informed decisions.

The implementation of AI Notice Pattern involves the usage of visual cues or textual indicators that clearly state the information is AI-generated. This can be done through notifications, disclaimers or prompts which encourage users to question or verify the AI’s output. The pattern aligns with the broader concept of collaborative UX stressing the importance of human-AI collaboration and active user involvement in the decision-making process. By incorporating this pattern into AI-powered applications, designers can create more responsible and user-centric experiences.

[caption id="attachment_31814" align="aligncenter" width="970"] AI Notice Pattern[/caption]

Suggested Prompts Pattern

Another collaborative UX pattern designed to improve the relationship between users and AI copilots is called the Suggested Prompts Pattern. It gives users the suggested prompts or options with the power of copilot's capabilities. The copilot can suggest actions or make recommendations based on their knowledge and abilities by offering specific prompts. These prompts can then suggest the alternative approaches to problem solving or suggest specific actions to achieve a desired outcome.

Implementation of the Suggested Prompts pattern is comprised of both business workflow understanding as well as designing user interfaces that integrate the suggested prompts. This can be done through visual prompts, buttons or interactive elements that present users with the available options.

It is important to have the prompts to be concise, relevant and contextually aligned with the user’s goal. This collaborative UX pattern can help improve the efficiency and effectiveness of user interactions with the copilot and encourage users to explore different possibilities and consider alternative options. This way, users can benefit from the copilot’s capabilities through suggested prompts while maintaining control and decision-making authority to themselves.

If you want to learn about UX for Copilots in depth then you do not want to miss this session of Microsoft Build, 2023. All the images of this post are also taken from this session.

Feedback Opportunities Pattern

The last UX pattern to discuss today is around the feedback. Feedback about the performance of AI and how it helps out user’s goal and its output. It gives an opportunity to the users to contribute to the improvement of overall copilot’s capabilities therefore known as Feedback Opportunities Pattern. These contributions can vary from sharing their insights to corrections to suggestions and many more. Thus, a feedback loop is created, facilitating continuous learning and improvement. Feedback mechanisms such as a review form, issue reporting or a rating system can be integrated within the user interface of the AI-powered application.

The continuous feedback is valuable for the copilots strengths and weaknesses. It is essential for the underlying technology and algorithms which can be used to address any shortcomings. Feedback Opportunities not only empower users to contribute to the copilot’s development but also foster a sense of ownership and engagement. By incorporating user feedback into the development process, user experience designers (or developers in my opinion) can enhance the copilot’s capabilities, resulting in a more refined and effective user experience.

The Emotional Iron Triangle

In addition to these patterns, it is also important that developers consider the "Iron Triangle" of quality, speed, and cost when building AI-first apps. This triangle can also be topped up with Emotion as a fourth dimension, making it a tetrahedron of factors to manage. As developers create AI-powered applications, they need to be mindful of managing users' emotions, ensuring they don't over-humanise the AI and maintaining a sense of responsibility in this era of AI.

Embrace the essence of humanity but beware of over-humanising. For in the quest to understand others, we must not lose sight of their unique individuality

The copilot stack with its non-deterministic nature, bring new challenges and opportunities. Developers while also considering user emotions, must learn to balance quality with speed and cost. One has to become a new kind of a developer who understands both the technical aspects of AI and the emotional aspects of UX design.

In a nutshell, creating successful AI-powered applications depends on creating a collaborative user experience that emphasises the partnership between humans and AI. In addition to implementing the above discussed design patterns, developers can also consider Iron Triangle with the user's emotions to create a strong pilot-copilot relationship. With this approach, AI-first apps become more engaging, effective and responsible driving growth in a rapidly changing digital landscape.

Until next time.

This blog post was originally published at the Microsoft Learn Community blog.

Build Sentiment Classifier, Translator and a Flight Tracker with the Generative AI

Arafat Tehsin — Mon, 24 Jul 2023 00:07:49 +0000

Last week I received an email from the Microsoft Reactor team informing me that I had 360 RSVPs on the day of my talk. While I was super excited, I was a bit nervous because I wanted to ensure that I deliver the content responsibly. If you value your audience's time, ensure your content is engaging, informative and well-organized. During my talk, I answered around 25 questions therefore I thought that I shouldn't just deliver the session and upload my slide deck and demos. Instead, I should add some more value to it and make it more inclusive for most of the developers. As a result, I created not just one, but two Polyglot Notebooks (fully interoperable with Jupyter Notebooks) and decided to write about it.

At first, I considered developing simple console applications for the demos. Although these console apps would have sufficed, the getting started guide would have been somewhat cluttered, requiring multiple apps. Therefore, I decided to use a Polyglot Notebook, allowing us to consolidate everything into a single file.

The biggest benefit of Polyglot Notebooks is its interoperability with Jupyter Notebooks & multi-language .NET support.

Introducing Generative AI Notebooks

As a long time .NET developer, I always try my best to bring tech like Reinforcement Learning, Computer Vision and Generative AI to the developer community using the similar languages and frameworks. Despite my personal interest in .NET, I also did not want to ignore the fact that many developers are also (and rightly so) using Python within this space. Therefore, I decided to create two notebooks, one for C# with .NET Interactive and the other one for Python. Both of the notebooks are identical in terms of the topics covered, so it depends on you which one you'd like to use.

I've created somewhat boilerplate code for a few use-cases, which is the topic of our blog post. Let's dive into each of them.

Sentiment Classifier

Potential Use-cases: Customer Feedback Analysis, Client Satisfaction Index, Call Centre Real-time Insights

Traditionally, machine learning models are always trained on a bigger dataset for better accuracy along with the right choice of algorithm with some hyperparameters. For example, if you want to build an engine that'd classify a sentiment out of a sentence, you may want to train it on at least 1000 records with a mixture of negative and positive sentiments. This way, the model may achieve as 70-80% of accuracy. I've done it in the past with my SentimentAnalyzer package which has already been used 24k times by developers. However, this was a bit of a task for me to train and then integrate it within my applications.

Large language models have changed this all. Now, with the magic of few shot learning, you can easily build a sentiment classifier. This can easily be done using the Completion API with GPT-3.

Below is the syntax on how I did it. You can try it using the notebook.

Remember to initialise a .NET or Python kernel before you run the subsequent steps of the notebook(s)

Summarisation and Translation

Potential Use-cases: Summary of the long emails, translating it from non-English to English and vice versa

Imagine you work in a hospitality services business, and you request your customers to fill a survey at the end of their service. Your customers are both native English speakers and those who speak other languages. You often receive around 100-200 words of feedback and due to the volume, it gets tricky for you to go through each of them. This is where you want to summarise the feedback with their appropriate sentiments and also get the proper translation of it.

Below example is a glimpse of how you'd like to do it.

Flight Tracker

Potential Use-cases: Flight tracking, postage tracking, food delivery

In 2020, I wrote two posts about how to build a Flight Tracker using Language Understanding Intelligent Service (aka LUIS), which is now upgraded as Language Understanding in the Azure Language Studio. The reason why I brought the similar use-case here is to show you how effective it is to use the power of generative AI to build a much better version of it with less code. 💯

Before we get there, imagine how you'd build it yourself? For this, you may need a source, destination or a flight number. If you do not have either of the locations or just the flight number, then you may not be able to track down the flight. This is at least required information for any flight for a simplest hypothetical use-case.

I have been working with Conversational AI since 2017 and pushed some useful solutions to production since then, I can confirm and you will also appreciate that this is not an easy problem. Your customer (who is an actual user of this bot) is a human. Humans are not naturally accustomed to following a pattern or sequence of the questions, all the time. That's why there's a good chance that humans may either provide a flight number, a destination or a source.

In order to solve this problem with Generative AI (Azure OpenAI Service), we just have to provide a good prompt, few shot examples and that's it. You will be able to see that it works quite well.

Protip: You may get a response returned in JSON or any format you'd like for your use-case.

Below is the syntax on how I did it. You can try it using the notebook.

DALL-E (Bonus 🎁)

If you look at the last section of the notebook, you will see that I have also created a boilerplate code for you to work with the DALL-E model which is still in preview. This creates an image out of the natural language utterance. Try it out yourself.

The power of generative AI has revolutionised the way we approach problem-solving in various domains. By leveraging these Generative AI notebooks and the advancements in AI technology such as Azure OpenAI Service (GPT-3, GPT-4 and so on), developers can now create more efficient and effective solutions with less code and effort. This blog post highlights the potential of generative AI in use-cases such as sentiment classification, summarization and translation, and flight tracking. As AI continues to evolve, we can expect to see even more innovative applications and streamlined development processes, making it easier for developers to deliver cutting-edge solutions to their users. Embracing these advancements will not only enhance our ability to solve complex problems but also foster a more inclusive and accessible environment for developers across various programming languages and platforms.

Until next time.

Break out of your comfort zone, and start learning new skills

Arafat Tehsin — Wed, 12 Jul 2023 05:08:12 +0000

When we’re kids, we explore our interests and abilities, adopting what we like and leaving what we don’t. In this way, we create our comfort zone, where we find less stress, increased efficiency, and improved confidence. But given today’s fast-paced world and ever-growing opportunities, staying in our comfort zone can limit our perspectives and lead to stagnation, ultimately resulting in a lack of personal and professional growth.

It’s important to be willing to stretch and learn new skills, especially in the tech community. There are countless examples of people who stepped outside of their comfort zone and achieved great success, including Microsoft Chief Executive Officer (CEO) Satya Nadella. I remember reading his book, Hit Refresh, in which he discussed leading different businesses, and breaking out of his comfort zone before becoming Microsoft CEO.

The question I want to address in this post is, How do you break out of your comfort zone? It can be challenging—but also rewarding. It’s definitely a process, so I recommend taking things one step at a time. In my career as a Solution Architect, I’ve discovered some useful tips on going beyond my comfort zone, and I’d like to share some of them with you.

Identify your fears

It’s absolutely natural to be afraid of something unknown to you. So, start by asking yourself: Are you scared of failure, rejection, or the unknown—and why?

Once you’ve identified your fears and the reasons for them, you can work on overcoming those fears. This process may involve small steps outside of your comfort zone, and these don’t need to be perfect on the first try. For example, let’s say you’re a technology expert and you want to teach others about it, but you’re afraid of addressing a large audience. A great way to counter this is to start by teaching a small group of people. You can also join a Microsoft Learning Room and co-present there with other experts.

Set clear goals

Clear goals can provide you with a solid sense of purpose, motivating you to go outside of your comfort zone and take on new challenges. Although there are different frameworks for setting up goals, I recommend that you follow SMART—that is, your goals should be:

Specific. Clearly define what you want to achieve.
Measurable. Set milestones to track your progress and to evaluate when you have achieved them. You can even divide individual goals into smaller milestones, making them less daunting.
Achievable. Be sure your goals are realistic and attainable.
Relevant. Build your goals on professional or personal aspirations to help maintain your motivation.
Time bound. Set deadlines by which to achieve these goals.

For example, if you’re a Microsoft Power Platform expert who wants to get out of your comfort zone and stretch your abilities, your SMART goal could be, I want to learn AI and find out how to add value to my business applications with it. A logical next step would be to start with a learning path on Microsoft Learn, such as Bring AI to your business with AI Builder or Improve business performance with AI Builder.

Embrace failure

Everyone experiences failure at one time or another, and this is to be expected when you venture out of your comfort zone. This is because you’re taking risks and trying out new things. Instead of avoiding failure, embrace it as part of the learning process. In fact, failure can give you valuable feedback and help you identify areas for improvement. Use it as a steppingstone toward great success, as it helps you build resilience and develop a growth mindset.

Surround yourself with supportive people

As a human, it’s important to have a solid support system around you—family, friends, mentors, colleagues, and others. Moving out of your comfort zone can result in many challenges, but people in your support circle can help keep you motivated, boost your confidence, and cheer you on as you overcome obstacles.

As you set your goals, look for people who can provide you with constructive feedback. Seek advice from those with experience in the area you’re pursuing. One place to connect with peers and experts who are exploring technology is the Microsoft Learn Community, where you can engage with others and get guidance on your journey.

Break out of your comfort zone

Saying goodbye to your comfort zone can be scary, but it’s important to do so—for your personal and professional growth. By identifying your fears, setting clear goals, embracing failure, and surrounding yourself with supportive people, you can set yourself up to achieve great success. Ready to explore new tech skills? Microsoft Learn provides you with an opportunity to learn new things every day. Invest your time trying out new technologies, building new skills, and breaking free from your comfort zone. To find out more about how we can learn together, join me in my Learning Room, AI for Everyone.

This blog was originally published by Arafat Tehsin at the Microsoft Tech Community.

Unlock the power of OpenAI's GPT with No-code AI Builder

Arafat Tehsin — Tue, 04 Jul 2023 12:00:52 +0000

As you know, Microsoft's AI Builder now supports prebuilt Azure OpenAI models, allowing users to easily integrate powerful AI capabilities into their applications and processes without the need for any coding or data science expertise. In this post, we will explore the benefits and potential use cases of leveraging these prebuilt models with AI Builder.

Microsoft's AI Builder simplifies the integration of Azure OpenAI models by providing a user-friendly interface for connecting to the OpenAI API. This enables users to access advanced AI capabilities, such as GPT and incorporate them into their applications and workflows with minimal effort. By utilizing prebuilt Azure OpenAI models, users can add powerful AI-driven features to their applications and processes, such as natural language processing, text generation, and more. This can lead to improved efficiency, better decision-making, and more personalized user experiences.

Helpful conversational experiences with Power Virtual Agents

These latest capabilities of Azure OpenAI in AI Builder are native to Power Automate and Power Apps. This basically means that anything in Power Platform can utilise the capabilities of the GPT models without going to external sources. As a result of that, I created a Power Virutal Agents (PVA) bot using the unified authoring canvas which calls a Power Automate flow to get the answers from Azure OpenAI 's GPT model.

Recently, I got the access to the prebuilt Azure Open AI models in AI Builder. You may not see this in your environment but you can always sign up for a limited preview.

In this post, we choose a use-case which normally does not get covered yet it is interesting and demanding. Hence, we will work on a Handyman Services business and build a bot around their tasks and related information. I've named this bot as a HandyBot and conveniently divided this blog post into two different use-cases. First one is fairly simple while other one is a bit governed. Both are useful in my opinion.

Power Automate with AI Builder's action

Let's create a simple flow that accepts a text input from PVA and then uses the latest AI Builder's action of creating a text with GPT on Azure OpenAI Service.

Coming back to our bot, we utilise the power of Copilot (Preview) and create a topic and plug it with the flow we created above. We know that our topic is simple but what it helps us in is, writing trigger phrases, which were on point. Below is the video example of it (I try not to use heavy .gifs in my blog posts)

Check out the video here.

In case you want to test it. You just have to click on the Test button and write your query (just like I did).

That's how simple it is to integrate GPT without writing a single line of code and connecting to the external sources using Custom Connectors.

One could now argue the use-case, exploitation or vulnerability of this as there's no check and balance in creation and may point at its practicality. I still think it is practical but I will take their point for the sake of an argument.

Approvals for the social media posts in Teams

Let's suppose you run a marketing department of a business called Handyman Services. Just like any other business, this also needs some social presence, we need to build a Power Virtual Agents bot that'd take a topic from the internal staff in Microsoft Teams, generate a text with GPT and send it for approval before showing it to the internal staff. Once the department lead approves it then the internal staff gets a private message from a Power Automate bot with the generated text. Yes, it could have been improved but you get my point, right?

So first of all, we'd build a flow again. This time, it is going to be a bit different (not a lot) from the last one.

You may notice one obvious hack here that we are using a fixed (or hardcoded) Recipient. Ideally, we should have got the email address from the Power Virtual Agents bot and we should use that email here. Despite the fact that my video walkthrough below shows you that I am passing an email address to the flow, there's a known bug currently in the authentication part which would restrict us to perform that operation. That is the reason we may go with the hardcoded email address.

Now as the flow is ready, let's see what how we build the Power Virtual Agents bot. We utilise the Copilot feature again as it helps us setup some trigger phrases quickly then we have to just tweak a few things to make it work. You can check the video below for everything I did in PVA.

Check out the video here

Once the bot is created, we may have to go to Settings ➡️ Security, choose Authentication and select Only for Teams. Now publish the bot.

Internal staff request for social media post

You have an internal staff named Adele. She works for a marketing department. Now we have this capability to use GPT model for post generation, Adele asks this question to the HandyBot.

Once the post is requested. Your marketing lead receives an approval request.

After the post is approved, Power Automate bot sends a message back to Adele with the GPT generated content by AI Builder's Power Automate action.

The integration of Power Virtual Agents with AI Builder (using Azure OpenAI's GPT) marks a significant step forward in the world of chatbots and AI-driven customer support. By leveraging the advanced natural language processing capabilities of GPT, businesses can create chatbots that provide more accurate, context-aware, and human-like responses to user queries. This leads to improved customer satisfaction, reduced burden on support staff, and more efficient operations. As we continue to explore the potential of AI and machine learning, it's exciting to see how innovative solutions like this will shape the future of customer support and user experience.

Until next time.

Create personalized experiences for your apps, bots and websites with Azure Personalizer - Part 3

Arafat Tehsin — Tue, 10 Jan 2023 12:51:14 +0000

Personalization in customer service is the practice of tailoring interactions with customers to their individual needs and preferences. This can be achieved through the use of customer data and technology such as machine learning and artificial intelligence.

One way that personalization can be used in customer service is to provide tailored responses to customer inquiries. For example, a customer service agent can use information about a customer's previous interactions and purchases to provide more personalized and relevant answers to their questions. This can help to improve the customer's overall experience and make them feel valued and respected.

Another way that personalization can be used in customer service is to provide personalized recommendations and suggestions to customers. For example, a customer service agent can use information about a customer's interests and preferences to recommend products or services that they might be interested in. This can help to increase customer satisfaction and loyalty, as well as boost sales and revenue.

Personalization can also be used to improve the efficiency of customer service operations. By using customer data and technology, customer service agents can more easily and quickly access the information they need to resolve customer issues. This can help to reduce response times and improve overall customer satisfaction.

So far, I've already written two parts on Azure Personalizer. The first one talks about building your own library. The second one talks about building a simulator for your learning loop. This will help you train your model ahead of production deployment. This is going to be my last post on this topic. In this post, we're going to see how we make use of Azure Personalizer into a chat bot.

What do we need?

As we're going to use the Azure Personalizer into a chat bot, we have to choose the right platform for that. Whilst there are many options in the market, I decided to go with the Microsoft Power Platform. There were two reasons to choose this platform over others.

One is that Power Platform allows you to easily create a chat bot using Power Virtual Agents (PVA) and integrate the Azure Personalizer APIs using Power Automate. Both without any code, even with the custom connector 🤘. The second reason is the recent update we witnessed about the Power Virtual Agents around unified authoring canvas. I was quite excited about it so thought to use it in one of our bots.

It totally depends upon you as how you want to secure Power Platform environment. It can be your sandbox, your trial instance or also a Developer Plan. I have got a M365 Developer Plan and I will be using that. My complete solution (Custom Connector + Power Virtual Agents) is available on my usual GitHub repo for the AI for Every Developer series. You can play with it.

The YUM Bot

The overall idea is to create a Power Virtual Agents bot for the customers to suggest the best dessert based upon the occasion and variety. This best action will be delivered by the Personalizer API via Power Automate and displayed it as an Adaptive Card back to the customer / user.

Custom Connector

In our previous posts, we worked on the library well as the simulator based on .NET and how you can easily use Personalizer SDK to train your learning loop. This time, we're going to use a Personalizer endpoint to interact with the service. Initially, I looked at the Power Automate to find out if there was any connector available for Azure Personalizer. Unfortunately, I did not find one, so I thought to create a custom connector with the name of Azure Personalizer (Community). If that's something new to you then you may want to check out some of our awesome Microsoft Business Applications MVPs on YouTube. They've explained this topic quite well.

Azure Personalizer (Community) connector has got two actions, Rank and Reward. It needs an API key of your Personalizer resource in the Azure to create the valid connection.

Rank Action

This action expects the same inputs as it receives from our .NET library i.e. Context Features, Actions, EventId and more. I've kept it as generalised as possible and that's why, all you have to do is to switch the layout of the input to entire array and paste your JSON.

Our first post about Azure Personalizer has all the details about Rank and Reward API. Both of the actions are dependent upon the same API.

It sends back the eventId and the best action in the form of rewardActionId

Reward Action

This action expects the eventId and reward value. The reward can be anything between 0 and 1. For the simplicity, we will be sending 0 (bad) or 1 (good).

As the independent publisher certification process requires a good time and effort for the submission therefore, it's not my focus for now but will surely visit later.

Chat bot

Now, let's talk about the Power Virtual Agents. We are going to use the public preview of the latest unified authoring canvas. They have got a bunch of new features and I've already covered some of them earlier. The idea over here is to enquire about two sets of information which is required by the Rank Action, Occasion and Variety. For the simplicity of this use case, we'll create a simple bot and switch off all the Custom Topics except Greeting. It is not mandatory but just saying!

Entities

After this, let's create two entities as a ClosedList. Start with Occasion and add synonyms for smart matching.

Do the same for Variety as well.

Topic (Book Order)

Just add at least 5 phrases to train the BookOrder topic. The more phrases you add, the better but 5 are also OK. As an example, you can add I want to place an order.

Before asking a question about the Occasion and Variety, let's add a variable and set its value as a table of items. You can do this by using the latest capabilities of PowerFx. This will be used later in our bot.

Later on, just add two questions with multiple choice options to ask about the Occasion and Variety and store each of them into variables. Once we get these two values, we call our custom connector we created above, specifically Rank Action.

After its processing, we get a response in eventId and rewardActionId. Our concern is more about rewardActionId as this is an action (choice of a dessert) suggested by the Personalizer. So, what we do is, we create another variable and we apply LookUp on the table of items we created above to get the information about the specific rewardActionId.

The reason we do this LookUp and set it up in a separate variable is because we want to populate the Adaptive Card dynamically right within the Power Virtual Agents canvas not in Power Automate.

Customer or user can choose yes or no depending upon their preferred choice. This submission is also handled by Adaptive Cards (I know it's not common yet). However, the data property inside Action.Submit can only contain string to work properly for now. Otherwise, it won't send an event back to PVA and you may wonder what you did wrong. 🤷‍♂️

So that's a wrap of a bot. Now if you run it, it should behave in a similar way as this one below. Bonus takeaway for you is the proactive slot filling in Power Virtual Agents. This means that if you provide an entity's name (birthday, meal or cake) already in your question, it will skip that entity's question to speed up the process, automatically. Super cool, isn't it?

In conclusion, personalization in customer service has many potential applications. It can be used to improve customer experiences, increase customer satisfaction, and improve the efficiency of customer service operations. By using customer data and technologies like Azure Personalizer, you can provide more personalised and relevant interactions for your customers, which can help to build stronger and more lasting relationships.

Until next time.

P.S: The introductory 3 paragraphs and the conclusion except for the highlighted text of this blog post is purely written by ChatGPT. I could have made it better or added more of my thoughts but word on the street compelled me to keep the way it is. 🙂

Create personalized experiences for your apps, bots and websites with Azure Personalizer - Part 2

Arafat Tehsin — Sun, 11 Dec 2022 08:27:25 +0000

In today's digital age, it is not easy for any business to work without personalization. Especially for those in the retail or consumer market, personalization has now become the fundamental aspect of their branding. It improves the customer’s experience and helps drive sales. Personalization increases the customer loyalty when you suggest them something of their interests. Let's start building the model for the similar experience for our customers.

In our last post, we talked about the basic concepts of Azure Personalizer and how you can build your own library (or use mine). In this post, we will try to build a simulator which is going to train our Personalizer model so that we can then use it in our actual apps, bots or the websites.

What do I need?

If you're not using the existing library which I have shared or you just want to copy it, then you may need a .NET SDK for Personalizer.

In my Personalizer Library, I am using the preview version of the .NET SDK

Simulator

Let's say you own an app that lets people buy paintings of their choice. The app contains thousands of paintings of all sorts, ranging from pets to landscape to streets and so on. You integrate Personalizer in this app so that your customers can start seeing the relevant paintings. You may have to do this by fetching the right category of paintings. Every time a painting is shown to the user, the Rank and Reward calls are made to the Personalizer to learn and improve.

However, the above discussed process is quite tedious. It can take weeks if not months before Personalizer starts suggesting the right action. Therefore, we need a way to train the Personalizer faster and the constructive way to achieve this, is by creating the Simulator.

Just to highlight that actual actions are already defined in the PersonalizeLibrary code.

Personalized Desserts

Let's understand the basic goal or objective we're trying to achieve in this simulator. Our business problem resolves around the Personalized desserts. For example, if someone wants to go to a birthday party but that person wants to take cupcakes for their friends then it'd be vanilla cupcakes. However, if someone wants to go just for the catch-up or a meal and wants to take the cupcakes with them then it'd be nutella cupcakes.

This means that based upon the different occasions and varieties, a specific dessert will be suggested as the top action. Below is a simple depiction of how it would look like:

You can argue that it was too simple to put and there are not many variations in the suggestions. I agree and the reason behind it was to make it simple so that people who are starting it for the first time can understand well.

The simulator will randomly pick-up the context features and send it to the Rank API. Once it receives a reward, our code will determine that actual award against a particular occasion and variety. If the determined award is same as the award returned by the Rank API, then it is sent as 1 into the Reward API with an event ID.

In our previous post, we have discussed briefly that your reward can be anything between 0 and 1.

UserSimulator.cs

In this class, we will just set-up the possible combinations as defined above. These will be stored as a part of dictionary where we will have a format of ocassion_variety as a key, whereas possible action ID as a value. This is how it is going to look like:

It also contains the code to get the response for the simulated action. Instead of explaining the basic code line by line, here you go with all the class:

Simulator.cs

In this class, we're going to setup our occassion and variety values so that we can simulate the context randomly. Once we get the randomized combination of the context, we then pass it to the Rank API. The response of the rank API is then stored in custom class called PersonalizerResponse which is a part of PersonalizerLibrary. If the returned response matches the response of the simulated context, then a reward of 1 is sent back to Reward API.

Note that SimulateEvents function is called from outside (such as Program.cs) so that it can run for N number of times to train the model.

In our previous post, we've kept our Model Update Frequency in Azure setting as 5 minutes

Once you start running the simulator, you may see that it will give you return random responses. This goes back to our topic of explore vs. exploitation I discussed in my previous post.

In the start, you may experience very random responses, but you will notice the improvement in your model later on.

I noticed that whenever you run around 500-1000 requests, you may need to wait a little more (not just 5 minutes as defined above) to see the improvement of the trained model. This will also show you an increase in the reward count at the later stages. So, that's one of the ways to create a Simulator so that when you can test your application's behaviors. This is clearly not a way to push it for production and then later train on that data as it will have a bunch of false results all the time, unless you're using an Apprentice Mode. It is a topic of some other day 🙂

Personalised experiences add more value to the overall experience than adding half-baked features into your solutions. In our next post, we'll be looking at the applications of the Personaliser. Primarily, our focus will be on the integration of Personalizer with either a web app, mobile or a chat bot.

Until next time.

Create personalized experiences for your apps, bots and websites with Azure Personalizer - Part 1

Arafat Tehsin — Mon, 31 Oct 2022 13:00:00 +0000

In the last couple of years, Artificial Intelligence (AI) has shown tremendous growth opportunities. It is rapidly improving and changing the way we interact, shop and work. AI has become so ubiquitous in our daily lives that it has eliminated a lot of laborious tasks. Many organisations ranging from large enterprises to small businesses are using AI to create productive and smart solutions. These solutions do not just boost the human creativity but also optimise the overall costs. However, the results of these AI based solutions are tailored for masses rather than a specific set of people. They clearly lack the relevance or personalisation aspect. As a result of that, in this post (and the next ones), we'll learn how you can easily create personalised experiences for your web or mobile apps, bots and much more just through Azure.

Why do we need this?

In the machine learning (or broadly AI based solutions) world, the baseline approach has always been the Supervised Learning. It does have great successes in many use-cases but that's not super useful everywhere.

Let's take an example of yourself. You use a shopping app to buy some clothes. This app shows the lists of the products which are usually featured or on sale. Now imagine, that the list which you see does not show any item of your interest. This means that it shows some different colors, size or style. You may click on it or maybe you don't. Therefore, almost every time you may want to search or filter out your favorites from the list and it is not going to be an easy process always. The supervised learning here fails because it's too much of a work to identify what's right for a specific set of contexts (or people). You may also want to avoid self-fulfilling prophecy problem.

So how would you feel as opposed to the list of your choice of clothing, based upon your own interests? I am sure you'd want that. Everyone wants that. So, the question is, can we really optimise for the best outcome, that too with real-time learning and amongst a given set of choices? The answer is yes with Reinforcement Learning.

Reinforcement Learning is an approach to machine learning that learns behaviors by getting feedback from its use.

Personalisation is the hallmark of a customer centric focus. It makes customers feel valued and helps businesses provide the CX which are narrowed or specific to their needs and wants.

Based upon some of the latest trends and our above example, 80% of the customers are those who are more likely to buy from those who provide personalised experience. In addition to this 72% of the customers are more interested in engaging with personalised messaging by the bots. Lastly, just to re-iterate its importance, 50% of the Australians think that they would switch to a competitor after one bad experience which means it is very critical for businesses to start thinking about it.

Azure Personalizer

Whilst there can be many ways, we as a matter of our interest, will use Azure Personalizer to build the personalised experiences for the customers. Personalizer uses reinforcement learning to make some smart decisions for your applications. We will also see how you can boost conversion and engagement as well as add real-time relevance to your product recommendations. Unlike the usual recommendation engines which offer a few options from the large dataset, Personalizer gives the single best outcome for the user every time. gives customer relevant experiences that improve over the time based upon their behavior. It improves the click-through rates and remain on of changing trends.

Personalizer can get you the best outcomes for variety of business use-cases. Some of them are:

News article based upon the interests of the reader
Food order on the basis of occasions and dietary requirements
Perfect gift for the Mother's Day

Personalizer service is not as simple as Vision or Language, it requires some good understanding of the concepts. I will try my best to ensure the clarity both from the writing and covering the technical bits from the .NET SDK. Please feel free to continue reading even if you're not going to use Personalizer with C# .NET - It is definitely going to help you, promise!

In rest of the post, I've taken a personalized dessert example, just to explain you the concepts in an easier way

Learning Loop

Each Personalizer resource you create from your portal is considered as one Learning Loop. This is essentially for a single personalization use-case. If you have got more than one in your app then you may want to create more than one. In our case, we will just create one. It's as simple as below.

Rank

Personalizer learns over the time what (actions) you should show your users in a given situation (or a context). Rank is basically an answer / single best decision a personalizer determines and responds back. It's also considered as one of the two primary APIs of Personalizer. Every time your app has to make a decision, the Rank API is called. This API expects actions (with an ID), features associated with each action and features that describe the current context. Each API call is known as an event and distinguished with an event ID (you can assume it as guid). Personalizer service then returns the ID of the best action which is determined by its model. Now I have spoken about quite a few new terminologies here, let's learn about each of them.

string eventId = Guid.NewGuid().ToString();
var request = new PersonalizerRankOptions(actionList, currentContextFeatures, null, eventId);
PersonalizerRankResult result = client.Rank(request);

Actions

Actions are the unique set of items which are meant to be chosen or selected. They can be anything from products to promotions to news articles to ad placements on the website and much more. Actions have got the list of features which describe them. Basically, they are the metadata about the content. For example, if the action is a dessert, then its features are comprised of sweet level, cuisine and category.

...
new PersonalizerRankableAction(
    id: "banoffee_pie",
    features:
    new List<object>() { new { variety = "dessert", sweetlevel = "medium" }, new { nutritionalLevel = 5 } }
),
new PersonalizerRankableAction(
    id: "mango_delight",
    features:
    new List<object>() { new { variety = "shake", sweetlevel = "medium" }, new { nutritionLevel = 2 }, new { drink = true } }
),
new PersonalizerRankableAction(
    id: "tresleches_cake",
    features:
    new List<object>() { new { variety = "cake", sweetlevel = "high" }, new { nutritionLevel = 3, cuisine = "mexican" } }
),
...

There's a limit of 50 actions but no limit on the number of features one action can have. It is also advised that features should be meaningful and effective for the specific action. It is also not necessary that all the actions should have the similar set of features.

Context

Context or Context Features are quite interchangeable (hence confusing). It is basically the metadata about the context in which the content (e.g., a personalised product) is presented. Context can literally be anything such as user, demographic information, weather, time, historical data, device and so on.

The recommendation is to use the name for context features that are UTF-8 based and start with the different letters, such as variety and occasion, device. Having similar namespaces (or names) of the context features can result in collisions in indexes used for machine learning.

The Rank API expects four parameters;

List of all the Actions
Current context features
Features to exclude
Event ID (guid)

{
    "contextFeatures": [
        {
          "occasion": "birthday",
          "variety": "cake",
          "weather": "sunny"
        }
        ...
    ]
}

Both for action and context, it is always recommended to have more dense features rather than a sparse feature. For example, there can be many bakers' confectioneries which can be classified as cakes and cupcakes. That's considered as a dense feature. Contrary to this, same features can also have a Name attribute which will be quite unique among every feature. This is a considered as a sparse feature.

Once the request is sent to the Rank API, then the API decides to either exploit or explore to return the top action (RewardActionId). Now what's that? Let's learn about them!

Exploit

When Personalizer service chooses an action which matches the highest probability in the rank. This is called exploitation. This happens (100 - exploration) % of the time.

Explore

When Personalizer service chooses an action which does not match the highest probability in the rank. The percentage of exploring the outcome is based upon the configuration in Azure. Currently, personalizer uses epsilon greedy algorithm to explore. This is different than the behavior in some A/B frameworks that lock a treatment on specific user IDs.

It is always recommended that exploration percentage is set between 10% to 20% and:

Not to have 0% as it will not use any benefits of Personalizer as in this case, it will lead to model stagnation, drift and eventually lower the performance.
Not to have 100% will negate the benefits of learning from user behavior as in this case, it will always try to get a random response rather than a top action item.

In order to understand the top action item (RewardActionId) returned by the Rank API, it is important for us to learn about the Rewards.

Reward

Reward is a score that your business logic determines as a measure of success, just to validate the decision taken by the Rank API (or broadly Personalizer) is good (leaning towards 1) or bad (leaning towards 0). For example, if the Rank API suggests a news article or promotional offer and user clicks on it, then the score can be considered as high as 1 because the user chooses it.

Reward is considered as second of the two primary APIs for Personalizer. Reward API expects two parameters

Event ID (the same guid you send to Rank API). This is an optional parameter, if it's not sent then Personalizer will generate automatically generate one.
Reward (between 0 or 1, depending upon the business logic)

The Reward API needs to be called within the timeframe specified on the above screenshot of configuration (Reward Wait Time) after the Rank API call is made for a given event. If Reward API is not called within the due time, then a Default score will be allocated instead, if it is defined in Azure. This data is then used to train the model (according to the learning policy as shown below) by recording the features and reward scores of each rank call.

public void SubmitReward(string eventId, float reward)
{
    client.Reward(eventId, reward);
}

Model

Every learning loop (defined above) has a model that learns about the user behavior. It gets training data from the combination of the results you send to Rank and Reward APIs. This training happens as per the model update frequency defined in the configuration. In my case, as I was training it using the simulator which I built to test my model. I set it up to update every 5 minutes as I was testing. Ideally, it can be hours or even days based upon your needs.

Learning Policy

Personalizer trains a model on the basis of its learning behavior defined in the configuration. This enables the functionality of how the machine learning works. There are two basic modes and you can see that I've chosen the first one as my use-case of suggestion the best dessert was OK to use it.

It takes at least 100 to 10,000 calls for Personalizer to start behaving well so you need to be mindful of tests

Online Mode

This is the default mode (or behavior) of a personalizer where it uses machine learning to build the model that predicts the top action (or content) for use-case. This is improved time by time when the Rank and Reward calls are made.

Apprentice Mode

Just like in real life where an apprentice learns from their experience, it's the same in case of Personalizer. This is different than an online mode where it tries to predict the random values right from the start impacting the application with the cold start problem. Rank calls, in this case, always returns the application's default actions i.e., the action your app would have taken without using the Personalizer. Once the Personalizer is trained well, the learning behavior can then be switched to the online mode for better results.

If you want to create your own library, then you may need to download the preview SDK for Personalizer from NuGet package manager

I have now created a complete library which can be used easily in your projects despite my dessert use-case. It's been one of the longest posts I have written so far on this topic, but I truly think that is really valuable for you to understand the basic concepts before you get your hands with it. Personalizer is a complex service and require some patience too (as it does not train in seconds or even minutes). In my next post I will try to discuss the simulation aspect of it as well as application of this service. If you've got any questions, please write down in the comments section.

Until next time.

Object Detection Simplified with AI Builder

Arafat Tehsin — Tue, 08 Feb 2022 12:13:05 +0000

Few months back, I posted a small video clip on my LinkedIn to show how I used ML.NET (w/ Azure), WinML & UWP to build a custom real-time object detection application. I received a few messages of folks on how I actually did that therefore, I decided to submit my talk & spoke to NDC Sydney, 2021 to actually showcase the working of it.

After its implementation, one of former colleagues asked if I could achieve this using Power Platform i.e. AI Builder (because of its object detection capability) and Canvas Apps. I responded in negative because I remember this limitation when I wrote about using AI Builder with Camera Stream in one of my previous posts.

The limitation was that we could not pass the image (or encoded base 64) to the AI Model in Canvas Apps because you had just 4-5 built-in controls which were actually not super helpful. Even if you had a requirement to pass an image (via camera stream) to your object detection model, you had to send it to Power Automate and process its response. So whenever I got a chance, I spoke about this limitation with the awesome team at Microsoft and to my fortune (with many others), we didn't have to wait long enough. 🙂

On November 8, at Microsoft Ignite 2021, Lobe & AI Builder team announced fantastic features to add intelligence to your Power Apps. Now, not only you can easily add your model as a data source to your Canvas Apps but also you can use them with PowerFx. This became the reason for me to re-visit my problem which I couldn't solve because of the discussed limitation.

The purpose of this post is to show how you can achieve near real-time object detection using AI Builder models with Power Apps. In case you're interested in building the AI Builder Object Detection model, you may want to check out this post where I have discussed its process step by step.

Object detection using Camera Stream

The capability of using the AI into your apps with Power Fx allows you to actually access your AI models by any control, be it a Gallery, DataTable and so on. Therefore, we will try to build (or mimic) the same experience as I did for my .NET app. Not because I couldn't think of any other idea but because I want to show the limitless capabilities of Power Platform. Yes I know one works offline while other does not but I think you got my point, didn't you?

So I already have got a template app ready which I will be using to add necessary controls to achieve above. 👆

Camera

Let's start by adding a Camera, set its StreamRate and OnStream properties with a variable that is going to capture the image from the stream. The property we call is captureTrigger

Video Walkthrough

Timer

Add the Timer control to the screen and hide it. Set its Duration to 2000 and OnTimerEnd should have Set(captureTrigger,100);which would set the trigger's value to 100 after every 2 seconds to take a picture from the camera.

Video Walkthrough

Note: Do not forget to add toggle Repeat and Auto start. I missed this in the video.

Gallery

Now, at the same position (X and Y axes), add the Vertical Gallery and align its height and width similar to the Camera. The reason for this is that Gallery will be used to create Bounding Boxes on top of the Camera control. Change the TemplateSize to 0 otherwise it is going stack the items vertically one by one (if there are more than one object identified in an image). Delete all the items inside your Gallery as we're going to add some new controls here.

Lastly, add go to Add Data and add your AI Model, set the Items property of a Gallery to 'Your Model Name'.Predict(YourCameraControl.Stream).results - that's the beauty of using your AI Model along with PowerFx in any control.

Video Walkthrough

Add a Container with Text Label and Rectangle inside your gallery control.

Container

Change its X value to ThisItem.boundingBox.left * YourCameraControl.Width
Change its Y value to ThisItem.boundingBox.top * YourCameraControl.Height
Change its Width value to ThisItem.boundingBox.width * YourCameraControl.Width
Change its Height value to ThisItem.boundingBox.height * YourCameraControl.Height

Text Label

Choose your choice of Color with the Border and set its value to 1
Change its PaddingTop to 1
Change its Height to something of your own choice, I kept it as 28
Keep the Width as Parent.Width
Also, please ensure you set X and Y both to 0
Set its Text property to ThisItem.tagName

Rectangle

Choose your choice of BorderColor with the Border and set its value to 8
Keep the Height as Parent.Height
Keep the Width as Parent.Width

Note: Please do not set the Color property, just set the color of the Border instead.

You may notice a bit of a lag it is because unlike .NET (or any other framework which runs offline), this is being run online and it takes a second or two to get the response back.

Bounding Boxes (Bonus Control)

There's a bonus custom control for you to use instead of the above discussed Gallery and that is Bounding Boxes.

Motivation

Few days back I had a chat with Jake Cohen and Bill Barnes from AI Builder & Lobe Team at Microsoft on the above topic and we discussed a couple of options to achieve that. During our discussion, Bill went ahead and instead of Gallery approach, he came up with a smart idea of using a HtmlText instead. He then converted HtmlText into a PowerApps Component called Bounding Boxes. As the name suggests, this custom component exposes a property called Boxes which actually takes a Bounding Box of an AI Model as an argument and shows the bounding box on the screen. The best part of this control is its weight which is not heavy as the Gallery and simple for anyone to use.

Video Walkthrough

The good news is, Bill has just shared his repo with me so you can have this (and several other controls) added to your Canvas Apps easily. Here's how you add the Bounding Boxes to your application. 👆

Add the Component to your app and then set the Boxes property of the component to 'Your Model Name'.Predict(StreamCamera.Stream).results so after every 2 seconds, it will start showing the bounding box as they are processed by the model.

As I discussed above that the purpose of this post is to show you the limitless capabilities of AI Builder. It is now up to your business needs where you really want to use this, for example, as a business owner of a store, you'd like to keep a periodic (~10 minutes) check on people wearing masks within a location, counting the number of allowed people in a café and so on. In case you get stuck with whatever I have shown you here - please feel free to comment down and I'll be happy to assist you.

Until next time.

Hacktoberfest for .NET Developers (C# Edition)

Arafat Tehsin — Sun, 03 Oct 2021 05:23:11 +0000

Hacktoberfest is celebrated during the month of October when open-source software enthusiasts, beginners, and the developer community participate by contributing to open-source projects. This is its 8th year and since then, it has been growing every year. Thousands of developers, creators and opensource maintainers have been doing a commendable job in educating the beginners, guiding fellow developers or learning from others, on day to day basis. Opensource is truly amazing!

Although I created CognitiveRocket repo for the purpose of sharing my demos, slide decks etc. but I still remember submitting my first ever pull request on my friend's (James Mann) repo which later became one of the reasons to start Bot Builder Community Project and to enable all the Microsoft Bot Framework enthusiasts and developers to join hands together to build fantastic tools for everyone. 🛠

As I recently registered for the Hacktoberfest, I thought to look for the projects of my interest and skillset (as a C# developer) where I can contribute my time and effort. Hence, I thought to share with you as well, in case you also want to participate in the similar projects.

Please note that this list is not exhaustive so it's a high chance that you may find a better project of your interest than something from this list. Just click on the project name and it will directly take you to the appropriate issues (with the exception of first one).

SentimentAnalyzer

SentimentAnalyzer is an on-device (offline) .NET Standard library to find out what customers think of your brand or topic by analyzing raw text for clues about positive or negative sentiment. I created this library 2 years back using ML.NET and I try to keep it updated enough so that it does not break for anyone who is using it. This library can be used by any of your .NET apps be it your bot, mobile, web or even an IoT app.

As it hit around 10,000 downloads from NuGet package manager, I moved it to a separate project for better maintainability.

There's already an issue opened as an enhancement with the appropriate tags, please feel free to contribute.

grok.NET

Cross platform .NET grok implementation as a NuGet package.

Bot Builder Community

A collection of repos led by the community, containing extensions, including middleware, dialogs, recognizers and more for the Microsoft Bot Framework SDK.

Uno Platform

The only platform for building native mobile, desktop and WebAssembly with C#, XAML from a single codebase. Open source and professionally supported.

Azure Functions OpenAPI Extensions

This extension provides an Azure Functions app with Open API capability for better discoverability to consuming parties.

dotnet/Rosyln

The Roslyn .NET compiler provides C# and Visual Basic languages with rich code analysis APIs.

dotnet/Maui

.NET MAUI is the .NET Multi-platform App UI, a framework for building native device applications spanning mobile, tablet, and desktop.

dotnet/Docs

The official Microsoft .NET repo that contains documentation. While it's not directly related to your development stack, I think it is still good (and active for Hacktoberfest).

MudBlazor

Blazor Component Library based on Material design. The goal is to do more with Blazor, utilizing CSS and keeping Javascript to a bare minimum.

MaterialDesignInXamlToolkit

Comprehensive and easy to use Material Design theme and control library for the Windows desktop. Built in with XAML & WPF, for C# & VB.NET.

Silk.NET

The high-speed OpenGL, OpenCL, OpenAL, OpenXR, GLFW, SDL, Vulkan, Assimp, and DirectX bindings library your mother warned you about.

FluentAssertions

A very extensive set of extension methods that allow you to more naturally specify the expected outcome of a TDD or BDD-style unit tests. Targets .NET Framework 4.7, .NET Core 2.1 and 3.0, as well as .NET Standard 2.0 and 2.1. Supports the unit test frameworks MSTest2, NUnit3, XUnit2, MSpec, and NSpec3.

As I wrote earlier, this is not an exhaustive list, it's just I recently came across and found out that they are willing to accept the contributions and mark them as hacktober-accepted.

If you think I should include your repo or something which you have found interesting, please let me know and I will add it to the list.

Keep hacking.

Chatterbox with OpenAI’s GPT-3 and Bot Framework Composer

Arafat Tehsin — Fri, 06 Aug 2021 08:35:41 +0000

Last week my level of excitement just went up to another level when I got an access to OpenAI's API. The API features a powerful general purpose language model, GPT-3, and has received tens of thousands of applications to date. The main reason for my excitement was to explore what it has to offer in regards to the conversational AI capabilities.

We're still in the lockdown and already ran out of all kind of combinations of games we had to play our little one, so I thought to build something for her, at least we can now get to the point, one liner answers from Google Assistant.

Unlike many other AI systems which are designed for one use-case, OpenAI’s API today provides a general-purpose “text in, text out” interface, allowing users to try it on virtually any English language task. GPT-3 is the most powerful model behind the API today, with 175 billion parameters. There are several other models available via the API today, as well as other technologies and filters that allow developers to customize GPT-3 and other language models for their own use.

As of today, the API remains in a limited beta as OpenAI and academic partners test and assess the capabilities and limitations of these powerful language models.

In this post, rather than going into the details of API and their offerings, I'd take you through how easily you can connect your bot built using Microsoft Bot Framework Composer and surface OpenAI's capabilities to your customers. While there can be many use-cases ranging from the classification to QnA to key points extraction, I'd keep it quite simple and just add the chitchat capability which involves questions and answers functionality of OpenAI's famous model called Davinci.

Davinci is the most capable engine and can perform any task the other models can perform and often with less instruction. For applications requiring a lot of understanding of the content, like summarization for a specific audience and creative content generation, Davinci is going to produce the best results.

The API works in a unique way and to build a whole conversation around that needs a few tricks with the Composer but we'll talk about that after the bot's part.

The real part, Chatterbox

Now, let's focus on our main motive of this post; to integrate OpenAI's API with Bot Framework Composer bot which we call Chatterbox. This will try to answer any question you'd put in your chat window or speak to your favorite smart assistant (Google or Alexa). In order to achieve this, you need to have an API key. Hold on, if you do not have it, it still is good for you to continue reading this post. Why? Because you'll learn about integration of any authenticated API along with connecting your bot with Google Assistant. Yes, you read that right, we'll also integrate it with the Google Assistant (Alexa is easier though).

It's assumed that you're familiar with Bot Framework Composer and the basic concepts of creating a good experience with chatbots.

Once you have the API key, you first have to go to the Guides section of OpenAI's Documentation, skim it and move to the Completion section of your API reference. Over there, you will learn about the API specification such as endpoint, verb, request and much more. Protip: Copy the example request for usage, tailor it according to your needs later.

After going through the docs, open up the Bot Framework Composer, create a new empty Bot. Remove Send a response action and add Begin a new dialog action instead. This will help you create a new dialog (you can name it as TalkingDialog) and then you can start working in that particular dialog. Please note that this is not the best way to create your bots and you may want to add some natural language capabilities to your bot. It is also possible that your bot may have some sophisticated / complex flow such as ordering food, tracking a parcel and so on. The main reason I did is to demonstrate you the working of API rather than connecting it with any NLP services or building some sophisticated capabilities into this bot.

Now in TalkingDialog, you first have to create an experience that'd keep asking question after one another. As Bot Framework Composer lacks a while loop functionality for now therefore, we'd try to achieve this using some basic properties (such as dialog.isLaunched) and adding two question (text) prompts. Note that isLaunched property needs to passed as option from the Greeting trigger.

Then, you just have to add a text prompt. In the text prompt, you'd write a simple introductory message. I am writing the exact same message as I'd pass it to the API. On the input section, you can see that I'm setting up a property user.question which means that whatever user will write in response to the introductory message (which is kind of a question, but not really) will be stored in that property. This response of a user is actually considered as a question for the OpenAI's API. Then after receiving a response from the API (which is an answer in our case), that response will be set as a text prompt again. As soon as user asks another question, the Repeat this Dialog action will be called and it will then follow the different path rather than going into the introduction one.

This video clip may help. 🙂

What's in the API?

The API is super simple which is authenticated with the bearer (key); the docs are really well-written and contain many examples. However, it gets a little tricky when it comes to sending prompts. The reason of that is, it expects a newline (\n) escape sequence, as well as the prefix such as Q and A. This is different for different kind of use-cases. Such as for lengthy, contextual chats, you may have to set prefixes as Human and AI. This is all good when you're sending a request using clients like Postman (example below).

However, it gets really tricky when you have to keep the last prompt, concatenate with the new one and then send it again and repeat. This is valid (and working) for the case when you want to build some contextual conversation.

In our case, we're not doing it because we're just sending question and receiving an answer and showing it to the user. Now, let's see what's going on inside the Send an HTTP Request action. This is the place where we're filling in the request and sending it to the OpenAI's API. Purposefully, I have tried to avoid any variables, just to show you the raw request rather than keeping it in a object. You can see that I am passing user.question dynamically into the prompt. Also, in header, I am filling up with Content-Type and Authorization keys. This is how you can pass the authenticated request.

Note: I've removed the authentication key on purpose.

You can check out this video to see how the HTTP Action looks like.

The response is store in dialog.result and with the little string manipulation, I am able to send it back as a prompt to the user.

${substring(dialog.result.content.choices[0].text, 2, length(dialog.result.content.choices[0].text) - 2)}

This is how it is going to look in the Webchat / Emulator.

Surface your Bot to Google Assistant

In my last post, I've talked about the Bot Framework Component model. This has added a whole set of new capabilities to the Composer. You can now use nuget or npm packages to your bots, without writing any code or moving away from the Composer. In your Composer client, you can go to Packages area to add multiple first party and community (third party) packages. You can also build it locally and add them to your bot (this is out of scope for this post so I will skip it).

So, in this post, I'm going to talk about adding an opensource community component of Google Actions SDK. This component is powered by the Actions SDK adapter which is primarily built by Gary Pretty (big thanks to him). Once you're in the Package Manager section, find Bot.Builder.Community.Components.Adapter.ActionsSDK and click on install on the right.

Bot Builder Community is a collection of repos led by the community, containing extensions, including middleware, dialogs, recognizers and more for the Microsoft Bot Framework SDK.

Before moving any further with the Package or the Bot, let's look at the Google Actions which will be used to communicate with your Google Assistant. So instead of re-inventing another guide or if you do not know what Google Actions are and how you can use them with your Bot Framework Composer bot, I'd recommend you to follow this comprehensive guide. Follow this till the 8th point of Configure your Actions SDK Project. Once you're done configuring your project, you may see all the options (left-hand side) like below in your Action.

If you still get stuck in any way of creating Google Actions for your bot then please let me know in the comments and I'll be happy to help you out.

Let's go back to our Bot Framework Composer bot and add the configurations to your Action. 🔧

If you're like me who is running this bot locally for testing purposes (or haven't really made up a mind to publish this action) then you may want to use ngrok as this will give you an ability to test complete functionality of your bot by running it completely local. This doc covers everything around it. Once you've setup the tunnel of your local port with ngrok then you can copy that link of your bot and set it to the Webhook url of your Google Action.

Now! You're absolutely OK to test your Bot with your Action. Click on Start (or Restart) bot in your Composer and move to your Google Actions.

Actions Simulator 👎 - Google Assistant 👍

The experience with the Simulator is quite limited (read: bad) and that's why I never recommend you to test your bot with that. Once you're done with the first or second response of your bot, you must test it with the actual assistant.

However, this is how the simulator experience should look like.

You do not necessarily need a Google Nest Hub or Home to work with this. It's just that I have one so I love using it. If you own an Android device (or even an iPhone), you can still use Google Assistant.

My use-case was more of an educational but as you can see, it is good enough to work with any industry and on any scale. Therefore, solving customer service problems, extracting useful information or tailoring the whole experience for your business shouldn't be a big problem.

OpenAI has been really fruitful for the scenarios I have used here and offline and when it is combined with tools like Bot Framework Composer, it can surely bring the game changing experiences to the different business verticals. I'd be happy to see your comments on your thoughts and challenges if you have come across with this one, so far.

Until next time.

Introducing Bot Framework Component Model for better sharing and extensibility

Arafat Tehsin — Tue, 08 Jun 2021 07:53:48 +0000

Hey there! 👋 I am back with the bot post again.

The Conversational AI has witnessed a tremendous growth just in past 2-3 years. According to latest research, Gartner predicts that by 2022, 70% of customer interactions will involve bots — an increase of 15% from 2018. This does not come as a surprise, considering a good amount of consumers say they prefer chatbots to other virtual agents.

Last couple of months must have been super overwhelming for the overall Conversational AI team at Microsoft. Whether it's the Bot Framework SDK or Composer with tons of new features, Power Virtual Agents seamless integration with Composer or the new channels and improvements at the Azure Bot Service; there have been significant updates for businesses. Cherry on the top is Build 2021 goodness!

As I said in my previous blog post where I discussed the automatic generation of the Form Dialogs that every new feature deserves the whole blog post because of their explorative nature and productive use-case.

Therefore, today I am going to write about something which is freshly baked, super hot from the oven and that is the way of creating packages for your bots for better sharing, re-usability and cost-effective solutions. Before we get into the details of packages and their implementation, let us first understand why is it so important that a dedicated team worked on this new model for months to bring this goodness to you.

If you're relatively new to the Bot Framework (or Composer) then I'd suggest you to have a look here as in this post, it is assumed that you're familiar with the concepts and implementation of Bot Framework.

Challenges with the Current State

If you've built a sophisticated bot using Bot Framework Composer (< v2.0.0) which is responsible for booking and tracking the food orders from your customers and you want to extend your bot to incorporate some middleware, custom adapters and other actions; you may have to face a lot hurdles to sort that out and that won't be straight forward at all.

Even if you get successful by addressing all of these challenges, you may have to go through the same issues again for another bot. Lacks re-usability.

Furthermore, if you create your bot using (Enterprise) Virtual Assistant Template, QnA Maker or just simple Echo bot using Visual Studio, you may end up having different experiences with each of them and building a common functionality around them may lead to a dead end.

Therefore, the Bot Framework team realized and this became the reason of the evolution of Bot Framework Component Model.

Meet Bot Framework Component Model

The Bot Framework Component Model is the new framework to share and re-use bot's functionality in the form of bot templates (Empty bot, Core bot with Language), packages (Custom Adapters, Middleware) or complete bots as skills. You can now create the bots with confidence as you won't find yourself in a situation where you can't customize or find it difficult to share the portions of your bot with others (unless it's poorly designed 🤐)

Bot Framework Composer is now considered as the standard go-to approach for all sort of bots.

This model is comprised of a few essential pieces;

Adaptive Runtime

Adaptive Runtime is the core of this model which wraps up the Bot Framework SDK and provides the extensibility points to add any functionality by importing packages, adding your custom coded extensions or even connecting to skills. This means that you just have to take care of your business logic rather than worrying about the runtime code.

Templates

Templates such as Empty Bot, Core Bot with QnA Maker etc. are created on top of Component Model. Since they're built by composing the packages (discussed below) therefore you can always edit / extend them according to your needs. For example, if you create a bot using Core Bot with QnA Maker, you can then add other packages such as Custom Adapters (Google, Zoom etc.) or Declarative Assets (LG) such as Welcome, Help & Cancel. Flexibility is the key.

Skills

Skills are just regular bots with the plug and play functionality. These bots can be connected to your existing bots to process the messages for you. This is done through the Skill manifest which establishes the contract that bots can follow and communicate with each other.

Packages

Packages are the components (or bits as per the product team) of a bot you want to re-use and share. Packages can be in the form of Declarative Dialogs (LG templates such as Welcome, Help etc.), Custom Adapters (Google, RingCentral, WhatsApp or Zoom etc.), Custom Actions (Adaptive Cards, Microsoft Teams etc.), Middleware and more. They can also be built in the combination of one another such as Custom Actions with Declarative Assets and so on.

Packages are just regular NuGet / npm packages depending upon the code language of your bot. On NuGet / npm, these packages are tagged with msbot-component to be displayed in Bot Framework Composer.

With Bot Framework Composer v2.0.0 release, the creation experience is now based upon Adaptive Runtime and that's why you can see a new little icon on your left hand side of the Composer which indicates a whole section of Packages.

These packages can easily be added to your bot by just clicking on the Install button. If they're custom actions, you'll be able to see them right away in the Menu. For other types of packages such as Middleware, Custom Adapters, Skills etc. they can be used as described by their creators.

Check out the video on this link to see how you can add the packages

Did you notice that the team at Microsoft has done a great job in bringing up all of those packages upfront which were essential for bot development. For example, Adaptive Cards, Teams, Telephony and so much more.

But is it just Microsoft who is doing it? What if I want to connect it to some external middleware (as we discussed in start) or surface our bot to Google Assistant? Do we have something for that? Short answer is yes.

Community Packages

Bot Builder Community, with the awesome collaboration and contributions from Microsoft, has done a great job in developing all of these fantastic packages which includes several Custom Adapters, Recognizers and Hand-off packages. You can easily have a look at the community packages by selecting the community options from the list. Right now, they are not filtered properly but you can always identify the Community packages with BotBuilder.Community.Components.<type>.<name> pattern.

Community Packages are 100% opensource and you're most welcome to get some inspiration from the existing implementation or provide your feedback by opening an issue

Local Packages

Now if you are like me and you'd like to create your own or use packages developed by your team internally, then you'd want to test them before you actually publish them to either npm or NuGet. So, to address this issue, we've got to set-up the local feed (same as we do for other .NET / js packages) and we just have to add a new feed by clicking on the Edit feeds button inside Package manager. Once you've setup the local feed, your packages will start showing up and you can easily install / use them in a similar way as shown above.

Check out the video on how you can add the locally developed packages to your bots

Packages are the game changer esp. for Composer first bots. They definitely eliminate the need of re-writing code and increase the extensibility with better sharing capabilities. Not only this, it also allows you to connect with Skills seamlessly. In my upcoming post, I'll walk you through how you can create different types of Packages using Bot Framework Component Model.

Until next time.