DEV Community: aliona matveeva

Send Node.js logs from Docker to Grafana Cloud with Alloy

aliona matveeva — Fri, 02 May 2025 10:34:30 +0000

tl;dr: alloy config, docker-compose.

Any service that's meant to live more than couple of weeks eventually reaches the stage where you feel the need to properly monitor it. You usually start with simple console.log logging, but soon realize it's not readable enough, it's not searchable enough and it's only available on your server. Probably inside Docker container.

I was exactly at that point, annoyed by constant need to ssh into my server to check one log line. I've wanted to play with Grafana ecosystem for a while, too. So it seemed to be the perfect moment.

In this post, I’ll walk you through a simple and minimal setup that streams my Node.js application's logs into the Grafana Cloud dashboard using Grafana Alloy and Loki.

This is what my final setup looks like:

Now let’s break down how to make it work.

Producing Logs

I'm using pino to generate logs from my service. I won't dive deep into the setup as the library outputs raw JSON to stdout by default. An example log line would look like this:

{"level":30,"time":1746117737323,"pid":45,"hostname":"96773881a0b3","module":"server.js","env":"development","msg":"Server is running"}

Docker’s default logging behavior captures all stdout and stderr from your container and writes them into a file. json-file is the default logging driver in Docker, you can confirm this using:

docker info --format '{{.LoggingDriver}}'

You can also find out the actual log file location for a specific container:

docker inspect -f '{{.LogPath}}' <container>

And this exactly what Alloy will be reading and forwarding to Grafana Cloud.

Onboarding to Grafana Cloud and Setting Up Loki

Before we start wiring things up locally, let's prepare the Grafana Cloud workspace.

Navigate to Grafana Cloud and sign up or log in. In the sidebar, select Connections → Add new connection, select Loki. This is the place that prompts you to set up your Loki connection and allows you to generate an access token for Alloy.

In section 2, Install Grafana Alloy, click the "Run Grafana Alloy" button to retrieve necessary information. We're interested in the token, Loki username (GCLOUD_HOSTED_LOGS_ID) and Loki URL (GCLOUD_HOSTED_LOGS_URL). You'll need them later to set up Alloy config.

Setting Up Alloy Config

The configuration for the collector is stored in a file with *.alloy extension. Let's create config.alloy in the project root.

// Step 1. Discover Docker containers and extract metadata.
discovery.docker "linux" {
  host = "unix:///var/run/docker.sock"
}

// Step 2. Extract a service name from the container name using a relabeling rule.
discovery.relabel "logs_integrations_docker" {
  targets = discovery.docker.linux.targets

  rule {
    source_labels = ["__meta_docker_container_name"]
    regex = "/(.*)"
    target_label = "service_name"
    }
}

// Step 3. Collect logs from Docker containers together with relabel information and forward to Loki receiver.
loki.source.docker "default" {
  host = "unix:///var/run/docker.sock"
  targets = discovery.relabel.logs_integrations_docker.output
  labels = {"platform" = "docker"}
  forward_to = [loki.write.cloud.receiver]
}

// Step 4. Send logs to Grafana Cloud Loki.
loki.write "cloud" {
  endpoint {
    url = sys.env("GRAFANA_LOKI_URL")
    basic_auth {
      username = sys.env("GRAFANA_LOKI_USERNAME")
      password = sys.env("GRAFANA_CLOUD_API_KEY")
    }
  }
}

This config basically defines a pipeline of operations Alloy performs to collect, transform and deliver the data. In our case, we want to send logs from Docker to Grafana Cloud Loki. To do that, we'll pass the credentials you got earlier to config - using environment variables for safety.

💡 Tip: Grafana has a great Alloy scenarios repo with examples for different setups and logs sources. It helped me a lot in understanding how the pieces fit together.

Putting It All Together in Docker Compose

Since I’m already using docker-compose to run my service, integrating Alloy was as simple as adding another service definition.

services:
    app:
        build: .
        environment:
            NODE_ENV: ${NODE_ENV}
        ports:
            - '8080:8080'

    alloy:
        image: grafana/alloy:latest
        container_name: alloy
        ports:
            - '12345:12345'
        volumes:
            # mount config file
            - './config.alloy:/etc/alloy/config.alloy' 
            # give access to running docker containers for discovery.docker
            - /var/run/docker.sock:/var/run/docker.sock 
            # give access to docker's log files directory (optional)
            - /var/lib/docker/containers:/var/lib/docker/containers:ro 
        environment:
            # pass environment variables for config.alloy
            GRAFANA_LOKI_URL: ${GRAFANA_LOKI_URL}
            GRAFANA_LOKI_USERNAME: ${GRAFANA_LOKI_USERNAME}
            GRAFANA_CLOUD_API_KEY: ${GRAFANA_CLOUD_API_KEY}
        command:
            - run
            - --server.http.listen-addr=0.0.0.0:12345
            - --storage.path=/var/lib/alloy/data
            - /etc/alloy/config.alloy

The Grafana Cloud credentials might be stored in a .env file on your server.

Run and Verify

Let's build and run the containers using docker-compose up -d and verify everything is up by looking at the docker ps output:

If both containers are up and healthy and there are no errors in logs, you can proceed to the Grafana Portal to verify the updates are reaching the Cloud.

Navigate to Explore, select your data source (Loki) and apply filters. The service_name filter label comes from the relabelling step in the Alloy pipeline, so now you can easily search logs for your specific service.

And just like that - the logs are flowing! You can play more with additional filtering and processing, for now I just added a basic JSON parsing for more readable output.

What’s Next?

I’m still getting familiar with Grafana Dashboards, so there’s more to explore. In particular:

Pino logs use numeric levels (e.g. 30, 40), which aren’t parsed as human-readable levels by default in Grafana;
I’d love to build some fancy filters and dashboards to better visualize logs;
I'm eager to try pino-http logging and see how it can be visualized in Grafana.

These will probably be another blog post(s), so stay tuned :) Thanks for reading and I'd love to hear any suggestions or comments you may have!

Hello World, We Are CatBytes

aliona matveeva — Fri, 02 May 2025 10:10:26 +0000

Hi everyone! We’re are the CatBytes Community and we're excited to officially launch our presence here on DEV!

🐾 Who we are

We're an online community built by women in tech, for women in tech. CatBytes is all about creating a safe and genuinely supportive digital space for women who want to break into the tech world — whether you're just starting out or leveling up your skills.

Our mission is simple:

get more women into tech and lift each other up along the way. 🚀

We run free weekly web development lessons, collaborate on real projects, and we’re lucky to have some amazing mentors — experienced software engineers and tech leads — guiding and cheering everyone on.

💻 What to expect

Here on DEV, we are looking to learn, share, and connect. So let's do that! 💜

Our first post is coming soon! We'll talk about the monitoring setup we're using for our community platform — streaming logs from a Dockerized Node.js app to Grafana Cloud using Alloy.

More posts, insights, and stories to come. 🐱💬

We’re just getting started here, so if this sounds interesting — follow along!
Have questions, feedback, or just want to say hi? Drop a comment below — we’d love to hear from you.

Thanks for stopping by — and we’re happy to be here!

I Built a Bot to Chat with Our Team’s Wiki Using Azure OpenAI Service

aliona matveeva — Wed, 16 Apr 2025 08:25:00 +0000

🛠️ TL;DR: I used Azure OpenAI on Your Data + Teams AI Library + a bit of Python to build a bot that chats with our internal wiki.
It fetches content from Azure DevOps Wiki, pushes it to blob storage, indexes it with Azure AI Search, and answers questions — all without bothering my teammates 😄

My team’s project has been around for many, many years. That means the internal wiki is huge, and while it holds a lot of useful information, it’s really hard to search. Especially if you don’t know exactly what you’re looking for.

Disclaimer: The views expressed in this article are my own and do not necessarily reflect those of Microsoft. I am a Microsoft employee, and this article is based on my personal experience.

So usually my interaction with team’s wiki would look something like this: I have a question and I ask some of my senior colleagues → I’m waiting for an answer → The answer is a link to a wiki page. :D

Which is kind of ridiculous, because

I could find an answer myself, it was already documented, but I failed to search;
Waiting for a reply might take hours, if the colleague is busy or away;
Distracting teammates breaks their focus.

Besides, what is the point of a Wiki if you still need someone else to find the right page for you?

So I thought it would be fun to be able to ask a bot first to search the wiki more effectively and get quicker results.

Luckily, in 2024 Microsoft announced something called Azure Open AI on Your Data — a set of Azure services that “enables you to run advanced AI models such as GPT-35-Turbo and GPT-4 on your own enterprise data without needing to train or fine-tune models” (according to official documentation). Sounds about right. Let’s experiment!

Getting started: Going over the sample tutorial

Microsoft offers a wide set of tools to support your AI development. There’s the Teams AI Library (available in C#, JS and Python) — an official SDK for integrating GPT-based language models with custom Teams apps, and Teams Toolkit — a Visual Studio Code extension that simplifies creating, developing and testing those apps.

I started with this simple tutorial that teaches us how to create a custom engine agent in Teams.

Basically, it helps you:

Set up your data as the knowledge source for the GPT model
Create a bot and connect it to your Azure resources

I began by creating a new resource group and an Azure Open AI Service instance inside of my Azure subscription, then proceeded with the guide.

If you’re going with this tutorial, it guides you through a very convenient set of preconfigured interfaces. It automatically creates:

A Blob Storage for your files
An Azure AI Search instance with an index
Then it indexes your uploaded files (i.e. prepares them for search). Note that you can only upload and index your files once if you’re going by the manual.

Once the resources are ready, you’ll switch to Visual Studio Code and set up a simple Teams bot application boilerplate where you will only need to populate .env file with your Azure resources information.

We end up with this setup:

At this point you can already try running the code in Teams TestTool. If you did everything right, the chat will load in a browser window and you will be able to have a tiny conversation with your data. Yay!

Fed with couple of Wiki pages, the model now can respond to questions with a context specific to the team.

Making it dynamic: Adding an Indexer to Azure AI Search

As I mentioned earlier, when you set up your data source through the Azure AI Foundry portal interface, it indexes the data you upload to the blob once at the set up. But what if the data has changed? For something like a team’s wiki, the data is never static, it constantly updates, and we need the bot to keep up. This is where we need an Indexer.

An Indexer connects your data source to the index: all three are parts of Azure AI Search. The indexer is a crawler that processes textual data in the source files and populates the search index.

The indexer is easily added via Azure Portal. Before doing so, first we’ll need to create a data source — basically a “connection” to the desired storage, Azure Blob Container in my case. Then select the index that was generated at the previous step. Finally, configure the indexer — it can run on a schedule or be triggered manually; I’m keeping it manual for now.

Once your indexer is in place, you can upload new files to the blob storage and then hit the “Run” button to re-index. The fresh data will be loaded into the search index and instantly available to the bot.

Automating it: Parsing data from Azure DevOps Wiki

Our team’s wiki has couple of hundreds notes so uploading it manually is not an option. And then of course there’s a need to re-upload the blob container when articles change and new notes pop up. Let’s write a tiny Python script to do the job.

Let’s break down what we want the script to be doing and go step by step.

Get access to Azure DevOps Wiki and download article contents

To get machine access to Azure DevOps, you can either use a Personal Access Token (PAT) generated in your account settings, or a Managed Identity that already has access. Obviously, I would prefer Managed Identity, but for some reason it wouldn’t work specifically with ADO. After hours of failed attempts I gave up and decided to use PAT stored in Azure KeyVault, which, to my surprise, was available to connect using Managed Identity.

credential = AzureCliCredential()
secret_client = SecretClient(vault_url=f"https://{keyvault_name}.vault.azure.net", credential=credential)
# this token will later be passed as an Authorization header with API requests
pat = secret_client.get_secret("AzureDevOps-PAT").value

While Azure DevOps does have a Python SDK, the coverage is not optimal, so I went with raw API calls using requests library.

# if recursionLevel=Full is passed to the request, it will return all SubPages' relative paths
response = requests.get("https://dev.azure.com/{ado_org}/{ado_project}/_apis/wiki/wikis/{ado_wiki_name}/pages?path={root_wiki_path}&includeContent=true&recursionLevel=Full&api-version=7.1-preview.1", headers={"Authorization": f"Basic {encoded_pat}"})

# a helper function to extract all page paths
def get_all_pages_paths(root_page: dict) -> list[str]:
    pages_result = [root_page['path']]
    for page in root_page.get('subPages', []):
        pages_result.extend(get_all_pages_paths(page))

    return pages_result

# get list of all available wiki pages paths
pages_paths = get_all_pages_paths(response.json())

Every subpage has a relative path you can plug into the wiki URL’s path query parameter, like this:

for subpage_path in pages_paths:
  # no need for recursionLevel parameter as we only need to load the page and its contents
  response = requests.get("https://dev.azure.com/{ado_org}/{ado_project}/_apis/wiki/wikis/{ado_wiki_name}/pages?path={subpage_path}&includeContent=true&api-version=7.1-preview.1")

  page = page_response.json()
  print(page['content']) # some markdown content

Load Wiki files to Blob Container

To access blob storage, I used azure-storage-blob lib. The SDK provides a very simple interface for connecting to a Blob Container and uploading data into it. Nice thing is that we don't even need to store files locally, and can use content loaded from Wiki API directly.

blob_connection_string = secret_client.get_secret("BlobConnectionString").value
blob_service_client = BlobServiceClient.from_connection_string(blob_connection_string)

# after connecting to a blob storage, file upload is done easily
blob_service_client \
  .get_blob_client(container=blob_container, blob=filename) \
  .upload_blob(file_content, overwrite=True)

Since the bot is supposed to help people navigate the wiki, I decided to also store original wiki page’s URL in the file metadata. This metadata can later be reused in model’s response.

Trigger Search Indexer re-run

After all the files are uploaded to blob container, we need to re-index the data so the latest changes become available to the GPT model for search. This can be done with a simple API call:

indexer_run_url = f"https://{search_service_name}.search.windows.net/indexers/{indexer_name}/run?api-version=2023-07-01-Preview"

response = requests.post(indexer_run_url, headers={
            "api-key": secret_client.get_secret("IndexerAdminKey").value,
            "Content-Length": "0"})

After putting all this code together, we get a simple script that will download all of our wiki contents into Azure Blob Storage and then re-trigger Search Indexer to update the index. In production system, I would deploy this code as an Azure Functions App that would run on a schedule — so updates happen automatically. That only requires installing azure-functions lib and slightly adjusting the code.

Customize The Search: Index custom blob metadata

By default, the indexer processes blob’s content and some standard metadata like the filename or filepath. But if we want to include some custom metadata — in our case, wikiPageLink that stores original wiki page URL — we will need to make some adjustments to both the indexer and the index schemas.

Adding a fieldMapping to the Indexer so it knows to process our custom metadata.

In the Search Indexer, let’s add a new entry into the fieldMappings array: metadata_storage_wikiPageLink (where metadata_storage_ is the default prefix for storage files metadata) value will be mapped to a wikiPageLink field in the index. Note that we also need to apply a mappingFunction because custom metadata is base64 encoded by default.

Next, we’ll update the Search Index to include the very same field and make it searchable and retrievable. This will return wikiPageLink value in a search result. The final touch — we need to let the model know it should actually use the value. I believe there are some default settings that just teach the GPT model to process content of the search result. To make it aware of additional data, we can update the base prompt.

The default prompt I used was: “You are an AI assistant that helps engineers find information about their work and specifically project in their internal wiki.”

Now I will add additonal instructions: “In the end of your response, please tell explicitly “You can read more about this in the wiki page…” and add wikiPageLink URL value of index used for generating the response (if available). Add multiple urls if used multiple indexes and adjust the message accordingly.”.

Let’s try to run it. Nice!

The bot response now includes a link to Azure DevOps wiki page.

It is worth noting that though at first glance it looks good, unfortunately at some point I realized the model was hallucinating — returning incorrect links or even making up unexisting URLs. This is because by default the custom metadata fields aren’t processed in the default sample boilerplate setup. And unfortunately, the Teams AI Library used in this example doesn’t currently allow you to customize the search results before passing them to the model.

As a temporary solution, I updated my script to include the original wiki URL directly into the blob content, however it is still not performing very well. Going forward, I think I’ll need to rewrite the bot to use direct API calls or explore other approaches that give me more control for application customization.

For now, though — this is just a clear area for improvement in an otherwise promising setup.

Final Touch: Deploying the Bot

Since this is still a new and rapidly evolving area, I found this part of the journey particularly challenging to navigate. Azure’s ecosystem for AI development is growing fast — and constantly changing. There are multiple libraries (like botframework and the new Teams AI Library), and a ton of tutorials and docs, some of which are already outdated.

I spent quite some time just trying to figure out how to properly deploy a bot that runs fine locally — and make it actually available in Teams for my colleagues.

In the end, I once again relied on some predefined functionality from the Teams Toolkit (using this tutorial), which helped generate a setup that looked something like this:

Here,

App Service — Hosts and runs your bot
App Service Plan — Provides compute resources (CPU/memory)
Azure Bot — Registers the bot with the Bot Framework and connects it to Teams
Storage Account (optional) — Supports deployment logs, state storage, etc.

Once deployed, the Azure Bot comes with a built-in Web Chat interface, so you don’t even need to upload it to Teams right away to test it. Very convenient.

However, I realized quickly enough that the the Web Chat is open to the public internet. Meaning… if I leave the setup as-is, anyone could talk to my internal wiki bot. Not great.

To properly secure access and make the bot truly internal-only, I’d need to configure an Azure VNet (Virtual Network), since exposing sensitive data through a public endpoint is clearly not acceptable. To be honest, I haven’t gone down that rabbit hole yet — and I have a feeling it would be a whole separate article. I’ll get to it. Sooner or later.

Challenges

Alright, the cherry on top of this cake — here are a few issues I ran into while building the bot with Python. Hopefully, this saves you some time:

Poor SDK coverage

Azure DevOps SDK didn’t support some of the operations I needed, in particular loading wiki pages contents. This wasn’t big of an issue since I only needed a few endpoints, but it did make the code a bit messy: I had to mix SDK usage in some places with raw API calls in others.

Problems with TokenCredential

As mentioned earlier, I couldn’t get Managed Identity to access Azure DevOps. While manually generating a token with az cli worked fine, using TokenCredential didn’t pick up the correct token automatically for some reason, so I ended up using PAT, which is not really the way to go on a long run. Here is some reference for the problem.

Python support in Azure Functions (free tier)

The cheapest Azure Functions plan doesn’t support Python out of the box. Not a huge issue for this project, but something to keep in mind for anything more serious.

Conclusion

This was a fun little adventure — and there’s still a lot to explore. I love how AI services like this are becoming more available to the public audience helping to make life a bit easier. If you’ve tried something similar or have any thoughts about my setup, I’d love feedback!

Did you find this post helpful? Hit like and follow to read more of them later :)

A Guide to Debugging Python code (and why you should learn it)

aliona matveeva — Tue, 15 Aug 2023 14:45:00 +0000

Any time you start writing code for your application, everything works from the first try, you never encounter exceptions and errors, users are happy and so are you. This would be wonderful, wouldn’t it? Unfortunately it’s only possible in a dream. The reality is that your code fails. It raises exceptions, it produces unexpected outcomes, it makes users unhappy.

This is why debugging skill is essential in a developer’s tool kit. Debugging is a process of determining and fixing errors in your code. And while basic errors can be spotted by eye in simpler code, as the application is getting more complicated, we need tools to help. This article discusses the importance of these tools and why you need to master them right now.

Who the f… is the debugging?

Application issues generally fall into two categories: raising exceptions and silently doing things wrong. Sometimes, exceptions are clear, revealing mistakes immediately. Yet, for complex cases the answer is not on the surface, be it a weird exception thrown or an absolutely unexpected result of a program run encountered. And this is where we need to debug.

There are several options for fixing your code. The “easiest” way is to chaotically change different parts of it, hoping for the best. Sometimes it helps, but to be honest, this one is too inefficient and will probably waste a ton of your time.

To efficiently make changes, understanding the problem and, hence, the possible source of it, is the key. We begin this with studying the traceback. Here, when you have understood the problem you might start staring at your code, trying to think the solution out. Maybe you will add print()’s here and there, and maybe they will even give you a clue. This approach is legit when your application is simple, but, surprisingly, many developers stick to it even after years of working, spending more and more time with the code getting more complex.

The third option is on the ‘evolutionary’ top of the previous two: the proper debugging as a union of tools, approaches and experience.

The goal of debugging and types of tools

The ultimate goal of debugging is to discover the issue and figure out how each line of code affects the corrupted flow, how each value changes as the program keeps running. After this the only thing that’s left is implement a better solution that covers the edge cases.

Debugging involves setting breakpoints, stepping through code and inspecting values. Existing debugging tools can be split into three main types:

standalone UI applications — for example, PuDB, Winpdb. These are applications designed specifically for debugging, and the UI is basically the same as in IDE, which then, in my opinion, is not worth wasting your time. However, you can always learn more about it by yourself if you’re interested;
code-in tools, where you set breakpoints by adding temporary code to your program;
IDEs. All of the popular modern development environments, such as PyCharm, Visual Studio, PyDev and many more, have integrated debugging tools, allowing you to run and inspect the code at the same time.

Now let’s talk a bit more about the last two points.

An example

To explain how to debug in action, I’ll show you the process with the example code below.

Let’s say you’re creating an application to work with some external weather API. You build a pydantic model to load the API response in it to use the data afterwards (for example, print it to the console). Take a look at the example. This code doesn’t work.

from pydantic import BaseModel

class ApiResponse(BaseModel):
    city_id: int
    temperature: float
    weather: str

# we don’t see the contents of this method, pretend this is what an actual HTTP API call returns
def fake_weather_api_request(city: str):
    cities = {"New York": {"id": 42, "temperature": 25.6, "weather_type": "sunny"},
              "Marrakech": {"id": 12, "temperature": 30}}
    return cities.get(city)

if __name__ == "__main__":
    weather_resp = fake_weather_api_request("New York")
    ny_weather = ApiResponse(**weather_resp)
    print(f"Weather in New York (city id={ny_weather.city_id}): \n" +
          f"Temperature: {ny_weather.temperature} \n" +
          f"Weather type: {ny_weather.weather}")

When you run it, you will see the following exception message:

Oh. What would that mean? If we’re going the easy way and trying to guess the problem, we might think — probably, the following fields don’t come with a response for certain cities. Let’s make them nullable!

class ApiResponse(BaseModel):
    city_id: int = None
    temperature: float = None
    weather: str = None

Awesome. It’s not failing anymore.

Or is it? What if we know for sure the API has both city id and weather type info for New York? Why is it None then? We could continue playing a guessing game, but this is actually the perfect moment to start debugging.

The Python Debugger

Python provides you with an awesome built-in code debugger: the *pdb * module. This is an easy to use interactive Python code debugger. In Python versions prior to 3.6, we would need to import the module in the beginning of the file to use the debugger. In modern Python, though, even that is not necessary anymore — simply putting the built-in breakpoint() function call whenever you want your program to pause will work.

Let’s set a breakpoint before the API call.

if __name__ == "__main__":
    breakpoint()
    weather_resp = fake_weather_api_request("New York")
    ny_weather = ApiResponse(**weather_resp)

Now you can run the application from the terminal:

The application execution has paused at the place where you set the breakpoint, and the pdb is now awaiting for your command input. Type “h” to view the list of available commands:

Now you can navigate over your code, print variables values at different stages of program execution, step into methods to see what’s happening inside, etc. Let’s step over the API call by using the command n (next) and then see what was the actual response body with p.

Okay, it turns out, the response does actually contain all three fields, but why are some of them displayed as None when printing it out? If you look closely at the ApiResponse model defined in the beginning of the file, you will see that some of the fields names are slightly incorrect: city_id instead of id, and weather instead of weather_type. And the actual API response indeed doesn’t contain these fields. Let’s fix this.

class ApiResponse(BaseModel):
    id: int = None
    temperature: float = None
    weather_type: str = None

And don’t forget to change field names in the print statement and remove/comment the breakpoint() call:

if __name__ == "__main__":
    #breakpoint()
    weather_resp = fake_weather_api_request("New York")
    ny_weather = ApiResponse(**weather_resp)
    print(f"Weather in New York (city id={ny_weather.id}): \n" +
          f"Temperature: {ny_weather.temperature} \n" +
          f"Weather type: {ny_weather.weather_type}")

After you run the code again — yay, it works!

Congratulations, you’ve just successfully finished your first round of debugging. :)

Debugging in IDEs

An IDE (integrated development environment) helps you develop software efficiently. It combines all of the main functions needed for software development — code editing, building, running, testing and, finally, debugging. IDEs make debugging considerably easier so unless you’re a rigid fan of a terminal, I would recommend trying debugging in an IDE.

Most IDEs operate on similar principles and it doesn’t make sense to review all of them. So I will show you a few examples in PyCharm, my preferred IDE, but you can easily do the same in any other development environment of your choice.

Let’s open the same program code in PyCharm. To set a breakpoint, you need to click on the left of the line you want to pause at.

This will stop the program executing after receiving the API response. To see it in action, hit the green bug button in the upper right corner of the window.

When the application execution flow will reach the marked line of code, PyCharm will open a debugging sub-window containing useful things.

On the picture above, (1) is the list of currently defined variables and their values. Each variable can be expanded and deeply examined on all levels. This is extremely useful when the variable value is a complex object with nested values of different classes. (2) is the latest received/changed value. This is helpful for fast checking, if the value is simple as a dictionary like in our case or a string. And last, but not the least, at (3) you can see the toolbox for stepping through the code. Their description together with some additional documentation on debugging in PyCharm can be found on the official website: https://www.jetbrains.com/pycharm/features/debugger.html.

I, however, would like to pay extra attention to the tiny calculator button next to all the stepping buttons. This button is called “Evaluate expression” and the name is self-explanatory. This will save you a lot of time when you want to see different outcomes of an expression without changing and re-running your code. In our weather API example we could use the tool to see what might be the response for a city other than New York.

Voila, we can see that for some cities, for example, Marrakech, the weather_type attribute is missing. We could also try loading the response into the ApiResponse model right away and see how it would behave.

Turns out, for Marrakech the weather_type field will be None.

If you try to evaluate an incorrect expression, PyCharm will show you an error. But this error is only visible within the “Evaluate” window and won’t break the program flow. After you’ve checked everything you wanted, you can close the window and continue stepping through the code just like before trying out those things.

The debugging skill is only gained through experience, but once you embrace it, you’ll be surprised you managed without it for so long. Making mistakes is absolutely natural, and writing and writing bug-free code is nearly impossible (if you code more than a couple of lines a year). What truly matters is your ability to investigate the problem and fix the issue.

Did you find this post helpful? Hit like and follow to read more of them later :)

Telegram Conversation Summarizer Bot with ChatGPT and Flask (Quart)

aliona matveeva — Fri, 21 Apr 2023 12:00:00 +0000

Everybody talks about ChatGPT right now. The exceptionally smart AI keeps dazzling the internet even a couple of months after its release. Having the ChatGPT available on the website is awesome, however, the real fun begins when you gain API access. This gives you a great opportunity to integrate smart AI into your projects and applications to make them more powerful and introduce amazing features.

This article brings you a guide on how to create your own Telegram bot and integrate ChatGPT with it, using the OpenAI's Python library. This may sound like the simplest thing to do, but let's spice things up a bit by introducing the summarize command to get you a summary of several posts in the chat.

The post assumes you have basic knowledge of Python. However, I recommend checking out Hyperskill's Python and Flask tracks to learn more about Python and developing web applications with Flask.

Setting everything up

Before jumping right into the code, you'll need to do some preparations to get all the required accesses.

Register your bot in Telegram and retrieve the Telegram access token (using @botfather in Telegram)
Get access to the Telegram Core API and retrieve api_hash and app_id.
Sign up to OpenAI and retrieve the API access token.

Save those secret strings and guard them with your life. No stranger should get access to them: this may lead to security violations.

Writing the skeleton

Note: the full final project code (split into stages with commits) is available on my GitHub, please refer here for details: https://github.com/yellalena/telegram-gpt-summarizer

Python libraries you need to install for this step: flask, pydantic, requests, and pyngrok.

Let's begin with writing the code for the very basic Telegram bot. It should receive messages from chat and be able to respond to them.
First thing first - create a directory for your project and initialize a Python virtual environment. By the way, if you use PyCharm, it will create a virtual environment for you.

At this stage, the goal is split into four parts:

Create a simple Flask app with one root route to handle webhook with telegram messages.
Create a class for the Telegram bot and make it able to send messages to a chat.
Make the application visible to the big internet.
Register the application address in Telegram.

This is what main.py looks like at this point:

app = Flask(__name__)

@app.route('/', methods=["GET", "POST"])
def handle_webhook():
    update = Update(**request.json)
    chat_id = update.message.chat.id

    response = f"This is a response for message: {update.message.text}"
    app.bot.send_message(chat_id, response)

    return "OK", 200

def run_ngrok(port=8000):
    http_tunnel = ngrok.connect(port)
    return http_tunnel.public_url

def main():
    app.bot = TelegramBot(Config.TELEGRAM_TOKEN)
    host = run_ngrok(Config.PORT)
    app.bot.set_webhook(host)
    app.run(port=Config.PORT, debug=True, use_reloader=False)

if __name__ == "__main__":
    main()

Few things need clarification:

I like to put all configuration things in one place, so I created the config.py file, which will be gathering and storing our tokens and other useful information from exported environment variables.
Telegram sends updates as a nested JSON, so let's create a set of pydantic models to parse the input to be more convenient later.
To expose the application to the web, I use ngrok. It makes your localhost's specific port visible to everyone else, giving it a temporary public address. This is why it's important to make sure you expose the same port you're running your Flask app on.
Finally, I initialize the bot and set a webhook to the ngrok's public URL, so that the bot knows it has to communicate to this URL whenever it receives any message.

To set up a webhook, you will need to send a request to your bot's telegram API address, built using your acquired secret token. The telegram bot code looks as follows:

import requests
from config import Config

class TelegramBot:
    def __init__(self, token):
        self.token = token
        self.bot_api_url = f"{Config.TELEGRAM_API}/bot{self.token}"

    def set_webhook(self, host):
        host = host.replace("http", "https")
        set_webhook_url = f"{self.bot_api_url}/setWebhook?url={host}"
        response = requests.get(set_webhook_url)
        response.raise_for_status()

    def send_message(self, chat_id, message):
        send_message_url = f"{self.bot_api_url}/sendMessage"
        response = requests.post(send_message_url, json={"chat_id": chat_id,
                                                          "text": message})
        response.raise_for_status()

Now that everything is ready (don't forget I omitted some of the basic code, you can find it in the repo), export your bot token in an environment variable and hit that "Run"!

Yay! It's alive!

Adding a brain

Amazing, the next step now should be adding a pinch of intelligence to our smart bot. Install an official OpenAI's lib for Python using pip: pip install openai.
After that, we will be able to create a helper class to communicate with the AI.

import openai

class OpenAiHelper:
    def __init__(self, token, model="gpt-3.5-turbo"):
        openai.api_key = token
        self.model = model

    def get_response(self, message_text):
        response = openai.ChatCompletion.create(model=self.model,
                                                messages=[{"role": "user", "content": message_text}])
        return response.choices[0].message.content

The API suggests a variety of models to use for your project. The most popular are, of course, the GPTs. GPT-4 is the one that made the most noise lately, but (and because of that) it has limited access now, so for easier testing purposes, I choose GPT-3 instead. No big deal, you can always choose whichever you like the most, just change the string name you pass to the helper.

Don't forget to add OPENAI_TOKEN property to the config and let's use the helper in the code.

First, of course, instantiate it in the main() method:

And then call this guy from the view function, just like that:

response = app.openai_helper.get_response(update.message.text)

Whoosh! The magic is happening!

Summarize it!

Python libraries to install for this step: quart, telethon.

I bet you've been there - you have been added to a chat with a group of friends who like to discuss interesting things or share some news or ideas. You've had a lot of stuff to do and you've missed all the fun in the chat. Next thing you see - a hundred unread messages there. Wouldn't it be nice if someone could give you a brief overview of what happened there instead of reading all that? Well, GPT can do that, of course. We only need to ask it.

This is where the fun begins. For some reason, Telegram's bot API doesn't allow bots to read conversation history. We have webhooks, and we have the explicit GetUpdates() method, but they only work if someone has mentioned the bot. Another option is to make the bot get all the updates if it's added as an admin, but this approach has a few cons as well. First, you would need to set up whole storage for the messages. Second, what if you want to summarize the conversation that was going on before the bot was added to the chat? Not our case.

Obviously, that's not the reason to give up. Telegram provides the Core API, and this one can help with retrieving a chat history. The only thing is that it's asynchronous. And the most popular Python library for it, Telethon, is asynchronous, too. And Flask is synchronous. Uh-oh.

And that's where the mysterious Quart mentioned in the title comes onto the stage. Quart is the Flask API re-implemented using async, await, and ASGI web server (rather than synchronous and WSGI). Its main advantage in our case is that the syntax is basically the same. Let's do a quick code reorganization.

The changes are simple. First, adjust imports and change every Flask to Quart:

Then, make all web application's methods async. And await all the properties and methods that have become asynchronous now:

If you're not sure about what's async Python, I encourage you to check this part of Telethon documentation on the asyncio basics.

I have also moved ngrok and TelegramBot to launch them in a separate method decorated with @app.before_serving. This is a Quart built-in decorator that will ensure everything inside this method will run before the web app is up and serving. It's required so that the bot and the helper are both initialized in the same event loop as the main application.

@app.before_serving
async def startup():
    host = run_ngrok(Config.PORT)
    app.bot = TelegramBot(Config.TELEGRAM_TOKEN)
    app.bot.set_webhook(host)
    app.openai_helper = OpenAiHelper(Config.OPENAI_TOKEN)

Running the app has slightly changed as well, but not much. Hypercorn is the ASGI server used to run Quart asynchronously, and if we want to specify an application's port, we need to do it in a config. Note that the main() is now also async and is run using asyncio:

async def main():
    quart_cfg = hypercorn.Config()
    quart_cfg.bind = [f"127.0.0.1:{Config.PORT}"]
    await hypercorn.asyncio.serve(app, quart_cfg)

if __name__ == "__main__":
    asyncio.run(main())

That's it. Let's check if the changes went smoothly for our bot. Run, text, enter:

It's speaking. Great. Now, get the chat history for AI to summarize. Let's use Telegram's Core API with the help of Telethon lib. There, we will need the last two secret strings you have - export them as environment variables too.

TelegramBot has slight changes in the __init__ method: it will need to have a new core_api_client property which initializes a Telethon's client and of course, you need to pass the Core API secrets as arguments.

And this tiny method will be responsible for retrieving the history:

async def get_chat_history(self, chat_id, limit=30):
        if not self.core_api_client:
            return []
        history = await self.core_api_client.get_messages(chat_id, limit)
        result = [f"{message.sender.first_name} {message.sender.last_name}: {message.message} \n"
                  for message in history if not message.action]
        result.reverse()
        return '\n'.join(result)

Telethon's get_messages has many more different parameters you can pass beside the limit ones. For example, it can reverse the history, or limit it by date instead of the number of messages. It's fun to play with and you can adjust your own bot in any way you'd love to.

Well, we're almost done. The final touch is to add a summarization option to the webhook handler. This is what getting an answer looks like:

# process "summarize" command
    if update.message.text.startswith("/summarize"):
        history = await app.bot.get_chat_history(chat_id)
        response = app.openai_helper.get_response("Please, briefly summarize the following conversation history:\n" +\
                                                  history)
    else:
        response = app.openai_helper.get_response(update.message.text)

    app.bot.send_message(chat_id, response)

Let's see it blooming!

After you run the application for the first time, it will ask you to log in to Telegram. That's ok: it's required to get access to message history and other private data the Core API has to offer us. Enter the same phone number you used to get access to Telegram Core API. You will receive a verification code inside your app, and after that you're good to go.

Add the bot to a conversation with friends and ask for a summary:

That's it! There's an endless list of things you can continue with: add processing other types of messages besides text, configure a number of messages to summarize from the chat, etc. Go for it and don't forget to push your code onto GitHub. Happy coding! :)

Don't forget to jump onto Hyperskill website to keep learning about developing web applications with Python and Flask. Here are links to some topics that you might find useful exactly for this project:

Error handlers: If you don't handle errors properly, there is a high chance of your application failing in the runtime or showing ugly tracebacks to the user. To avoid that, read about types of errors and how it's best to handle them in a Flask app.
Logging: That's one of the most important things when it comes to testing and debugging your application. Writing meaningful and readable logs is a must for a software developer. Check this topic to learn how to perform logging in Python.
Intro to SQLAlchemy: When you decide you want to store some application data, whether it be any user info or conversation history, you will need to communicate to the database. This topic introduces you to the basics of SQLAlchemy which makes working with databases easy and convenient.

Hyperskill is a project-based learning platform that offers a personalized curriculum and a variety of tracks to help people from different backgrounds gain market-relevant skills through online education. It's not only giving you solid pieces of theory but allows you to practice the skills right away - and practice makes learning perfect.

Did you find this post helpful? Hit clap and follow Hyperskill and me to read more of them later :)

One Minute Notes: delegates in C#

aliona matveeva — Mon, 02 Jan 2023 22:46:13 +0000

I am staring a new section in my blog — the One Minute Notes, short notes on some important IT topics that will take only one minute to read and get a brief understanding of the concept. In this episode of One Minute Notes, let’s talk about what are delegates in C# and why do we need them.

⚡

Delegates might be considered as placeholders for functions that can be called later. Just like declaring a variable to store some integer value, you can declare a variable to store a function as a value, which allows you to change the function to be called in the runtime.

Here’s an example of defining and using a delegate. Of course, this is the most naive implementation. A real-life example would probably contain an if-clause or switch to choose between functions to delegate based on some condition.

public delegate int BinaryOperation(int a, int b);

class SampleClass
{
  public static int Add(int a, int b)
  {
     return a + b;
  }

  static int Subtract(int a, int b) => a - b;

  static void Main(string[] args)
  {
    BinaryOperation operation = Subtract;
    int x = operation (10, 2); // 8

    operation = Add;
    int x = operation(10, 2); // 12
  }
}

Fun fact: .NET Base Class Library also has its own predefined delegates that cover most popular use-cases, so it’s usually unnecessary to implement delegates of your own. The Action delegate represents a void function, which might take a parameter(s) with generic type. For example, Action<string> printText = x => Console.WriteLine(x); There’s also a Func delegate which defines functions that return a value. Overall, it has the same signature, but the last parameter defines the return value type. In the example above, the Add function might also be assigned to the Func<int, int, int> delegate.

Did you find this One Minute Note helpful? Hit like and follow me to read more of them later🙂

One Minute Notes: Master-Slave Database Replication

aliona matveeva — Sat, 08 Oct 2022 15:58:10 +0000

⚡

Data replication is a process of copying data between application’s databases. This is done for consistency, reliability, and performance of a service.

When replicating, the node where the data is modified and copied from is called master, and the node that receives data copy — a slave.

There are two types of data replication.

Single Master is the type of replication where data is modified on one (master) node only. After the changes are performed, the data from master database is replicated to slave database(s) that have read-only access for its clients.

This is the simple configuration of Master-Slave replication model, as it excludes any risk of conflicts.

Single Master replication is useful when you want to achieve safe data reading or have an always up-to-date database backup.

Some of Single Master replication’s weaknesses are the difficulty of write requests’ scale up and the need of manual failover handling.

Multi Master, in turn, is the configuration where data updating is available for multiple master nodes, that at the same time act as slaves for each other as well. As the data is modified and replicated everywhere “at the same time”, it enhances data’s availability in case one of the masters fail.

In this case it’s important to handle possible conflicts with concurrent changes on system level.

Multi Master replication is useful when you want to improve data writing and provide your service with a quick failover option.

One of main Multi Master replication’s weaknesses is the risk of losing some of transaction in case of one of master nodes failure.

Did you find this One Minute Note helpful? Hit like and follow me to read more of them later🙂

Using ADFS as the Identity Provider for your Flask application

aliona matveeva — Mon, 03 Oct 2022 19:18:29 +0000

Federated identity allows users to access different resources using the same set of credentials. It makes the workflow more efficient not only by improving user experience but by increasing security. At the same time, it also makes it easier to develop your own applications, saving you from implementing custom auth every time.

There are multiple different federated identity management systems, and in this article I’ll explain how to use one of them — Microsoft AD FS — as the identity provider in your Flask application. This article is intended for beginners.

The setup

For a better understanding of the structure and idea of the flow, I put everything on the scheme:

Here, the User wants to access some data on Resource X. As Resource X doesn’t have any credential storage or authentication mechanism, the Flask application needs to verify the user’s identity for Resource X — and this happens via AD FS. Usually, after the user identity was confirmed on the identity provider side, the IdP is sending a callback request to the Service Provider server. This callback request contains some trusted data (usually a token or user information) that will later be used to access the desired resource.

AD FS Prerequisites

Note that before actually writing a Flask app, you or your administrator need to configure the AD FS to be the identity provider. There are multiple manuals around the Web and I’m not going to repeat them. Here’s one of the many.

These are the main things that we’ll need to have configured to establish a relationship between our Service and Identity Providers —

for AD FS:

relying party SSO service URL
party trust identifier
custom claim rules

for Flask app:

identity provider URL
sign in URL
AD FS certificate

The talking protocol

Many identity providers use SAML 2.0 protocol to exchange security data, AD FS is one of them as well. SAML-based identity federation enables using single sign-on (SSO). SSO simplifies password management and user authentication, and that’s exactly what we need to make user experience smooth and comfortable. You can read more on SAML here.

We’ll be using the python3-saml library by OneLogin to manage SAML communication. This library helps to easily add SAML support to your Python web application, and I love its simplicity yet such a usability at the same time.

The python3-saml library has brilliant documentation, and for details you can always refer there, but as I was building my Service Provider app, I spent some time figuring out how do some of the things work, so I thought I’d put it all in one place.

The coding

The first thing after installing the library is configuring the settings.json file — the file that’s actually responsible for establishing a connection to the IdP. For my simple app, the bare minimum of the settings looks like this:

{
  "strict": true,
  "debug": true,

  // service provider -- our app -- data
  "sp": {
                // the sp_domain is your SP's address, in our case, that's the address
                // of the Flask application. the metadata URL is used by AD FS to identiyfy the SP
                // based on the info provided in the settings file
        "entityId": "https://<sp_domain>/metadata/",
       // specifies info about the callback endpoint that will handle IdP's SAML response. 
                // for our example, here the sp_domain will also contain the 'adfs' path, as in the adfs_route below
                // ACS stands for Attribute Consumer Service
        "assertionConsumerService": {
            "url": "https://<sp_domain>/?acs",
            "binding": "urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST"
        },
                // specifies info about the log out callback endpoint.
       // SLS stands for Single Logout Service
        "singleLogoutService": {
            "url": "https://<sp_domain>/?sls",
            "binding": "urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect"
        },
        "NameIDFormat": "urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified",
        "x509cert": "",
        "privateKey": ""
    },
   // identity provider -- ad fs -- data
   "idp": {
                // the idp_domain is set for the AD FS server.  
       // the /adfs/services/trust URI contains IdP's metadata
        "entityId": "http://<idp_domain>/adfs/services/trust",
       // specifies the location where the log in request will be sent
        "singleSignOnService": {
            "url": "https://<idp_domain>/adfs/ls/",
            "binding": "urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect"
        },
       // specifies the location where the log out request will be sent
        "singleLogoutService": {
            "url": "https://<idp_domain>/adfs/ls/",
            "binding": "urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect"
        },
                // the AD FS certificate 
        "x509cert": "ExampleCertString"
    }
}

This is basically it. Now we only need to configure Flask’s endpoints mentioned above, so the app is ready to communicate with the Identity Provider.

The settings.json file is loaded to the python-saml toolkit by initializing an OneLogin_Saml2_Auth object. This object will later be used to perform SAML2.0-based actions. It also requires a ‘request’ object that contains the current request’s data. Again, see the perfect documentation where they provide detailed information on each of these objects and how to configure everything for your needs.

I will just show you the final view that handles every AD FS interaction:

import os, requests
from Flask import request

@app.route("/adfs", methods=["GET", "POST"])
def adfs_route():
    req = prepare_request(request)
    # project_dirpath should contain full path to the root directory of the Flask project
    auth = OneLogin_Saml2_Auth(req, custom_base_path=os.path.join(%project_dirpath%, 'saml'))

    error = None

    # single sign on. the 'entrypoint'
    if "sso" in request.args:
        return redirect(auth.login())

    # process AD FS callback response
    elif "acs" in request.args:
        not_auth_warn = not auth.is_authenticated()
        auth.process_response()
        if not auth.get_errors():
            session["samlUserdata"] = auth.get_attributes()
            self_url = OneLogin_Saml2_Utils.get_self_url(req)
            if "RelayState" in request.form and self_url != request.form["RelayState"]:
                return redirect(auth.redirect_to(request.form["RelayState"]))
        elif auth.get_settings().is_debug_active():
            error = auth.get_last_error_reason()

    # process log out
    elif "slo" in request.args:
        session.clear()
        return redirect(url_for("adfs_route"))

    if "samlUserdata" in session:
        attributes = session['samlUserdata']
        user_id = attributes.get("USER_ID")[0]
        ### send request to the Resource X here ###
        # this is a dummy code just to show the idea
        resourcex_data = requests.get(f"https://resourcex.com/{user_id}")
        return render_template('data.html', data=resourcex_data)

    return render_template('index.html', error=error)

As simple as it is. In this example, I don’t process log out via AD FS, as for the minimum application and to demonstrate communication with AD FS Identity Provider terminating the session on our side only is pretty much fine. However, if you want to use the Single Log Out as well, you will need to configure the “sls” part of the endpoint, which will handle the AD FS log out callback after calling auth.logout().

Bonus — issues I ran into

While developing the Flask Service Provider application, I ran into a couple of issues that I’d like to share here to save your time:

At first the received AD FS callback had this ‘invalid_response’ error inside auth.get_errors(). The error message looked extra weird, like "The response was received at https://sp_domain.com/adfs instead of https://sp_domain.com/adfs". This is not a mistake, the URLs in the error message could be just the same. Turns out, sometimes, when your Flask app is hiding behind some proxy, some port confusion might appear. To solve that, consider configuring ‘server_port’ in the request dict built for the Saml2_Auth object.
I was a bit confused with this NameID thing, what are the different formats and how are they used and even why. As my AD FS was configured by an admin, they didn’t have the NameID specified, which caused errors. Apparently, the NameID is an important SAML element usually used to identify the subject SP and IdP are communicating about (in our case, the user). However, it’s not required. If you’re building a very basic app, it might not be needed, and the python3-saml lib allows you to turn NameID checking off. This is easily done by setting “wantNameId” attribute to false in advanced_settings.json file. If you want to know more about NameID formats, it’s worth checking section 8.3 of this SAML documentation.

Conclusion

As the number of services we use grows every day, we make our users remember more and more different credentials for different websites and applications. To simplify the process, most people reuse the same login/password pair for multiple accounts. But that’s not recommended as it creates a huge risk of an account being stolen or abused. Adding an Identity Provider to your organization solves this issue, improving user experience together with security. With AD FS and python3-saml it’s also easily implemented, and the bare minimum to set everything up is described in this article. Hope that helps!

5 Flask project ideas and why you should implement them

aliona matveeva — Tue, 05 Jul 2022 07:33:38 +0000

When learning a new skill, you need as much practice as possible because it’s the practice that makes your knowledge and applied skills stronger. Developing web apps is no exclusion: you can read tons of blog posts or crawl through hundreds of tutorials, but until you’ve touched it and figured out how it works by yourself, you will barely remember anything.

Sometimes you even know your weaknesses and might want to improve your knowledge in particular areas but coming up with a project idea focused on some specific topic can be almost as challenging as actually implementing it afterwards.

I gathered some Flask web applications ideas along with topics you can learn while working on them. Here we go:

Weather app

Weather app is a good start for Flask beginners. It doesn’t require a lot of core code writing, so you can focus on building the good UI for your application.

Using any open weather API (for example, OpenWeather API) you can easily get weather information on almost any place in the world. And then create nice HTML templates to display this information in a pleasant way.

Building a project from scratch all by yourself is indeed interesting, but if you need some pivots on how to organise the work, you can try Weather App at Hyperskill, where all the stages of implementing the project are described in detail.

Topics to work on: UI in Flask, Jinja, working with third party resources (APIs)

Movie Database

The name speaks for itself: if you want to create a movie database, you’ll have to store a lot of connected data. Movies, Directors, Actors… more and more.

All the instances are tightly connected to each other and designing and implementing this kind of relationships can get you some headache, but in the end, it is a very interesting and no doubt useful journey as storing data is something you’ll have to do in almost any project, so you better have some knowledge on it.

Topics to work on: Working with databases, designing tables and relations, SQLAlchemy in Flask

Todo list

I know, I know. Todo list is one of the most obvious things you usually see recommended to implement as it’s usually very easy. But if you want to get some more of a challenge, what if we turn a simple todo list idea into idea of creating some kind of simple Trello copy?

Besides database things like creating correct relationships for entities and storing all the data, you can add a simple log in system to the project, which will lead you to practice working with user sessions/user accesses in Flask.

Topics to work on: Authorization/Authentication, Flask sessions, working with databases, designing tables and relations

Web Calendar

Web calendar is an application simple enough to make it as a REST API: creating and storing calendar events should be pretty easy, while retrieving the data via raw HTTP request should be implemented carefully so that the information is convenient to consume by the end-user. So that’s a good opportunity to learn more about REST API concept in general, the way you actually design this kind of application, and of course about applying REST approach in Flask, building endpoints and marshalling data for output.

If you still don’t know where to start, check this Web Calendar project at Hyperskill.

Topics to work on: REST API, flask-restful, class-based views, marshalling data

Shared expense tracker

Shared expense tracker is not only a useful app in real life, it also allows you to practice validating user data before actually saving and using it. Also, this project needs to be tested good, as there are so many ways to break it: insert invalid data, split the bill incorrectly, etc. Building this project you can not only work with validating data, for example, using WTForms, but also learn about testing a Flask application, both unit and integration.

Topics to work on: Validating data, working with forms, Flask test context, pytest

These five project ideas can of course be adjusted precisely to your needs. But I described the areas each of them suits better to practice on. So go ahead, choose one and start implementing. You’ll get good experience, learn new things to do in Flask and probably will have some fun (only joking. Of course you will). And don’t forget to push your project on to your GitHub! Happy coding :)

Profiling Flask application to improve performance

aliona matveeva — Sun, 28 Feb 2021 09:57:52 +0000

In my earlier experience my Flask apps were pretty simple and everything worked just as it has been created from the scratch. I loved it, but then the inevitable thing has happened: my application got really slow and I HAD to do something about it. In this post I’ll tell my story about looking for the bottleneck of my Flask app, solving the problem and will share some amazing tools I used for it.

So I have a Flask application and a MySQL database which contains a lot of one-to-many super-nested objects. And by saying ‘super-nested’ I’m not hyperbolising: an object of total 3000 table rows from five different tables is a common case. At the beginning everything was fine, but at some point processing the request started to take 3-5 or even 10 seconds! Oh.My.God. This was something I didn’t want to be happening, so I started to think what might cause the problem.

At first, I naively went through my code base and tried to determine the area of possible bottleneck. The answer seemed to be obvious: I’m using the marshmallow library to serialise and validate data before inserting and after selecting it from the database. This is it, right? Googling “marshmallow + slow” returned a few articles that confirmed my suspicions: it’s the library. One of the articles I found was by some guys from Lyft, that said: “We’re using marshmallow and it’s so slow, so we created toasted-marshmallow which is 15x times faster”. Amazing!- I thought. This is what I need. At that point my requests took an average of three seconds. So I updated my code to using toasted marshmallow and prepared myself a red stripe and scissors. Sent a request… bam. Six seconds. That was impressive.

I got pretty upset because I thought I have to rewrite half of my app’s logic. And then a colleague asked me if I tried using a profiler? Yes, at this point I have to admit I didn’t know profilers existed. I’m happy I do know now, though!

What is a code profiler? Long story short, it’s a tool for dynamic code analysis that helps to detect performance problems, also known as bottlenecks of your program. The profiler gathers information on various metrics of how your program works, and based on this information you can identify where to move on with code optimisation.

There are a lot of profiling tools for Python code, and most of them are built-in — like profile or cProfile. Since I’m speaking about Flask application, let’s see what the world has especially for it. There is a beautiful lib called flask-profiler, which has a web interface with some cool features such as route or date filters. But Flask also has a built-in in werkzeug's profiler. It looked awesomely easy in use, so it was the first — and the last — one I tried.
To use the built-in profiler you’ll need to add only two lines of code to your project:

from werkzeug.middleware.profiler import ProfilerMiddleware
app = ProfilerMiddleware(app)

You can also configure it, for example, specify the profile_dir or set restrictions for the stats you want to see.

After adding the two lines before the Flask app.run() function and executing the program, overview result on each request will be displayed in the stdout.

Sometimes this short result can give you an idea on what’s slowing your program down. But usually it’s interesting to see thee detailed analysis, which is put into the chosen profile directory as *.prof files.

There are a few tools to visualise the profile dumps. Some of them providing a full GUI for navigatig within your profiling results ( RunSnakeRun), some of them represent the analysis result as a Graph (gprof2dot).
I stopped on snakeviz, which is a browser based visualizer.
It is easy installed using pip install snakeviz, and then simply run with snakeviz profile_dir. The result looks something like this, and you can dive in to each of the visual parts to see its more close detalization, which is in my opinion is super cool and handy.

After analysing the visual representation of the profiling results, it turned out that most of my performance problems were coming from service-database interaction. When I knew exactly which lines of code took inexcusably lot of time to execute, I was able to solve my problems whether by rewriting the queries, or improving the code itself. Improving code performance via enhancement of database interaction is a topic for a whole new article, so I will probably write about my experience with it later.

To sum up: of course, automatic profilers are not perfect and they won’t 100% show you the mistakes in your code. Sometimes it’s simply ‘staring at your code’ method that actually works really good. But at least with the help of profilers you can detect the weak area which in most cases is more than enough.

Thank you for reading and I hope it was somehow useful. :)