DEV Community: Dang Hoang Nhu Nguyen

[BTY] Day 14: How to send notification from your Python app to Slack?

Dang Hoang Nhu Nguyen — Mon, 21 Feb 2022 16:03:27 +0000

This post was for 18.02.2022

A lot of time, you want to be noticed when something happens to your application on the server, so you can take action as soon as possible.

Sending Slack messages to a channel in case of exceptions (with phone notifications, emails included in settings) is one of the simplest ways I found to deal with the mentioned issue.

Disclaimer: This is a legacy custom integration - an outdated way for teams to integrate with Slack. You may visit the Slack documentation for update and the modern way to do so.

1. Create new Slack workspace

2. Setting up

Create new channel ⇒ Right click and select Open channel details

Integrations > Add an App

Search and select Incoming Webhooks

Add Incoming WebHooks to the target channel (e.g #test-notification)

Setup Instructions (from the documentation)

We'll guide you through the steps necessary to configure an Incoming Webhook so you can start sending data to Slack.

Webhook URL

https://hooks.slack.com/services/xxxxxxxxxxxxxxxxxxxxx

Sending Messages

You have two options for sending data to the Webhook URL above:

Send a JSON string as the payload parameter in a POST request
Send a JSON string as the body of a POST request

For a simple message, your JSON payload could contain a text property at minimum. This is the text that will be posted to the channel.

A simple example:

    payload={"text": "This is a line of text in a channel.\nAnd this is another line of text."}

This will be displayed in the channel as:

Adding links

To create a link in your text, enclose the URL in <> angle brackets. For example: payload="text": "" will post a clickable link to https://slack.com. To display hyperlinked text instead of the actual URL, use the pipe character, as shown in this example:

    payload={"text": "A very important thing has occurred! <https://alert-system.com/alerts/1234|Click here> for details!"}

This will be displayed in the channel as:

Customized Appearance

You can customize the name and icon of your Incoming Webhook in the Integration Settings section below.However, you can override the displayed name by sending "username": "new-bot-name" in your JSON payload. You can also override the bot icon either with "icon_url": "https://slack.com/img/icons/app-57.png" or "icon_emoji": ":ghost:".

Channel Override

Incoming webhooks have a default channel, but it can be overridden in your JSON payload. A public channel can be specified with "channel": "#other-channel", and a Direct Message with "channel": "@username".

Example

Putting this all together, here is a sample curl request for posting to a channel.

    curl -X POST --data-urlencode "payload={\"channel\": \"#test-notification\", \"username\": \"webhookbot\", \"text\": \"This is posted to #test-notification and comes from a bot named webhookbot.\", \"icon_emoji\": \":ghost:\"}" https://hooks.slack.com/services/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

This will be displayed in the channel as:

Python Code

    import requests
    from loguru import logger

    class SlackWebhookBot:
        def __init__(self, webhook_url: str,
                     channel_name: str = "test-notification",
                     bot_name: str = "guardian",
                     bot_emoji: str = ":guardsman:",
                     timeout: int = 15):
            """Class to send messages to a provided Slack webhook URL.

            You can read more about Slack's Incoming Webhooks here:
                https://api.slack.com/messaging/webhooks

            Args:
                webhook_url: The webhook URL to send a message to.  Typically
                    formatted like "https://hooks.slack.com/services/...".

            Kwargs:
                timeout: Number of seconds before the request will timeout.
                    This is used to prevent a hang and is set to a default
                    value of 15 seconds.
            """
            self.webhook_url = webhook_url
            self.channel_name = channel_name
            self.bot_name = bot_name
            self.bot_emoji = bot_emoji
            self.timeout = timeout
            self.headers = {
                'Content-Type': 'application/json',
            }

        def send_notification(self, message: str) -> bool:
            """Sends a message to the webhook URL.

            Per the Slack Incoming Webhook example.  The body of the request
            (for plain text) should be formatted as follows:
                `{"text": "Hello, World!"}`

            Args:
                message: Plain text string to send to Slack.

            Returns:
                A boolean representing if the request was successful.
            """
            success = False
            payload = {
                'channel': self.channel_name,
                'username': self.bot_name,
                'icon_emoji': self.bot_emoji,
                'text': message,
            }
            try:
                requests.post(
                    self.webhook_url,
                    headers=self.headers,
                    json=payload,
                    timeout=self.timeout
                )
            except requests.Timeout:
                logger.error('Timeout occurred when trying to send message to Slack.')
            except requests.RequestException as e:
                logger.error(f'Error occurred when communicating with Slack: {e}.')
            else:
                success = True
                logger.info('Successfully sent message to Slack.')

            return success

    if __name__ == '__main__':
        SLACK_WEBHOOK_URL = "https://hooks.slack.com/services/xxxxxxxxx"
        slack = SlackWebhookBot(SLACK_WEBHOOK_URL)
        slack_test_message = '[TESTING] Sending this warning message when we have any problems.'
        logger.info(slack.send_notification(message=slack_test_message))

[BTY] Day 13: Yopass - Share Secrets Securely

Dang Hoang Nhu Nguyen — Mon, 21 Feb 2022 15:46:32 +0000

This post was for 17.02.2022

Yopass is a project for sharing secrets in a quick and secure manner.

The message is encrypted/decrypted locally in the browser and then sent to yopass without the decryption key which is only visible once during encryption, yopass then returns a one-time URL with specified expiry date.

Because it's an open-source project, you can deploy it anywhere on your own. It has full transparency with the possibility to audit and submit features.

Check it here: https://yopass.se/

[BTY] Day 6-12: Read Practical MLOps book

Dang Hoang Nhu Nguyen — Wed, 16 Feb 2022 14:46:04 +0000

This post was for 09.02.2022 to 15.02.2022

I have finished the book Practical MLOps: Operationalizing Machine Learning Models, written by Noah Gift.

https://www.amazon.com/Practical-MLOps-Operationalizing-Machine-Learning/dp/1098103017/

For me, this book has given me a comprehensive understanding of operating ML systems in general, with real-world use cases and sample code. Few sections I had skipped because those chapters are designed for specific cloud services that I have not planned to use in the near future.

However, the interesting point is, this book also has a lot of appendixes that made me feel excited. To have a successful career and a balanced lifestyle, technical knowledge with a 9-to-5 job is not enough. You have to know about how to grow yourself in terms of hard skills and soft skills: how to manage a project, how to do a side project to create your technical portfolio, how to learn new things, how to market yourself, how to invest and have a passive income, how to keep healthy by eating clean food or doing exercise, ...

I will post my keynotes on this book soon. Recommend anyone to read this book if you're looking for a guide to build an ML system.

env in prod and dev

Dang Hoang Nhu Nguyen — Sun, 13 Feb 2022 10:27:15 +0000

[BTY] Day 5: ctime() does not refer to creation time

Dang Hoang Nhu Nguyen — Wed, 09 Feb 2022 17:13:53 +0000

This post was for 08.02.2022

In Python, I usually retrieve ctime (means creation time for me) of a file using the following code:

from pathlib import Path
f = Path(file_path). # file_path is the path to any files in your system
file_stat = f.stat()
print(file_stat.st_ctime)

BUT,

I've got troubles when relying on this approach, because:

ctime() does not refer to creation time on *nix systems, but rather the last time the inode data changed.

I found this from this answer: https://stackoverflow.com/a/237084/6563277

The reason is that I have modified ctime by doing something to the inode: chmod and chown on the file.

[BTY] Day 4: AI Roadmap

Dang Hoang Nhu Nguyen — Tue, 08 Feb 2022 15:43:40 +0000

This post was for 07.02.2022

I come across this Roadmap from my colleague. The link is here: https://i.am.ai/roadmap/#note

Quote from the guideline:

The purpose of these roadmaps is to give you an idea about the landscape and to guide you if you are confused about what to learn next and not to encourage you to pick what is hip and trendy.

And many more!

Hope I could find the right roadmap for myself in the AI journey!

[BTY] Day 3: Improve software engineering skills as a researcher

Dang Hoang Nhu Nguyen — Sun, 06 Feb 2022 07:57:15 +0000

All the information are from this article of Lj Miranda: https://ljvmiranda921.github.io/notebook/2020/11/15/data-science-swe/. Please read the original for more details and related resources about the following notes.

Aside from developing Deep Learning models, you have to know how to create a machine learning application that receives HTTP requests, then deploy it as a containerized app. This task, aka. building Machine Learning (ML) Service, relates to software engineers that we (assume you're researchers like me) are lacking in skills.

Why?

Improves engineering sensibilities.

Most applications treat ML models as software components.

Increases familiarity with the ML workflow.

We’re familiar with the ML experimentation workflow. In addition, there is also a productization workflow where we deploy our models, perform A/B testing, take care of concept drift, and more.

Another tool under your belt to create more cool stuff.

Even if you won’t be working as a full-fledged ML Engineer or Developer, the technologies you’ll learn while building an ML Service enables you to do more things!

How?

1. Be comfortable with UNIX commands and a version-control system like Git.

2. Structure your Python project in a modular fashion

 my_project/
  ├── api
  ├── docs
  ├── experiments
  ├── README.md
  ├── requirements.txt
  ├── src
  │   ├── entrypoint.py
  │   └── my_module
  │       └── module_file.py
  └── tests
      ├── my_module
      │   └── test_my_module.py
      └── test_entrypoint.py

3. Learn how to write an API on top of your model using Flask or FastAPI

4. “Containerize” your application using Docker

You want to use Docker for two things: (1) reproducibility and (2) isolation.

5. Learn how to deploy to a Cloud Platform

What’s next?

From here on in, you can keep improving your app by:

Minimizing the size of your Docker image using multi-stage builds.
Cleaning-up your repository. Model files shouldn’t be committed but stored in a storage service (e.g. Google Cloud Storage or AWS S3)
Adding a Continuous Integration / Continuous Deployment (CI/CD) pipeline so that any change on Github is automatically reflected on your deployed app. I often use Github Actions for this (e.g., any change in the master branch is deployed automatically).
Improving security! Make use of Docker args or .env to secure API tokens, passwords, and whatnot. Ideally you shouldn’t be committing any secrets on Git (it can still be recovered if you deleted it!). Be careful!

[BTY] Day 2: REST Resource Naming Guide

Dang Hoang Nhu Nguyen — Sun, 06 Feb 2022 06:58:57 +0000

This post was for 05.02.2022

There are only two hard things in Computer Science: cache invalidation and naming things. -- Phil Karlton

Recently, I have had to build an API service for an internal project in my company. URI naming has stressed me out.

Fortunately, I found an interesting article that lists out the best practices to name REST Resource when you design API for your application.

The article is here: https://restfulapi.net/resource-naming/.

Best Practices

Use nouns to represent resources

RESTful URI should refer to a resource that is a thing (noun) instead of referring to an action (verb) because nouns have properties that verbs do not have – similarly, resources have attributes. Some examples of a resource are:

Users of the system
User Accounts
Network Devices etc.
and their resource URIs can be designed as below:

http://api.example.com/device-management/managed-devices 
http://api.example.com/device-management/managed-devices/{device-id} 
http://api.example.com/user-management/users
http://api.example.com/user-management/users/{id}

For more clarity, let’s divide the resource archetypes into four categories (document, collection, store, and controller). Then it would be best if you always targeted to put a resource into one archetype and then use its naming convention consistently.

For uniformity’s sake, resist the temptation to design resources that are hybrids of more than one archetype.

document

A document resource is a singular concept that is akin to an object instance or database record.

In REST, you can view it as a single resource inside resource collection. A document’s state representation typically includes both fields with values and links to other related resources.

Use “singular” name to denote document resource archetype.

http://api.example.com/device-management/managed-devices/{device-id}
http://api.example.com/user-management/users/{id}
http://api.example.com/user-management/users/admin

collection

A collection resource is a server-managed directory of resources.

Clients may propose new resources to be added to a collection. However, it is up to the collection resource to choose to create a new resource or not.

A collection resource chooses what it wants to contain and also decides the URIs of each contained resource.

Use the “plural” name to denote the collection resource archetype.

http://api.example.com/device-management/managed-devices
http://api.example.com/user-management/users
http://api.example.com/user-management/users/{id}/accounts

store

A store is a client-managed resource repository. A store resource lets an API client put resources in, get them back out, and decide when to delete them.

A store never generates new URIs. Instead, each stored resource has a URI. The URI was chosen by a client when the resource initially put it into the store.

Use “plural” name to denote store resource archetype.

http://api.example.com/song-management/users/{id}/playlists

controller

A controller resource models a procedural concept. Controller resources are like executable functions, with parameters and return values, inputs, and outputs.

Use “verb” to denote controller archetype.

http://api.example.com/cart-management/users/{id}/cart/checkout http://api.example.com/song-management/users/{id}/playlist/play

Consistency is the key

Use consistent resource naming conventions and URI formatting for minimum ambiguity and maximum readability and maintainability. You may implement the below design hints to achieve consistency:

Use forward slash (/) to indicate hierarchical relationships

The forward-slash (/) character is used in the path portion of the URI to indicate a hierarchical relationship between resources. e.g.

http://api.example.com/device-management
http://api.example.com/device-management/managed-devices
http://api.example.com/device-management/managed-devices/{id}
http://api.example.com/device-management/managed-devices/{id}/scripts
http://api.example.com/device-management/managed-devices/{id}/scripts/{id}

Do not use trailing forward slash (/) in URIs
As the last character within a URI’s path, a forward slash (/) adds no semantic value and may confuse. It’s better to drop it from the URI.

http://api.example.com/device-management/managed-devices/ http://api.example.com/device-management/managed-devices  /*This is much better version*/

Use hyphens (-) to improve the readability of URIs

To make your URIs easy for people to scan and interpret, use the hyphen (-) character to improve the readability of names in long path segments.

http://api.example.com/device-management/managed-devices/
http://api.example.com/device-management/managed-devices    /*This is much better version*/

Do not use underscores ( _ )

It’s possible to use an underscore in place of a hyphen to be used as a separator – But depending on the application’s font, it is possible that the underscore (_) character can either get partially obscured or completely hidden in some browsers or screens.

To avoid this confusion, use hyphens (-) instead of underscores ( _ ).

http://api.example.com/inventory-management/managed-entities/{id}/install-script-location  //More readable

http://api.example.com/inventory-management/managedEntities/{id}/installScriptLocation  //Less readable

Use lowercase letters in URIs

When convenient, lowercase letters should be consistently preferred in URI paths.

http://api.example.org/my-folder/my-doc       //1
HTTP://API.EXAMPLE.ORG/my-folder/my-doc     //2
http://api.example.org/My-Folder/my-doc       //3

In the above examples, 1 and 2 are the same but 3 is not as it uses My-Folder in capital letters.

Do not use file extensions

File extensions look bad and do not add any advantage. Removing them decreases the length of URIs as well. No reason to keep them.

Apart from the above reason, if you want to highlight the media type of API using file extension, then you should rely on the media type, as communicated through the Content-Type header, to determine how to process the body’s content.

http://api.example.com/device-management/managed-devices.xml  /*Do not use it*/

http://api.example.com/device-management/managed-devices    /*This is correct URI*/

Never use CRUD function names in URIs

We should not use URIs to indicate a CRUD function. URIs should only be used to uniquely identify the resources and not any action upon them.

We should use HTTP request methods to indicate which CRUD function is performed.

HTTP GET http://api.example.com/device-management/managed-devices  //Get all devices
HTTP POST http://api.example.com/device-management/managed-devices  //Create new Device

HTTP GET http://api.example.com/device-management/managed-devices/{id}  //Get device for given Id
HTTP PUT http://api.example.com/device-management/managed-devices/{id}  //Update device for given Id
HTTP DELETE http://api.example.com/device-management/managed-devices/{id}  //Delete device for given Id

Use query component to filter URI collection

Often, you will encounter requirements where you will need a collection of resources sorted, filtered, or limited based on some specific resource attribute.

For this requirement, do not create new APIs – instead, enable sorting, filtering, and pagination capabilities in resource collection API and pass the input parameters as query parameters. e.g.

http://api.example.com/device-management/managed-devices
http://api.example.com/device-management/managed-devices?region=USA
http://api.example.com/device-management/managed-devices?region=USA&brand=XYZ
http://api.example.com/device-management/managed-devices?region=USA&brand=XYZ&sort=installation-date

If you have any useful resources, please help me know by commenting on the post. Thank you!

[BTY] Day 1: 7 Reasons Not to Put an External Cache in Front of Your Database | ScyllaDB

Dang Hoang Nhu Nguyen — Fri, 04 Feb 2022 13:57:02 +0000

I've passed through a few days relaxing from the Tet holiday (aka. Lunar New Year). So now, we start again from the beginning!

All information is extracted from this article: https://www.infoq.com/vendorcontent/show.action?vcr=207f2eed-6333-49ed-802e-5e5e3567f4e9

Disclaimer: I've not used ScyllaDB yet.

My key takeaways

Traditional side caching vs transparent caching (NEW)

The Linux page cache
The Linux page cache, also called disk cache, improves operating system performance by storing page-size chunks of files in memory to save on expensive disk seeks. The Linux kernel treats files as 4KB chunks by default. This speeds up performance, but only when data is 4KB or larger. The problem is that many common database operations involve data smaller than 4KB. In those cases, Linux’s 4KB minimum leads to high read amplification.

The seven fails of external caches

An external cache adds latency
An external cache adds needless expense
External caching decreases availability
Application complexity — your application needs to handle more cases
External caching ruins database caching
External caching complicates data security
External caching ignores the database’s intelligence and resource

Scylla's embedded cache

The Scylla NoSQL database offers a better approach to caching, one that addresses the significant problems covered above, while also delivering the performance gains that caching promises.

Scylla uses a special-purpose cache instead of Linux default cache. It can dynamically tune itself to any workload and always controls their eviction and memory footprint.
Scylla can dynamically balance the different types of caches stored.
Once data is no longer cached in memory, Scylla will generate a continuation task to read the data asynchronously from the disk using direct memory access (DMA), which allows hardware subsystems to access main system memory independent of CPU.
The C++ Seastar framework on which Scylla is built will execute the continuation task in a μsec (1 million tasks/core/sec) and will rush to run the next task. There’s no blocking, heavyweight context switch, waste, or tuning.
Because Scylla has a very efficient internal cache it obviates the need for a separate external cache, making for a more efficient, reliable, secure and cost-effective unified solution.

[BTY] Day 5: Lessons when you're learning code

Dang Hoang Nhu Nguyen — Sun, 30 Jan 2022 10:49:13 +0000

This post was for 28.01.2022

Those lessons are from a Youtube video I came across recently.

Link here: https://www.youtube.com/watch?v=s6dMWzZKjTs&list=PL0s6MZJVJ8HZ0eYoUt3KM8WR1AQJWQnXc&index=5

Lesson 1: Don't learn things randomly,
but follow a pre-made roadmap or a structured path because you will be able to see your progress.

Lesson 2: Focus on one thing at a time.
Stay on one subject until you have a decent grasp of the basics

Lesson 3: Be an active learner.
Don't just passively consume content. Sitting back and watching an instructor go through materials without trying anything yourself is not going to get you very far.

Lesson 4: Don't just memorize.
It's normal to use Google, StackOverFlow.

Lesson 5: Build stuff.
Of course. Learning by doing!

[BTY] Day 4: Read through a MLops case study in real scenario

Dang Hoang Nhu Nguyen — Fri, 28 Jan 2022 00:44:50 +0000

From business context:

Turn to an ML System as follows:

The data process in a bond scoring ML system

The training and inference pipeline for a bond risk model

I like the wrapping up section of the author:

The best advice I give to companies now is to be proactive about tooling portability. Upgrades to MLOps tooling are going to happen every 18 months to 3 years because there are too many stages of MLOps for one tool to be the jack of all trades and the best in breed practices are evolving. Being able to quickly adapt to the right ensemble of tools for your team and your organization will accelerate the adoption of ML in your organization, and ensure ML can provide convincing value to stakeholders.

Read it here: https://alexchung1.medium.com/case-study-of-mlops-in-a-hedge-fund-from-zero-to-30m-f524b05788ff

[BTY] Day 3: Tabby - A terminal for the modern age

Dang Hoang Nhu Nguyen — Thu, 27 Jan 2022 04:45:09 +0000

As its introduction,

"Tabby is an infinitely customizable cross-platform terminal app for local shells, serial, SSH and Telnet connections."

I'm using TMUX with multiple sessions, windows, panels setup. However, handling terminal windows in my local machine is intimidating. Tabby came to me like a cure!

Check it out: https://tabby.sh/