DEV Community: Mritunjay Sharma

Introduction to Thanos!

Mritunjay Sharma — Fri, 02 Oct 2020 18:04:07 +0000

You can hate me for this but I am not an Avengers Series fan and I am not going to introduce you to the Thanos we meet there 😛

Which Thanos are we going to talk about then? And do you really need to know anything as pre-requisites to understand what you will be learning in this blog? Nope :)

Before Thanos, let's discuss Monitoring

There's this interesting saying-

"Running a product without any monitoring is like NOT running the product at all"

Yes, that's right! Before we discuss 'what', let's discuss 'why'!

Let's make you run a Startup

Now, let us assume that you are reading this blog and you are running a startup that employs multiple applications and is being used by thousands of users. Wow, already feeling amazing? 😎 Yes, you should but let's see one fine day - one of your services breaks and multiple users start facing some kind of outage? (I pray that it doesn't happen with you on weekend) What will happen then? Well, I don't know exactly but it's a possibility that the game of pressures will surely begin which you will transfer to the Engineering Team and the Team will be passing it to you and you just can't see hell breaking loose everywhere whether it's your team or it's the users.

So what can you do to avoid this kind of situation or at least deal better? Yes, you are right - Monitoring!
Won't the situation become really better if you know it in advance about the errors that can come up with the users? You and your engineering team will love it, in fact!
Monitoring is therefore very important. It not only helps you get alerted but also helps you observe trends that can drive your decisions, low-level debugging, gaining insights and automation as well.

That's all about Monitoring - but, hey, who will help you monitor your applications?

Prometheus: Anyone said Monitoring?

You all wondering this blog is about Thanos but we are discussing Monitoring, Startup and now even Prometheus?

Relax, relax, relax. We are going to come to Thanos but we need to discuss all this before meeting Thanos. You know meeting Thanos is not easy, right? Prometheus says you meet me and it's a promise I will let you meet Thanos!

Prometheus is your leading open-source monitoring and alerting solution and we are going to discuss in brief what it does and what are the certain limitations that led to Thanos?

Prometheus in itself is a very vast solution to understand and this blog being beginner-friendly will not try to go into jargons. In plain simple words - Prometheus tries to focus on one thing and it does that extremely well. It offers a data model and a query language that helps you analyse how your applications are performing.

So far so good but now we discuss few limitations in Prometheus that led to Thanos:

Imagine the petabytes of data being accumulated? If we just used Prometheus, at best, we can have retention of one week of data - What if the Engineering team needs data of a year back? Prometheus, in itself, was not enough for the Retention of data.
Also, if we are trying to retain the data, can it be done in a cost-effective way? What about the sacrifice that we have to make with responsive query times?
Suppose, different Prometheus servers are up and you want metrics from a single query API? This again became a strong reason behind the birth of Thanos.
One important feature of Prometheus is its High Availability(HA) and it's great but you see that if we get the capability to merge the replicated data collected via its HA setups?

All these problems led to the birth of Thanos!

Thanos

Let's talk about Thanos now! It began at Improbable, a London-based company, by Bartek Plotka and Fabian Reinartz, and is now a full-fledged Open Source Project following the Cloud Native Computing Foundation Governance structure.

So, what does Thanos does? Yes, it solves the above problems but how? Let's figure it out by discussing Thanos architecture:

Solving Global Query View + HA: If we talk in simple words - Thanos is built on top of Prometheus and to get a global view on top of an existing Prometheus setup, we need to interconnect it.

The Thanos Sidecar component does exactly what a side-car means. It is deployed next to each running Prometheus server and acts as a proxy.

Thanos Side-Car component (Pic Source: https://improbable.io/blog/thanos-prometheus-at-scale)

Retention of historical Data using Object Data Storage Services (Like S3, Azure, GCP, etc)

Thanos using Object Data Storage Services (Pic Source: https://improbable.io/blog/thanos-prometheus-at-scale)

Downsampling: Yes, the historical data can be huge and this also means that you need to worry about the complexity of algorithms as it will affect the query times. Thanos thus uses the technique of Downsampling (A process that reduces the sampling rate of signal). Thanos will help Prometheus achieve this which Prometheus alone cannot do.

Thanos Downsampling (Pic Source: https://improbable.io/blog/thanos-prometheus-at-scale)

Ruler class: Recording rules is important to reduce cost, latency and complexity of the monitoring stack. Prometheus although offers it in itself but there are few cases where it has limitations like:
- alerting when two or more clusters are down.
- Rules that are not in the scope of Prometheus Data Retention.
- The case of storing all the rules and alerts at a single place.

To solve all these cases, Thanos offers a separate component called Ruler that evaluates rules and alerts of Thanos queries.

Thanos Ruler Component (Pic Source: https://improbable.io/blog/thanos-prometheus-at-scale)

So this was a basic introduction that I tried to give you all about Thanos and I hope that this introduction will invoke interests in you to dive deeper in Thanos.

I am attaching links of a few excellent blogs, tutorials and videos that acted as a reference for me and will help you as well further:

Links to Blogs

Links to Videos

Link to Interactive Tutorial (No external setup required)

https://www.katacoda.com/thanos/courses/thanos

PS:

Thank you so much for reading the blog!😃

Being a beginner myself with Thanos, I thought learning anything new will be better by writing about it and teaching it to others ✨

However, there is a possibility that I might have made inadvertent errors or missed something.

Please feel free to improve the blog by posting your feedback or you can reach out to me on Twitter 💫 (https://twitter.com/mritunjay394)

Using GitHub Actions to Automate Image Classification with Microsoft Azure Custom Vision Services

Mritunjay Sharma — Thu, 17 Sep 2020 19:09:36 +0000

My Workflow

I created this new GitHub Action available here with which we can automate the entire process of image classification with Azure Custom Vision.
With this - all a user needs to do is:

Add the 4 SECRETS which are basically Azure Custom Vision Services Credentials. The process is explained in the README
Upload Image Dataset in the GitHub Repo and modify the tags, tagsVar and trainSize accordingly in the below yml file.

Here's a sample workflow YML file using this Action:

name: Auto Custom Vision Classifier
on:
  push:
    paths:
    - '**.yml'
jobs:
  build_model:
    runs-on: ubuntu-latest
    steps:
    - name: Automating Image Classification with Microsoft Azure Custom Vision Training and Prediction
      uses: mritunjaysharma394/autoCustomVision@v1.0
      with:
        tags: "[Hemlock,Japanese Cherry]" # Rename it according to the folder name under images/ which will also be our name to the tags
        tagsVar: "[hemlock_,japanese_cherry_]" # Rename it according to the symmetry of file names under images/tag/ 
        trainSize: "10" # Train Size of Each Tag
        endpoint: ${{ secrets.AZURE_ENDPOINT }}
        trainingKey: ${{ secrets.AZURE_TRAINING_KEY }}
        predictionKey: ${{ secrets.AZURE_PREDICTION_KEY }}
        predictionResourceid: ${{ secrets.AZURE_PREDICTION_RESOURCE_ID }}

Submission Category:

DIY Deployments

Yaml File or Link to Code

Link to GitHub Repo of the Action and the Workflow:

mritunjaysharma394 / autoCustomVision

GitHub Action automating the entire process of image classification with Microsoft Azure Custom Vision Service

Automating Image Classification with Microsoft Azure Custom Vision Training and Prediction

This GitHub Action automates the entire process of image classification with Microsoft Azure Custom Vision Service.

How to use

Step 1:

To use the Custom Vision Service you will need to create Custom Vision Training and Prediction resources in Azure. To do so in the Azure portal, fill out the dialog window on the Create Custom Vision page to create both a Training and Prediction resource.

Step 2:

Get the Endpoint, Training Key, Prediction Key and Prediction Resource ID credentials andd save it as Secrets in your GitHub repository settings. You can find the items at the Custom Vision website . Sign in with the account associated with the Azure account you used to create your Custom Vision resources. On the home page (the page with the option to add a new project), select the gear icon in the…

View on GitHub

Additional Resources / Info

Working Video Example:

Mritunjay Sharma

@mritunjay394

So submitted my second GitHub Actions and it’s workflow for the #ActionsHackathon 🥳💃

What it does is absolutely magic by Automating the entire Image Classification process with #Azure Custom Vision.

All you need is a 'push'🤓

PS:Thank you so much @HaimantikaM for the video

19:22 PM - 17 Sep 2020

2 7

Project where it is being used:

Haimantika / test_autocustomvision

Automating Python Code Formatting using GitHub Actions

Mritunjay Sharma — Tue, 08 Sep 2020 20:40:29 +0000

My Workflow

It is the first time that I have built a GitHub action and I am really thankful to this Hackathon for making me learn amazing new things 😇 !

The action that I have made is known as autoyapf. It is a GitHub action for yapf, an open-source tool that automatically formats Python code to conform to the PEP 8 style guide.

I have created a simple workflow that automates Python code formatting whenever 'push' event triggers. In simple words - every time the maintainer pushes the code from their own playground or merges a Pull Request from a contributor it will be formatted automatically with the PEP 8 Style guidelines using the yapf tool. This action is a win-win situation for both contributors and maintainers of projects involving Python!

Sample Workflow yaml File:


name: Formatting python code
on:
  push:
    paths:
    - '**.py'
jobs:
  autoyapf:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@master
        with:
          ref: ${{ github.head_ref }}
      - name: autoyapf
        id: autoyapf
        uses: mritunjaysharma394/autoyapf@v2
        with:
          args: --style pep8 --recursive --in-place .
      - name: Check for modified files
        id: git-check
        run: echo ::set-output name=modified::$(if git diff-index --quiet HEAD --; then echo "false"; else echo "true"; fi)
      - name: Push changes
        if: steps.git-check.outputs.modified == 'true'
        run: |
          git config --global user.name 'Mritunjay Sharma'
          git config --global user.email 'mritunjaysharma394@users.noreply.github.com'
          git remote set-url origin https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/${{ github.repository }}
          git commit -am "Automated autoyapf fixes"
          git push

The autoyapf GitHub is being used in my own another Open Source Python project with the name sedpy

I have written a detailed blog explaining how the autoyapf GitHub action is being used in the above project which can be found below:

How to automate code formatting for Python projects with GitHub Actions - A study

Mritunjay Sharma ・ Sep 8 ・ 5 min read

#actionshackathon #python #codequality #github

Submission Category:

Maintainer Must-Haves

Yaml File or Link to Code

mritunjaysharma394 / autoyapf

Automating yapf formatting.

autoyapf

Worried whether your Python code is following the style guidelines or not? Well, you don't need to do that now 😄

autoyapf is a GitHub action for yapf, an open-source tool that automatically formats Python code to conform to the PEP 8 style guide It is in essence, the algorithm that takes the code and reformats it to the best formatting that conforms to the style guide, even if the original code didn't violate the style guide.

This action is designed to be used in conjunction with the 'push' trigger. In simple words - everytime the maintainer pushes the code from their own playground or merges a Pull Request from a contributor it will be formatted automatically with the PEP 8 Style guidelines using the yapf tool. This action is a win-win situation for both contributors and maintainers of projects involving Python! Yes, you heard it right!

This…

View on GitHub

Additional Resources / Info

[Note:] # (Be sure to link to any open source projects that are using your workflow!)

The workflow is being used in:

mritunjaysharma394 / sedpy

Building a cross-platform alternative of sed (stream editor)

sedpy

[

sedpy is an open-source project being built as a cross-platform alternative of sed for easier and more flexible stream line editing across BSD and GNU systems.

What is sed and what is sedpy?

The sed command, short for stream editor, performs editing operations on text coming from standard input or a file. sed edits line-by-line and in a non-interactive way.

This means that you make all of the editing decisions as you are calling the command, and sed executes the directions automatically. This may seem confusing or unintuitive, but it is a very powerful and fast way to transform text, especially as part of a script or automated workflow.

The problem with sed is that it's not purely cross-platform. Linux uses GNU-based sed while macOS has the BSD version of sed.As a result - both have quite important syntactic and usage difference. At the same time, sed…

View on GitHub
NAMANJAIN2606 / clean-up

It is also being considered to be used in RTEMS:

RTEMS / rtems-source-builder

Build tools and packages from source.

RTEMS Tools From Source
-----------------------
The RTEMS Source Builder is a tool to aid building packages from source used by
the RTEMS project. It helps consolidate the details you need to build a package
from source in a controlled and verifiable way. The tool is aimed at developers
of software who use tool sets for embedded type development and is not limited
to building just for RTEMS. Embedded development typically uses cross-compiling
tool chains, debuggers, and debugging aids. Together we call these a 'tool
set'. The RTEMS Source Builder is not limited to this role but designed to fit
with-in this specific niche. It can be used outside of the RTEMS project and we
welcome this happening in other open source or commercial projects
The project is part of the RTEMS Project. The project's websites are
 RTEMS Project Website:
   https://www.rtems.org/

 GIT Source Repository:
   https://git.rtems.org/rtems-source-builder.git/

 Documentation:
   https://docs.rtems.org/branches/master/rsb/index.html

 Bugs:
   https://devel.rtems.org/query?component=RSB

Please refer

…

View on GitHub

Thank you so much for your valuable time going through this!

Please free to share feedbacks and suggestions for improvements! 💃

How to automate code formatting for Python projects with GitHub Actions - A study

Mritunjay Sharma — Tue, 08 Sep 2020 19:36:01 +0000

This is a beginner-friendly post that will tell you how easy it will become to conform to Python coding styles using autoyapf GitHub Action in your Python projects!

Introduction

To begin with - let's discuss first what are GitHub Actions and what magic they bring 😃?

GitHub Actions

GitHub Actions is that piece of magic which enables you to automate, customize, and execute your software development workflows right in your repository.

But hey? What are words without actions? And so what are actions without workflows? Yes, you got it right! For GitHub Actions to work you need to use Workflows but what are Workflows?
A workflow is a configurable automated process made up of one or more jobs. Now, these jobs can be like the accounting of the trigger for the GitHub action to work - like push on the main branch.

Workflow files use YAML syntax and must have either a .yml or .yaml file extension.

For GitHub Actions to work you must store workflow files in the .github/workflows directory of your repository.

Brief Introduction to building a GitHub Action

Note: In order to use an existing GitHub action all you require is just a Workflow YAML file in .github/workflows. The files mentioned in this part of the blog are required when you start building your own GitHub Actions 😄

In order to make almost any GitHub Action - you require at least these files:

Dockerfile — GitHub actions are of three types - Docker actions, Javascript actions and composite run steps action. In our case- we are going to use Docker Action and so Dockerfile is used to define the container environment that runs the action.
action.yml — This will define inputs, outputs, Entrypoint for our action, and other details like name of the action, description, author name, etc.
script — An action might require a script to perform a task, in our case we just need an Entrypoint script named as entrypoint.sh
README.md — This is optional, but if we want to publish our action on the Marketplace we need it. README.md file is useful to provide examples that are needed for the action to run without any hassles.

This is the basic skeleton of how the GitHub actions are built. If you want to start with making your own first action, check out this official Hello World tutorial

Let's move on to the main part of preparing and setting up the workflow for the autoyapf GitHub Action we are going to use.

What is autoyapf?

autoyapf is a GitHub action for yapf, an open-source tool that automatically formats Python code to conform to the PEP 8 style guide. It is in essence, the algorithm that takes the code and reformats it to the best formatting that conforms to the style guide, even if the original code didn't violate the style guide.

This action is designed to be used in conjunction with the 'push' trigger. In simple words - everytime the maintainer pushes the code from their own playground or merges a Pull Request from a contributor it will be formatted automatically with the PEP 8 Style guidelines using the yapf tool. This action is a win-win situation for both contributors and maintainers of projects involving Python!

Choosing the Project

The first step is always choosing the repository where you will be using the GitHub action.
Since the GitHub action is made for python projects, I am going to use an open-source Python project that I created recently called sedpy

mritunjaysharma394 / sedpy

Building a cross-platform alternative of sed (stream editor)

sedpy

[

sedpy is an open-source project being built as a cross-platform alternative of sed for easier and more flexible stream line editing across BSD and GNU systems.

What is sed and what is sedpy?

The sed command, short for stream editor, performs editing operations on text coming from standard input or a file. sed edits line-by-line and in a non-interactive way.

This means that you make all of the editing decisions as you are calling the command, and sed executes the directions automatically. This may seem confusing or unintuitive, but it is a very powerful and fast way to transform text, especially as part of a script or automated workflow.

The problem with sed is that it's not purely cross-platform. Linux uses GNU-based sed while macOS has the BSD version of sed.As a result - both have quite important syntactic and usage difference. At the same time, sed…

View on GitHub

Preparing the Workflow

We'll go to our repository and click on the Actions tab. That will then reflect an option like set up a workflow yourself. Click it.

This will create a YAML file in the path:
.github/workflows/main.yml

You can rename the file to whatever you prefer like, in this case, I will give it the name autoyapf.yml instead of main.yml.

Now, let's dive in the autoyapf.yml file:

name: Formatting python code
on:
  push:
    paths:
    - '**.py'

What we have done until here is:

Given a name to the workflow: Formatting python code
Defined what event will cause the workflow to trigger. In this case, it is triggered when the python file is pushed in the repository.

Next step is to define the jobs, further, we added:

jobs:
  autoyapf:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@master
        with:
          ref: ${{ github.head_ref }}
      - name: autoyapf
        id: autoyapf
        uses: mritunjaysharma394/autoyapf@v2
        with:
          args: --style pep8 --recursive --in-place .
      - name: Check for modified files
        id: git-check
        run: echo ::set-output name=modified::$(if git diff-index --quiet HEAD --; then echo "false"; else echo "true"; fi)
      - name: Push changes
        if: steps.git-check.outputs.modified == 'true'
        run: |
          git config --global user.name 'Mritunjay Sharma'
          git config --global user.email 'mritunjaysharma394@users.noreply.github.com'
          git remote set-url origin https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/${{ github.repository }}
          git commit -am "Automated autoyapf fixes"
          git push

Here's a brief detail about almost everything the above snippet does:

The jobs that the workflow has to perform will run on the GitHub hosted runner which is ubuntu-latest in our case.

jobs:
  autoyapf:
    runs-on: ubuntu-latest

We define the steps of jobs next.
- The first step checks out the repository where we want to workflow to execute. Since I am using it inside sedpy, it will check out this repository's master branch

steps:
      - uses: actions/checkout@master
        with:
          ref: ${{ github.head_ref }}

In the next step - we checked out the GitHub action autoyapf which will perform all the formatting using yapf tool with the arguments supplied as: args: --style pep8 --recursive --in-place .

- name: autoyapf
        id: autoyapf
        uses: mritunjaysharma394/autoyapf@v2
        with:
          args: --style pep8 --recursive --in-place .

This step checks if the above step has made any changes in the code. If it has made any changes then the next step will push the changes to the sedpy repository.

- name: Check for modified files
        id: git-check
        run: echo ::set-output name=modified::$(if git diff-index --quiet HEAD --; then echo "false"; else echo "true"; fi)

This will check the condition as detailed above and we will push the changes using run utility that makes us use terminal-like commands in the .yml files.

      - name: Push changes
        if: steps.git-check.outputs.modified == 'true'
        run: |
          git config --global user.name 'Mritunjay Sharma'
          git config --global user.email 'mritunjaysharma394@users.noreply.github.com'
          git remote set-url origin https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/${{ github.repository }}
          git commit -am "Automated autoyapf fixes"
          git push

Woohoo! 💃 We have completed our workflow autoyapf.yml and the complete file looks something like this:


name: Formatting python code
on:
  push:
    paths:
    - '**.py'
jobs:
  autoyapf:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@master
        with:
          ref: ${{ github.head_ref }}
      - name: autoyapf
        id: autoyapf
        uses: mritunjaysharma394/autoyapf@v2
        with:
          args: --style pep8 --recursive --in-place .
      - name: Check for modified files
        id: git-check
        run: echo ::set-output name=modified::$(if git diff-index --quiet HEAD --; then echo "false"; else echo "true"; fi)
      - name: Push changes
        if: steps.git-check.outputs.modified == 'true'
        run: |
          git config --global user.name 'Mritunjay Sharma'
          git config --global user.email 'mritunjaysharma394@users.noreply.github.com'
          git remote set-url origin https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/${{ github.repository }}
          git commit -am "Automated autoyapf fixes"
          git push

See the below working example to see this GitHub action in action

If you have liked this action, please feel free to leave a star and yes it will be a blessing if you can contribute to it as well! Link below:

mritunjaysharma394 / autoyapf

Automating yapf formatting.

autoyapf

Worried whether your Python code is following the style guidelines or not? Well, you don't need to do that now 😄

This…

View on GitHub

I hope this blog helped you how to start with creating workflows for GitHub actions (in our case: autoyapf) and you have learned something amazing! If you have doubts, suggestions or feedbacks, please feel free to ping me here:https://twitter.com/mritunjay394 !

Thank you so much for taking out time to read this blog! Sending you all the good vibes ✨

DEV Community: Mritunjay Sharma

Introduction to Thanos!

Before Thanos, let's discuss Monitoring

Let's make you run a Startup

Prometheus: Anyone said Monitoring?

Thanos

Links to Blogs

Links to Videos

Link to Interactive Tutorial (No external setup required)

Using GitHub Actions to Automate Image Classification with Microsoft Azure Custom Vision Services

My Workflow

Submission Category:

Yaml File or Link to Code

mritunjaysharma394 / autoCustomVision

GitHub Action automating the entire process of image classification with Microsoft Azure Custom Vision Service

Automating Image Classification with Microsoft Azure Custom Vision Training and Prediction

How to use

Step 1:

Step 2:

Additional Resources / Info

Working Video Example:

Project where it is being used:

Haimantika / test_autocustomvision

Automating Python Code Formatting using GitHub Actions

My Workflow

How to automate code formatting for Python projects with GitHub Actions - A study

Mritunjay Sharma ・ Sep 8 ・ 5 min read

Submission Category:

Yaml File or Link to Code

mritunjaysharma394 / autoyapf

Automating yapf formatting.

autoyapf

Additional Resources / Info

mritunjaysharma394 / sedpy

Building a cross-platform alternative of sed (stream editor)

sedpy

What is sed and what is sedpy?

NAMANJAIN2606 / clean-up

RTEMS / rtems-source-builder

Build tools and packages from source.

How to automate code formatting for Python projects with GitHub Actions - A study

Introduction

GitHub Actions

Brief Introduction to building a GitHub Action

What is autoyapf?

Choosing the Project

mritunjaysharma394 / sedpy

Building a cross-platform alternative of sed (stream editor)

sedpy

What is sed and what is sedpy?

Preparing the Workflow

mritunjaysharma394 / autoyapf

Automating yapf formatting.

autoyapf