DEV Community: Athul Cyriac

Harnessing Headers for Analytics

Athul Cyriac — Wed, 02 Dec 2020 05:03:58 +0000

Recent days, I have a hard time going to sleep. I have no idea why but it may be because I wake up late or sit in front of my computer all day long. Anyways this sleeplessness paved way for quite a handful of ideas and the most recent one was an Analytics Server.
I was heavily inspired by Shynet for its simplicity and aesthetics and I had been using it for getting the analytics of this Blog and a few Personal Projects. Shynet was built with Django which again piqued my interest.
I had the usual NIH-ish thought, why not build one. Why borrow when you can create. I only wanted a barebones Analytics Server with the main outputs like how man URL hits do I receive on the site, what are the referrers, which devices are being used to access and so on...
And from that sleeplessness idea, I tried my hand on it the next day and I would say I succeeded. I also told in https://lobste.rs that I am going to build one over the weekend and I did.

The core is hacky but it gets the job done.

You can find the whole source code here.

Starting to start

I tend to work on my side projects between my Google Meet classes and I tend to be lazy most of the time. Reduce the complexity, reduce the dependency and reduce unwanted additions. This is a new way of working for my projects and before I used to implement a new feature if I was in a good mood and worldly surroundings were okay. Now I follow my project guidelines, which I had jotted down.

Initially, I googled for the same and an old 2014 blog post came up. It was in Python but with Flask and gave me an idea on how it should work. It gave me a lot of insights on how everything should be glued together to work with and how to use the Js side of things effectively.

FastAPI was in my mind because I could host that and use it for some sites of mine with ease. I hacked up the basic working server in a few hours and tried to get some data from it. All this data needed to be stored somewhere and I used deta.sh's Base which is a NoSQL DB with an easy API for working with Python.

The base working model was barebones and didn't provide many insights on anything. It did as it was told to do, nothing more nothing less.

Goldmine of Sorts.

To get more insights into the working of Shynet, I found out that the HTTP-headers were used to get all the data. An HTTP-Header response contains almost every needed data to make something like this. Imagine the data being sent to any server for every page load, it may be hard to comprehend but it's a lot of data.

{
  "accept": "image/webp,image/apng,image/*,*/*;q=0.8",
  "accept-encoding": "gzip, deflate, br",
  "accept-language": "en-IN,en-GB;q=0.9,en-US;q=0.8,en;q=0.7",
  "referer": "https://athul.github.io/",
  "sec-fetch-dest": "image",
  "sec-fetch-mode": "no-cors",
  "sec-fetch-site": "cross-site",
  "user-agent": "Mozilla/5.0 (Linux; Android 9; LLD-AL10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Mobile Safari/537.36",
  "x-amzn-trace-id": "Root=<some_trace_id>",
  "x-forwarded-for": "2.3.4.5",
  "x-forwarded-host": "<host_url>",
  "x-forwarded-port": "443",
  "x-forwarded-proto": "https",
  "x-original-forwarded-for": "1.2.3.4",
  "x-real-ip": "1.2.3.4",
  "x-request-id": "1234gasdjhgi",
  "x-scheme": "https"
}

This is a sample header request from my website to the Analytics server. See, we can get many details from these if we've got the right tools. There is a site called https://ipapi.co/ and we can get quite a handful of details from them wit just the IP address, we could get data like the network provider, the location of the User logged in and so on...

These are some Gems I learned from Reading Shynet's Source code. Shynet uses a DB called MaxMind GeoIP DB.

Okay, now I can use the Data. But where would I see all these? I could go to Deta's DB explorer and Read all these as JSON and get the idea. But what's the point in reading JSON and understanding JSON, without getting any meaningful insights??

Processing the Received Data

Once the data is received to the DB, to get some meaningful insights, had to process it a bit further. The data had to be sorted and provided and should be sent to the Frontend.
This was the first time I learned about and used the Sorted and Counter functions of Python. To get the Device details from the 'user-agent' key in headers, I used the user-agent package in Python. All this data would be used to render the Graphs and Tables in the Frontend.

API or Templates?

FastAPI is widely used for its API capabilities using async requests and all. But FastAPI is built on top of Starlette which is a cool web framework. It not only supports REST API's but also supports templating with Jinja2. FastAPI does this too. I had been building toy APIs for a while now, and creating an API for transferring the data and doing stuff, but I felt like this should be kept private and be not accessible from Prying hands. So with no prior experience with Jinja2 templating, I finalized on using templates.

Using templates made it quite a lot easier to manage and work on. It was daunting at first but I later picked up the pace and made my way through it.

Jinja2 has many good features that you could work on and use. It works on Js files as well. It can be picked easily with some trial and error. The docs are also quite good except for the filters part since it doesn't have many examples. Filters are another cool feature of Jinja2.

Charts for the Win !!

Okay so templating is fixed and I opted for Tailwind CSS for frontend styling and Chart.js for the Graphs, but I later shifted to Frappe Charts because I wanted to try it out and that was quite a good match. Frappe charts have an easier API and small size compared to Chart.js and for a Js noob like me, it was quite easy.
Using templates inside the Js file also boosted the development time. Using graphs/charts also improved aesthetics and made more sense of the Data. The only tradeoff was that it increased page load time.

Current Features

Easily Deployable on Deta Micros
Small Js Code. It tracks the PageLoad time and current URL
Graphs and Insights
Country and Network Provider and location info

I know many features are missing, I would implement them if I get the time and if anyone requests them.

I'm also planning on writing on a Tutorial for building the same from scratch

Screenshots

Welcome Page

Graphs on Hours and Devices

Visited URLs and referrers

Device and user agents

All Sessions

Per Session Page

If you found this useful, consider donating me on BMC ☕️ or Paypal and can reach out to me on Twitter 😄

FastAPI Auth + Login Page

Athul Cyriac — Sat, 21 Nov 2020 14:34:47 +0000

Photo by Rod Long on Unsplash

This past week I had been working on something(will release it in the coming days) which used FastAPI for the Backend and Jinja for Templating. I had never used Jinja before and it was quite useful and easy to use. The abstraction is enjoyable. No need for an API to send the data to the frontend. Then I had to implement an Authentication system for the frontend since the project consists of some sensitive info. I had to refresh my HTML form skills for the same and I found that the resources available for these were less.

I'm assuming that you know Python and you have used FastAPI at some point. You can check my other blog post about FastAPI and Deta to build a mailing list server here

Auth

FastAPI has an excellent auth system but that being said it's hard to implement everything if you're on a schedule. I had searched on GitHub for some helper libs and found the perfect and easier one. It supports cookie auth too 😍. It's called fastapi_login and it made the Auth part a lot easier. I'll be using fastapi_login for implementing the login/auth with 🍪.

To Install fastapi_login, you can just,

$ pip install fastapi_login

You might also need to install the Form handling package of FastAPI, python-multipart using pip too.

I'm also assuming you have a file which uses FastAPI for routing and all, I'm calling that main.py and in that main.py you can just call it using the import statement of Python. We'll only implement the Auth side of things in the file. We'll import a builtin FastAPI form class for Auth, a Response type, status code class and a function from the same.

# main.py
from fastapi import Depends,status # Assuming you have the FastAPI class for routing
from fastapi.responses import RedirectResponse,HTMLResponse
from fastapi.security import OAuth2PasswordRequestForm
from fastapi_login import LoginManager #Loginmanager Class
from fastapi_login.exceptions import InvalidCredentialsException #Exception class

SECRET = "secret-key"
# To obtain a suitable secret key you can run | import os; print(os.urandom(24).hex())

manager = LoginManager(SECRET,tokenUrl="/auth/login",use_cookie=True)
manager.cookie_name = "some-name"

the tokenUrl should be our auth endpoint and this route will be implemented later. We also need a DB for the user data(username and password) and for example purposes, I'll create a python dictionary with the username and unhashed password.

The unhashed password method should not be used and you should use an encrypting lib like bcrypt for creating a hashed password

In our endpoint, we'll use this DB to fetch the user if the password is correct or not and if successful we'll Redirect the user to another route on the server itself. Like a Dashboard or something.

Continuing on main.py

DB = {"username":{"password":"qwertyuiop"}} # unhashed

@manager.user_loader
def load_user(username:str):
    user = DB.get(username)
    return user

@app.post("/auth/login")
def login(data: OAuth2PasswordRequestForm = Depends()):
    username = data.username
    password = data.password
    user = load_user(username)
    if not user:
        raise InvalidCredentialsException
    elif password != user['password']:
        raise InvalidCredentialsException
    access_token = manager.create_access_token(
        data={"sub":username}
    )
    resp = RedirectResponse(url="/private",status_code=status.HTTP_302_FOUND)
    manager.set_cookie(resp,access_token)
    return resp

@app.get("/private")
def getPrivateendpoint(_=Depends(manager))
    return "You are an authentciated user"

The decorator @manager.user_loader will use the function load_user to check whether the user exists in the DB. Next, we define our endpoint called /auth/login if you recall correctly this is the same URL we used for the manager object. If the user is not identified we'll throw the InvalidCredentialsException exception. If the authentication was successful then it will redirect to the specified URL in our RedirectResponse class. If you're redirecting to a GET endpoint make sure to add the status code too. Once we set our response class we set the cookie with response class and access token.

This is enough for the Auth part. You can check out the repository for docs and this gist by the author of fastapi_login for more details.

Login Form

FastAPI has a builtin Template class with Jinja2. here we won't be using Jinja since we don't have any data from the backend to be used up front. It's totally upto you if you want to use it. Here is the code,

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Login</title>
    <link rel="stylesheet" href="https://unpkg.com/twinklecss@1.1.0/twinkle.min.css"/>
</head>
<body>
    <div class="flex p-4 m-6 justify-center">
        <form class="bg-white shadow-md rounded px-8 pt-6 pb-8 mb-4" method="POST" action="/auth/login" >
          <div class="mb-4">
            <label class="block text-gray-700 text-sm font-bold mb-2" for="username">
              Username
            </label>
            <input class="shadow appearance-none border rounded w-full py-2 px-3 text-gray-700 leading-tight focus:outline-none focus:shadow-outline" id="username" name="username" type="text">
          </div>
          <div class="mb-6">
            <label class="block text-gray-700 text-sm font-bold mb-2" for="password">
              Password
            </label>
            <input class="shadow appearance-none border rounded w-full py-2 px-3 text-gray-700 mb-3 leading-tight focus:outline-none focus:shadow-outline" id="password" name="password" type="password">
          </div>
          <div class="flex items-center justify-between">
            <button type="submit" class="bg-blue-500 hover:bg-blue-700 text-white font-bold py-2 px-4 rounded focus:outline-none focus:shadow-outline">
              Sign In
            </button>
          </div>
        </form>
      </div>
</body>
</html>

I had used Twinkle CSS which is a size-friendly build of Tailwind CSS but you can use any CSS framework of your choice. You can see that the form attribute action is set to the /auth/login endpoint which is our route which handles the auth part. So when we click the "Sign In" button we'll be taken to the /private route if successful. Adding this to FastAPI is quite easy. I had named the above file as login.html inside a templates folder. We'll use HTMLResponse class to return the HTML file and render that in the browser. This will be the additional code to main.py.

import path

pth = path.dirname(__file__)

@app.get("/",response_class=HTMLResponse)
def loginwithCreds(request:Request):
    with open(path.join(pth, "templates/login.html")) as f:
        return HTMLResponse(content=f.read())

This will render the login screen on the / endpoint. Here is a working Demo

Here is the Full Source Code for the Python side of things

from fastapi import FastAPI,Depends,status
from fastapi.responses import RedirectResponse,HTMLResponse
from fastapi.security import OAuth2PasswordRequestForm
from fastapi_login import LoginManager #Loginmanager Class
from fastapi_login.exceptions import InvalidCredentialsException #Exception class
import path

app= FastAPI()

SECRET = "secret-key"
# To obtain a suitable secret key you can run | import os; print(os.urandom(24).hex())
pth = path.dirname(__file__)
templates = Jinja2Templates(directory=path.join(pth, "templates"))

manager = LoginManager(SECRET,tokenUrl="/auth/login",use_cookie=True)
manager.cookie_name = "some-name"

DB = {"username":{"password":"qwertyuiop"}} # unhashed

@manager.user_loader
def load_user(username:str):
    user = DB.get(username)
    return user

@app.post("/auth/login")
def login(data: OAuth2PasswordRequestForm = Depends()):
    username = data.username
    password = data.password
    user = load_user(username)
    if not user:
        raise InvalidCredentialsException
    elif password != user['password']:
        raise InvalidCredentialsException
    access_token = manager.create_access_token(
        data={"sub":username}
    )
    resp = RedirectResponse(url="/private",status_code=status.HTTP_302_FOUND)
    manager.set_cookie(resp,access_token)
    return resp

@app.get("/private")
def getPrivateendpoint(_=Depends(manager))
    return "You are an authentciated user"

@app.get("/",response_class=HTMLResponse)
def loginwithCreds(request:Request):
    with open(path.join(pth, "templates/login.html")) as f:
        return HTMLResponse(content=f.read())

If you found this useful, consider donating me on BMC ☕️ or Paypal and can reach out to me on Twitter 😄

Building a Search for Static Sites made in Go

Athul Cyriac — Tue, 10 Nov 2020 16:33:33 +0000

In these past few months, I had a plan to set up a wiki for myself. I was exploring my options on what tool I should use. This thought came up in September and this was the same time for FOSS United's Hackathon. Post Hackathon discussions on the group was quite interesting and this was where the whole idea of a Digital Garden came up. On further research this seemed a good option for me to organize my notes and ideas. I looked at my Options, There were Roam Research and Obsidian which were quite cool but Roam was paid and Obsidian was an electron app😬. I also wanted to make these Publically available when it was good, so hosting it was also necessary for me so I went on with my searching. Then there was Foam and TiddlyWiki but sadly for me these were less hackable except for tiddlywiki. Then comes zettel which was a project for the said hackathon. In simple terms zettel is a SSG/CLI for markdown files. It supported wikilinks([[]]) and was made with Go 😻. I had been contributing to it's codebase. I had implemented the syntax highlighting and Math Features to it.

Search was a most wanted feature for me since it would reduce my time to look into my articles when I'm would not be in my home(post pandemic). So I tried to implement it

Searching to set up search

Since Zettel was made with Go, I first searched on how to implement a search feature in a Hugo¹ website. The results I got from google was from the official Hugo docs. There were a few commercial services and a few hacked up ones. The fastSearch gist was quite interesting and matched to my use-case. The next step was how I could port that from Hugo to zettel.

Indexing

Further searching got me the idea of how search works. For starters, it requires an Index file with the data on details about what we would be searching. Here is a rough representation for need of search indexes

It's just like the index of a book where you can find what's inside the book by looking at index. The Index file is important because it makes the search faster. It's quite time consuming to scrape the whole website to create the index for use in search and since we don't have a server side to scrape the data, this method becomes a no-go for our case. We want to create an index at build time which is quite a lot easier.

Zettel have this feature of showing a graph about the connected markdown pages. It uses a graph data type to create this and shows in the frontend using sigma.js . The Markdown files are the nodes and the connections are the edges. Sigma JS uses a JSON file to generate in the frontend. This JSON file or graph.json is generated at build time.

So I used the same trick to generate the Index file for search. I created a new structure for the Search Index, it looks like this

type PostData struct {
  Title     string   `json:"title"` // the Title of the page
  Permalink string   `json:"permalink"` // the Permalink/Link of the Page
  Tags      []string `json:"tags"` // the Tags of the page if available
}

At build-time this will generate a json file like this

[
    {
        "title": "Adguard Telegram Bot",
        "permalink": "posts/adbot.html",
        "tags": [
            "go",
            "code",
            "other"
        ]
    },
    {
        "title": "Config for Zettel local dev with Air",
        "permalink": "posts/air-zettel.html",
        "tags": [
            "code",
            "other"
        ]
    }
]

And this will act as the search index for the pages. Next was to work on the frontend with some Js and CSS

Fu'z'ing

After the Index we need to use a Js library to parse the json index and return us the result. fastSearch gist uses Fuze.js so I too used the same. Fuze.js is a fuzzy search library which takes in a JSON index and returns from the JSON index if the item is found as Js Arrays. The code for making the search work was available on the gist but needed some modifications for the current use case. I'm not that familiar with Js but I could read and understand the code. I was able to ahck the code up and made it working.

// Code by Craig Mod under MIT License
// https://gist.github.com/cmod/5410eae147e4318164258742dd053993
var fuse; 
var list = document.getElementById('searchResults');
var first = list.firstChild;
var last = list.lastChild;
var maininput = document.getElementById('searchInput');
var resultsAvailable = false;
var baseURL = window.location.pathname.split("/")[1]
if (baseURL.length === 0 || baseURL.includes("html") || baseURL.includes("posts")) {
    baseURL = window.location.origin
} else {
    baseURL = "/" + baseURL
}
console.log(baseURL)
loadSearch()
document.addEventListener('keydown', function(event) {
  if (event.keyCode == 27) {
      document.activeElement.blur();
      resultsAvailable = false;
      list.style.display="none"
  }
  if (event.keyCode == 40) {
    if (resultsAvailable) {
      event.preventDefault();
      if ( document.activeElement == maininput) { first.focus(); } 
      else if ( document.activeElement == last ) { last.focus(); } 
      else { document.activeElement.parentElement.nextSibling.firstElementChild.focus(); } 
    }
  }
  if (event.keyCode == 38) {
    if (resultsAvailable) {
      event.preventDefault();
      if ( document.activeElement == maininput) { maininput.focus(); } 
      else if ( document.activeElement == first) { maininput.focus(); } 
      else { document.activeElement.parentElement.previousSibling.firstElementChild.focus(); } 
    }
  }
});
document.getElementById("searchInput").onkeyup = function(e) {
    list.style.display=""
  executeSearch(this.value);
}
function fetchJSONFile(path, callback) {
  var httpRequest = new XMLHttpRequest();
  httpRequest.onreadystatechange = function() {
    if (httpRequest.readyState === 4) {
      if (httpRequest.status === 200) {
        var data = JSON.parse(httpRequest.responseText);
          if (callback) callback(data);
      }
    }
  };
  httpRequest.open('GET', path);
  httpRequest.send(); 
}
function loadSearch() { 
  fetchJSONFile(baseURL + "/data/search.json", function(data){
    var options = { 
      shouldSort: true,
      location: 0,
      distance: 100,
      threshold: 0.4,
      minMatchCharLength: 2,
      keys: ['title','tags']
    };
    fuse = new Fuse(data, options); 
  });
}
function executeSearch(term) {
  let results = fuse.search(term); 
  let searchitems = ''; 
  if (results.length === 0) { 
    resultsAvailable = false;
    searchitems = '';
  } else { 
    for (let item in results.slice(0,5)) { 
    const title = results[item].title.replace(new RegExp(term, "gi"), (match) => `<mark class="searchHgl">${match}</mark>`);
    tags = results[item].tags.map((value)=>{
      return value.replace(new RegExp(term, "gi"), (match) => `<mark class="searchHgl">${match}</mark>`);
    })
    searchitems = searchitems + '<li><a href="' + baseURL + '/' + results[item].permalink + '" tabindex="0"><span class="title">' + title + "</span> — " + tags + "</a></li>";
    }
    resultsAvailable = true;
  }
  document.getElementById("searchResults").innerHTML = searchitems;
  if (results.length > 0) {
    first = list.firstChild.firstElementChild; 
    last = list.lastChild.firstElementChild; 
  }
}

I also added some fun highlighting for the resulted search terms using the <mark> tag of HTML(semantic HTML FTW🙌). I modified the Js code from the original one and removed the CMD + / option to start the search. The keys to search from the index are the title and tags. Indexing the contents will add a lot of page load time and the JSON will be of much bigger size when scaled.

The CSS part was a tad tricky since I don't know CSS. I had some help from my friend who over a call made it look better and more compact.

Here is the End Result 👇

I have added the whole code for the JS part and the CSS part as a GitHub Gist 😁 with instructions too. If you're interested in Digital Garden's and Zettel, make sure you checkout

Maggie Appleton's Repo for Digital Gardens
A subreddit for Digital Gardeners
Personal fork of Zettel

If you found it useful, you can donate me on BMC ☕️ or Paypal and can reach out to me on Twitter

Hugo is a static site generator in Go, https://gohugo.io ↩

Telegram Bot with Go and Adguard Statistics

Athul Cyriac — Sun, 11 Oct 2020 14:58:50 +0000

Photo by Christian Wiediger on Unsplash

Recently I had this idea of setting up a network-level adblocker in my Home. But sadly I don't use a fibre/broadband connection 🤷‍♂️ so I thought of setting up one on my laptop. The initial choice was PiHole(obviously) but due to the extremities of setting it up on Mac, I jumped ship to Adguard Home on getting some feedback from friends. It also piqued my interest when I saw Adguard Home was written in Go and the frontend in React rather than the PHP frontend of PiHole. It was easy to set up and running and they give the instructions quite well.

I use Brave as my daily driver, It blocks almost all the ads(even Spotify ads on the web player)

AdBot

No, It's not a bot that sends you ads(😝). It's a one-way bot(as of now) which means I can only receive messages like the statistics of the ads blocked overtime via adguard. I wrote the bot in Go to achieve independency of installing packages and making it run on a Raspberry Pi with a single binary. This binary can be hooked up with a CronJob to send the data over a specified time. So, if I'm not home, I can get the statistics of the number of users on my network and related stuff. If you're unaware of creating a Telegram Bot and getting a chat id, refer Telegram Bot Documentation

I hooked it up with a graph generator to send me an Ascii line graph and a Pie Chart image with the most blocked domains. The whole thing was hacked in 2 hours. It was a fun build. I'll explain the code piece by piece.

The Code

As I told you the code is in Go. For starters, I used my favorite HTTP client Insomnia to check the API endpoints on the Adguard Home's Server. Funnily enough, the only required header for such a request is an agh_session token. This is fixed for a server and you only need to get it once from the server and use it indefinitely for the bot. They also have a "not-so-much" working CSRF header auth. It'll work with a single token the whole time. I used Insomnia's code-generator to bootstrap the bot for receiving the JSON Payload. The only things you'll need will be the Adguard Home installed and the agh_session token. Here is a gif of how you could get one

HTTP request

Here is the generated code from insomnia.

package main

import (
    "encoding/json"
    "fmt"
    "io/ioutil"
    "log"
    "net/http"
)
func main() {

    req, _ := http.NewRequest("GET", "http://127.0.0.1/control/stats", nil)

    req.Header.Add("cookie", "agh_session=<some sha string>")

    res, err := http.DefaultClient.Do(req)
    if err != nil {
        log.Println(err)
    }

    defer res.Body.Close()
    body, _ := ioutil.ReadAll(res.Body)
}

The next step was to unmarshal the JSON to be used in Go.

JSON → Go

The JSON payload received was long since it had the data regarding the blocked domains too. I used a Json to Struct to convert the JSON to its valid struct types. But some JSON arrays were hard for it to comprehend so it made each struct key for each key, which was not the expected one. So I used Go Maps to make it workable. This is the struct type of the JSON payload

// Stats represents the data from Adguard as JSON
type Stats struct {
    ProcessingTime    float64              `json:"avg_processing_time"`
    BlockedFilter     []float64            `json:"blocked_filtering"`
    DNSQueries        []float64            `json:"dns_queries"`
    BlockedFilterNum  float64              `json:"num_blocked_filtering"`
    DNSQueriesNum     float64              `json:"num_dns_queries"`
    PT                float64              `json:"num_replaced_parental"`
    NumSB             float64              `json:"num_replaced_safebrowsing"`
    NumSS             float64              `json:"num_replaced_safesearch"`
    ReplacedPT        []float64            `json:"replaced_parental"`
    ReplacedSB        []float64            `json:"replaced_safebrowsing"`
    TimeUnits         string               `json:"time_units"`
    TopBlockedDomains []map[string]float64 `json:"top_blocked_domains"`
    TopClients        []map[string]float64 `json:"top_clients"`
    TopQueriedDomains []map[string]float64 `json:"top_queried_domains"`
}

So this struct unmarshals the Json to Go readable types. This makes it easier to handle the data wrangling in Go. Here is the code which parses the JSON and uses it.

package main

import (
    "encoding/json"
    "fmt"
    "io/ioutil"
    "log"
    "net/http"
)

// Stats represents the data from Adguard as JSON
type Stats struct {
    ProcessingTime    float64              `json:"avg_processing_time"`
    BlockedFilter     []float64            `json:"blocked_filtering"`
    DNSQueries        []float64            `json:"dns_queries"`
    BlockedFilterNum  float64              `json:"num_blocked_filtering"`
    DNSQueriesNum     float64              `json:"num_dns_queries"`
    PT                float64              `json:"num_replaced_parental"`
    NumSB             float64              `json:"num_replaced_safebrowsing"`
    NumSS             float64              `json:"num_replaced_safesearch"`
    ReplacedPT        []float64            `json:"replaced_parental"`
    ReplacedSB        []float64            `json:"replaced_safebrowsing"`
    TimeUnits         string               `json:"time_units"`
    TopBlockedDomains []map[string]float64 `json:"top_blocked_domains"`
    TopClients        []map[string]float64 `json:"top_clients"`
    TopQueriedDomains []map[string]float64 `json:"top_queried_domains"`
}

func main() {

    req, _ := http.NewRequest("GET", "http://127.0.0.1/control/stats", nil)

    req.Header.Add("cookie", "agh_session=<some sha string>")

    res, err := http.DefaultClient.Do(req)
    if err != nil {
        log.Println(err)
    }

    defer res.Body.Close()
    body, _ := ioutil.ReadAll(res.Body)

    var stats Stats

    if err := json.Unmarshal(body, &stats); err != nil {
        log.Println(err)
    }

Nothing fancy of any type, just plain Go code. The next is to generate the Ascii graph from the data we got.

Ascii 📈

There was a package in Go which generates Ascii graphs, not to reinvent the wheel, I just hooked it up to work with the data received. I reduced it's height to enclose it inside a single message. Here are the code and output of the graph


func (s *Stats) generateGraph(tp string) string {
    defer log.Println("Graph Generated")
    graphof := map[string][]float64{
        "DNS": s.DNSQueries,
        "BLK": s.BlockedFilter,
    }
    caption := map[string]string{
        "DNS": "Number of DNS Queries",
        "BLK": "Number of Blocked Queries",
    }
    graph := asciigraph.Plot(graphof[tp], asciigraph.Height(10), asciigraph.Caption(caption[tp]))
    return graph
}

This is not a function but a method in Go. This depends on the Struct type Stats(specifically the pointer) to work. I also introduced a map to get the data of the specified stuff like no. of DNS queries and the number of blocked queries.

Here is a sample output of the Graphs from the data I received both the DNS Queries and Blocked queries


DNS Query Graph :

 888 ┼╭╮
 799 ┤││               ╭╮   ╭
 710 ┤││               ││   │
 622 ┤│╰╮             ╭╯│   │
 533 ┤│ │ ╭╮          │ │   │
 444 ┤│ │ ││          │ │   │
 355 ┤│ │ │╰╮        ╭╯ │   │
 266 ┤│ │ │ │        │  │   │
 178 ┤│ │╭╯ │        │  │   │
  89 ┤│ ╰╯  │        │  │   │
   0 ┼╯     ╰────────╯  ╰───╯

    Number of DNS Queries

-----

Blocked Graph:

 213 ┼                      ╭
 192 ┤                      │
 170 ┤                 ╭╮   │
 149 ┤╭╮   ╭╮          ││   │
 128 ┤││   ││          ││   │
 106 ┤││   ││          ││   │
  85 ┤││  ╭╯│         ╭╯│   │
  64 ┤││  │ │         │ │   │
  43 ┤│╰╮ │ │         │ │   │
  21 ┤│ │ │ │        ╭╯ │   │
   0 ┼╯ ╰─╯ ╰────────╯  ╰───╯

    Number of Blocked Queries

The graph data is an array of 24 elements which means data for each hour. I don't use my laptop 24 hours so the 0 lines in some places. We call this method from the main function just as we unmarshal the JSON.

Pie 📈

The Pie Chart will be the data of the Top Blocked Domains. I used github.com/wcharczuk/go-chart as the charting library for the pie graph. Initially, it was a bit hard to work with but got the idea of it after some time tinkering with it. It creates an image, unlike the Ascii Chart which is just string. Here is the code for the generating the Pie Chart image.

func (s *Stats) pieGrph() {
    defer log.Println("Pie Graph Generated and Image Send")
    var chartValues []chart.Value
    for i := range s.TopBlockedDomains {
        for k, v := range s.TopBlockedDomains[i] {
            values := chart.Value{Label: fmt.Sprintf("%.f:%s", v, k), Value: v}
            chartValues = append(chartValues, values)
        }
    }
    pie := chart.PieChart{
        Width:  512,
        Height: 512,
        Values: chartValues,
    }
    f, _ := os.Create("output.png")
    defer f.Close()
    pie.Render(chart.PNG, f)

This too is a method. This method creates a 512x512 image and saves it as output.png. This image will be sent to us via telegram. We also call this method from the main function just as we unmarshal the JSON.

Sending the Message

Telegram has an HTTP API which makes it quite easy to send messages via bots. But I used a package in Go which handles the Telegram API quite okay, so I used it. Here is the code for sending the Ascii Graph via telegram,

func (s *Stats) sendTGMessage() {
    defer log.Println("Message Sent...")
    percent := (s.BlockedFilterNum / s.DNSQueriesNum) * 100
    message := fmt.Sprintf("Total DNS Queries : %.f\n\nDNS Queries Blocked : %.f\n\n-----\n\nPercent of Queries Blocked: %.2f%%\n\n-----\n\nDNS Query Graph :\n\n`%s`\n\n-----\n\nBlocked Graph:\n\n`%s`\n", s.DNSQueriesNum, s.BlockedFilterNum, percent, s.generateGraph("DNS"), s.generateGraph("BLK"))
    c := tbot.NewClient("<telegram_bot_token>", http.DefaultClient, "https://api.telegram.org")
    if _, err := c.SendMessage("<telegram_chat_id>", message, tbot.OptParseModeMarkdown); err != nil {
        log.Printf("unable to send message: %v", err)
    }
    s.pieGrph()
    if _, err := c.SendPhotoFile("<telegram_chat_id>", "output.png", tbot.OptCaption("PieGraph of Blocked Domains")); err != nil {
        log.Printf("unable to send image: %v", err)
    }
}

This sends a bit more numerical values which are related to Adguard. It just takes in the image in output.png and sends it to the chat. Here is the output image. The ugly thing is that there is a lot of data than the pie graph can hold so it renders some of the text quite poorly but I don't mind that. Here is the output for the above method.

Total DNS Queries: 5273

DNS Queries Blocked: 927

-----

Per cent of Queries Blocked: 17.58%

-----

DNS Query Graph :

 888 ┼╭╮
 799 ┤││               ╭╮   ╭
 710 ┤││               ││   │
 622 ┤│╰╮             ╭╯│   │
 533 ┤│ │ ╭╮          │ │   │
 444 ┤│ │ ││          │ │   │
 355 ┤│ │ │╰╮        ╭╯ │   │
 266 ┤│ │ │ │        │  │   │
 178 ┤│ │╭╯ │        │  │   │
  89 ┤│ ╰╯  │        │  │   │
   0 ┼╯     ╰────────╯  ╰───╯

    Number of DNS Queries

-----

Blocked Graph:

 213 ┼                      ╭
 192 ┤                      │
 170 ┤                 ╭╮   │
 149 ┤╭╮   ╭╮          ││   │
 128 ┤││   ││          ││   │
 106 ┤││   ││          ││   │
  85 ┤││  ╭╯│         ╭╯│   │
  64 ┤││  │ │         │ │   │
  43 ┤│╰╮ │ │         │ │   │
  21 ┤│ │ │ │        ╭╯ │   │
   0 ┼╯ ╰─╯ ╰────────╯  ╰───╯

    Number of Blocked Queries

So that's the whole bot. Here is the full code, I'll upload it to GitHub later 😁


// main.go

package main

import (
    "encoding/json"
    "fmt"
    "io/ioutil"
    "log"
    "net/http"
    "os"

    "github.com/guptarohit/asciigraph"
    "github.com/wcharczuk/go-chart"
    "github.com/yanzay/tbot"
)

// Stats represents the data from Adguard as JSON
type Stats struct {
    // Average Processing Time for DNS Queries
    ProcessingTime float64 `json:"avg_processing_time"`
    // When the Filters did the Job. The slice consists of 24 elements(24 hours)
    // Each item represents the Blockings of Each Hour
    BlockedFilter []float64 `json:"blocked_filtering"`
    // Number of DNSQueries received by Adguard.
    //The slice consists of 24 elements(24 hours)
    // Each element is the queries received in an hour
    DNSQueries        []float64            `json:"dns_queries"`
    BlockedFilterNum  float64              `json:"num_blocked_filtering"`
    DNSQueriesNum     float64              `json:"num_dns_queries"`
    PT                float64              `json:"num_replaced_parental"`
    NumSB             float64              `json:"num_replaced_safebrowsing"`
    NumSS             float64              `json:"num_replaced_safesearch"`
    ReplacedPT        []float64            `json:"replaced_parental"`
    ReplacedSB        []float64            `json:"replaced_safebrowsing"`
    TimeUnits         string               `json:"time_units"`
    TopBlockedDomains []map[string]float64 `json:"top_blocked_domains"`
    TopClients        []map[string]float64 `json:"top_clients"`
    TopQueriedDomains []map[string]float64 `json:"top_queried_domains"`
}

func main() {

    req, _ := http.NewRequest("GET", "http://127.0.0.1/control/stats", nil)

    req.Header.Add("cookie", "agh_session=<some sha string>")

    res, err := http.DefaultClient.Do(req)
    if err != nil {
        log.Println(err)
    }

    defer res.Body.Close()
    body, _ := ioutil.ReadAll(res.Body)

    var stats Stats

    if err := json.Unmarshal(body, &stats); err != nil {
        log.Println(err)
    }
    stats.sendTGMessage()
    stats.pieGrph()
}
func (s *Stats) generateGraph(tp string) string {
    defer log.Println("Graph Generated")
    graphof := map[string][]float64{
        "DNS": s.DNSQueries,
        "BLK": s.BlockedFilter,
    }
    caption := map[string]string{
        "DNS": "Number of DNS Queries",
        "BLK": "Number of Blocked Queries",
    }
    graph := asciigraph.Plot(graphof[tp], asciigraph.Height(10), asciigraph.Caption(caption[tp]))
    return graph
}

func (s *Stats) sendTGMessage() {
    defer log.Println("Message Sent...")
    percent := (s.BlockedFilterNum / s.DNSQueriesNum) * 100
    message := fmt.Sprintf("Total DNS Queries : %.f\n\nDNS Queries Blocked : %.f\n\n-----\n\nPercent of Queries Blocked: %.2f%%\n\n-----\n\nDNS Query Graph :\n\n`%s`\n\n-----\n\nBlocked Graph:\n\n`%s`\n", s.DNSQueriesNum, s.BlockedFilterNum, percent, s.generateGraph("DNS"), s.generateGraph("BLK"))
    c := tbot.NewClient("<telegram_bot_token>", http.DefaultClient, "https://api.telegram.org")
    if _, err := c.SendMessage("<telegram_chat_id>", message, tbot.OptParseModeMarkdown); err != nil {
        log.Printf("unable to send message: %v", err)
    }
    s.pieGrph()
    if _, err := c.SendPhotoFile("<telegram_chat_id>", "output.png", tbot.OptCaption("PieGraph of Blocked Domains"));err != nil {
        log.Printf("unable to send image: %v", err)
    }
}

func (s *Stats) pieGrph() {
    defer log.Println("Pie Graph Generated and Image Send")
    var chartValues []chart.Value
    for i := range s.TopBlockedDomains {
        for k, v := range s.TopBlockedDomains[i] {
            values := chart.Value{Label: fmt.Sprintf("%.f:%s", v, k), Value: v}
            chartValues = append(chartValues, values)
        }
    }
    pie := chart.PieChart{
        Width:  512,
        Height: 512,
        Values: chartValues,
    }
    f, _ := os.Create("output.png")
    defer f.Close()
    pie.Render(chart.PNG, f)
}

That's it. Try a $ go run main.go to try out the bot. For any queries feel free to create a comment or reach out to me via Twitter.

Shameless Plug but you can donate to me on BMC ☕️ or Paypal

Happy Hacking ⚡️

Introducing Jiofi-CLI. A CLI interface for getting your Jiofi Device Info.

Athul Cyriac — Sat, 01 Aug 2020 07:22:34 +0000

JioFi is a portable wifi hotspot for personal use by Jio. They have somewhat generous plans and are a major telecom provider in India. Jiofi has a web dashboard to display the current details regarding the data usage, battery status and number of devices connected to the network. But I am too lazy to go to the dashboard to check the data out. So I built a small Python CLI to monitor the jiofi device from the network. I call it Jiofi-CLI. You can find the source code on GitHub at Jiofi-CLI. It's easy to install and you can install it with pip. The Python Package Manager.

The Build Breakdown

Before digging in deep, I saw a repo on GitHub regarding jiofi that scrapes data from the dashboard of a Jiofi and this seemed a legit solution for getting the info. So Initial solution which came into mind for getting the data was Web Scraping!! 😈. So went on to scrape the dashboard without any prior knowledge in web scraping. I read a few articles on using Beautiful Soup with Requests for fetching the data and parsing the HTML. I went forward with it for a bit and I found out that the data part won't be shown in the scraped output. I was sad😓. I don't have any idea on moving forward without the data. Then the geek inside me woke up. "If the data can't be scraped, then the data has to come from somewhere". A sudden Ctrl+Shift+i took me to the chrome developer console.

I spend half an hour tryin to figure out where the data is coming from. Trying the Networks tab finally yielded some luck. I saw a pattern in the waterfall tab in the Network section. About 300+ requests are send in the span of 10 mins. Okay that seems interesting. I checked all the XHR requests and found the motherlode. A series of POST requests were being sent to the JioFi's internal API server. That was a xxx-form-url-encoded requests to the endpoint. An internal API⚡️⚡️. I checked the request headers and saw that a specific keyword was used to fetch the data. These were for LTEStatus, DeviceStatus, WANStatus and so on. I tried to get the data from Insomnia which is a REST Client since Postwoman had shown me a few errors and I was too lazy to fix the errors with Postwoman for that time. Those errors were due to local endpoints and not anything serious.

It worked on Insomnia, I received the data that was to be used for the web dashboard or in simple terms, I mocked the web dashboard of JioFi with Insomnia. Win 🙌 Win. The response was like any other API response, in JSON. Here is the sample response.

{
  "Page": "GetLTEStatus",
  "Mask": "",
  "connected_status": "Attached",
  "connection_time": "00:00:57:19",
  "signal_strength": "Normal",
  "operating_mode": "TDD",
  "operating_band": "Band 40",
  "bandwidth": "20 MHz",
  "earfcn": "38750",
  "active_cell_id": "123",
  "plmn": "405 862",
  "apn_in_use": "jionet",
  "gcellid_pcellid": "12345",
  "eCellid": "12345",
  "ecgi": "1234567",
  "rsrp": "-106",
  "rsrq": "-13",
  "rssi": "74",
  "sinr": "9.8",
  "cinr": "not support",
  "tx_rate_total_data_transferred": "11.91 Kbps / 918.67 Kbyte",
  "rx_rate_total_data_received": "359.30 Kbps / 18.31 Mbyte",
  "ul_operator_limit": "50000000",
  "dl_operator_limit": "100000000"
}

This JSON response once hooked up with Python will give the precise data. I used Python's requests library to fetch the json from the URL. I divided all the Pages(Page of the specific Keyword, like "Page": "GetLTEStatus") to their respective functions to print the specific data to Stdout. Nothing fancy, just the print function of Python.

Once I made out these functions, I need to implement a CLI interface to be the icing on the cake. Initially went on with Argparse since it's builtin and has a lot of cool features. Argparse then felt a bit daunting, so I tried it with Click. Click also showed some errors due to some function's optional arguments, I dropped it. So a=searching for another one, I found out about fire. It had a really easy API and it just helped me finish the job quite quickly. I was completed with a CLI and ready for packaging. My friend GKR helped me on packaging the CLI and it was on pypi after a few hours(I was lazy to finish it duh...) and that was my first python package on Pypi. You can find it here.

This is the output of Jiofi command when installed on your system

$ jiofi
NAME
    jiofi

SYNOPSIS
    jiofi COMMAND

COMMANDS
    COMMAND is one of the following:

     devices
       Prints a tabular view of all the connected devices in the network

     speed
       Prints the Current upload and download speed

     basic
       Get basic Details like Battery charge and state, no of connected devices and data used in <time>

     usage
       Get Data usage in Upload and Download data in <time>

     device
       Get Details of the device, Battery Charge, Battery State, Phone number or MSISDN

This was the only place I used fire to hook it up with the functions

fire.Fire({
      'devices': getDevices,
      'speed': getBandwidth,
      'basic':getBasicDetails,
      'usage':getWanInfo,
      'device':deviceDetails
  })

Finding out about the Internal API was a 🤯 for me and it feels great to hacked on this. You can install the cli by

$ pip3 install jiofi --user

And this will install system wide. You can use it from anywhere with the terminal. I have tested it on Jiofi4 and heard that Jiofi6 has a different API, rewriting this to Go is in my mind and would make it fast too 🤪. There are a few bugs here and there and I try to fix it on my spare time. If you feel like contributing, feel free to open an Issue or a PR on the GitHub Repository, and can contact me via email.

Happy Hacking ⚡️

athul / jiofi-cli

Get Details about your Jio-fi Network from the Terminal

Jiofi CLI

Jiofi-CLI is a Command Line Interface to check your Jiofi Network from your terminal than using the web interface. It only shows very specific details like battery details, data usage and devices connected.

Using it

Install jiofi with pip. It works on Python version >=3.6.

pip install jiofi --user

Once you installed it, run it by typing jiofi in your terminal.

$ jiofi

You'll be welcomed by the help page of jiofi 😄

NAME
    jiofi
SYNOPSIS
    jiofi COMMAND
COMMANDS
    COMMAND is one of the following
     devices
       Prints a tabular view of all the connected devices in the network

     speed
       Prints the Current upload and download speed

     basic
       Get basic Details like Battery charge and state, no of connected devices and data used in <time>

     usage
       Get Data usage in Upload and Download data in <time>

     device
       Get Details of the device, Battery Charge, Battery State, Phone number

…

View on GitHub

Auto Updating Profile Readme with Waka-Readme

Athul Cyriac — Mon, 20 Jul 2020 05:37:35 +0000

Profile Readme are a new way to show off your GitHub profile and gives you all the powers of Markdown for doing it. Let's discuss a way you can add a nerdy stats to your profile readme with Python and GitHub Actions and how I use them to update my Profile Readme.

So I would like to introduce you to waka-readme. It's built with Python and can be used to generate a graph like this in your Profile Repo. It mostly uses regex and GitHub's REST API for Python to update the readme. This graph will be updated every day at 00.00 UTC on schedule thanks to GitHub Actions.

Week #29 : July 13 - 18

Python      8 hrs 52 mins       ███████████████████░░░░░░   75.87% 
Go          1 hr 15 mins        ██░░░░░░░░░░░░░░░░░░░░░░░   10.79% 
Markdown    52 mins             █░░░░░░░░░░░░░░░░░░░░░░░░   7.43% 
Docker      16 mins             ░░░░░░░░░░░░░░░░░░░░░░░░░   2.32% 
YAML        7 mins              ░░░░░░░░░░░░░░░░░░░░░░░░░   1.07%

Currently waka-readme has nearly 50 ⭐️s on GitHub and 3 Contributors and about 64 people use this in their Profile. It's not that fancy in any way but totally nerdy in my opinion🤓.

FlashBack

I don't know if any of you remember, but there was a time when pinnable gists came and the GitHub world just went for new tricks and hacks for creating pinnable gists. I saw some people rickrolling with pinnable Gists. That was a perfect time time for creating some cool hacks and GitHub Actions were still in the early beta phases. I was a user of WakaTime and the waka-box workflow and had been using it till the Profile Readme came.

Waka-Readme

Once I was able to get the profile readme on my GitHub profile, I tried to hack on it a bit and the first hack was to include the WakaTime stats in the profile readme. The first method I used was to scrape it from the Gist I made for waka-box by simply a GET request to the Gist's raw URL. Then I thought that this wouldn't be useful for any other people since that would simply introduce a bit more overhead for people not using waka-box. So I went to hack on it a bit more and used WakaTime's API to get my data and generate a graph with Python. I used the percentage part of the API response and divided that with 4 and used that value to generate a graph with 25 blocks of █ and ░. This will give a good graph for the metrics in regards with the percentage of time spent on a language. This data is collected from the WakaTime plugin in your code-editor and it support almost editors the last time I checked.

I encourage you to update your Profile readme till it gives a whole idea of who you are and if you have time do try waka-readme and fell free to open an issue or ping me on twitter @athulcajay for any queries plus drop a ⭐️ if you found it useful. Your feedbacks are highly appreciated.

athul / waka-readme

Wakatime Weekly Metrics on your Profile Readme.

Dev Metrics in Readme

WakaTime Weekly Metrics on your Profile Readme:

Prep Work

You need to update the markdown file(.md) with 2 comments. You can refer here for updating it.
You'll need a WakaTime API Key. You can get that from your WakaTime Account Settings
- You can refer here, if you're new to WakaTime
Optional You'll need a GitHub API Token with repo scope from here if you're running the action not in your Profile Repository
- You can use this example to work it out
You need to save the WakaTime API Key (and the GitHub API Token, if you need it) in the repository secrets. You can find that in the Settings of your Repository.Be sure to save those as the following
- WakaTime-api-key as WAKATIME_API_KEY = <your wakatime API Key>and
- The GitHub Access Token as GH_TOKEN=<your github access token>
You can follow either of the Two Examples…

View on GitHub

FastAPI + Deta = ⚡️

Athul Cyriac — Tue, 07 Jul 2020 15:21:56 +0000

Cover Photo by Chris Ried on Unsplash

FastAPI is a a Python Framework for building RESTful APIs. It has all the simplicity of Python with a added advantages of Async⚡️, automatic Schema Generation and OpenAPI and Python Types✨(with Pydantic). FastAPI is relatively a new Project and is gaining quite a good traction in the Dev world. It has got nearly 16k stars on GitHub and about 160 contributors. The main plus side of FastAPI is the documentation which is Top Notch.

Deta is a company that is building a cloud platform which is developer friendly and to reduce the hassle from idea to app.

"Deta is building a cloud for the developers with less build "bells and whistles", only what's needed to get the job done – a Micro Cloud."

Personally what attracted me to deta was the ease to implement. I was having a hard time to connect FastAPI with SQLalchemy since I am a noob. Then Deta entered or at least then I came to know about it.

I was skeptical at first, but the simplicity of working with Deta amazed me and their Slack community is ✨. The creators are almost active everytime and they will help you in anyway possible to overcome an issue.

What we'll build

We're going to build a Personal Mailing List API with

FastAPI for the API and Logic
Deta.sh as our DB
Deploy it on Deta Micros

We're only going to build the API, you can use Mailgun or SendGrid to send the emails. Our API will consist of 4 Operations.

Add a new Person to our List
Get all the emails from our list
Update the email of a specific user in our List
Delete a Subscriber/User from our List

Setting up FastAPI and Deta

Hope you have Python 3.6+ interpreter on your machine. If you don't make sure to download it and install and verify by running python in your terminal

$ python3
Python 3.7.6 (default, Dec 30 2019, 19:38:26)
[Clang 11.0.0 (clang-1100.0.33.16)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

The above output is on my machine and it will change from machine to machine.

Next thing is to have a folder and a virtual environment, for that we could use either Pipenv or venv. Pipenv is a package for handling and managing Python envs automatically, you can install pipenv with,

$ pip install --user pipenv

Once we installed Pipenv to set up our environment follow the commands to install

FastAPI
Uvicorn - The ASGI Server
Deta - Python Wrapper for the Deta Bases

$ mkdir mailing-list
$ cd mailing-list
$ pipenv shell #it will make the virtual environment
$ pipenv install fastapi uvicorn deta

If we're using venv, then for creating a venv and installing the packages, follow

$ mkdir mailing-list
$ cd mailing-list
$ python3 -m venv venv
$ source venv/bin/activate
$ pip install fastapi uvicorn deta

These are for Unix like systems and on Windows to activate a Virtualenv follow,

venv\Scripts\activate

Once we have set up our Dev Environment. Let's move on to building the Mailing List.

For Deta, create an Account in Deta and create a new Project and get the Project Key. Save the Project key since it will be shown once. You can save it as an environment variable by executing the below code in Unix like(Linux/Mac).

export DETA_PROJECT_KEY = <project key>

and on Windows, use

set DETA_PROJECT_KEY = <project key>

We'll also save the key in a .env file. This env file s important for the deployment process. For that we need to create a file called .env and save the Project Key like

DETA_PROJECT_KEY = <project key>

That's it. Lets move on to the code

The Code

Let's start by making a simple REST API endpoint with FastAPI on a file called main.py

# main.py
from fastapi import FastAPI

app = FastAPI()

@app.get("/")
async def helloworld():
    return "Hello World"

This is just a hello world app with FastAPI and for running this, we'll need uvicorn. Execute the following in the terminal/cmd,

$ uvicorn main:app --reload
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [13050] using statreload
INFO:     Started server process [13052]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

Our API will be actve the port 8000. FastAPI has inbuilt support for OpenAPI, Swagger and Redoc. To view the docs for our App, just go to localhost:8000/docs. To check if our endpoint works, just execute

$ curl http://127.0.0.1:8000/
"Hello World"

So now, let's move on to our main parts

New Subscriber

We'll be using the docs by Deta for all references.

We need to create a new endpoint for a someone to subscribe to our Mailing List. So we'll just need the Person's Name and Email to be in our DB.

Take care that in some countries its illegal to send emails to people without consent. We'll make a delete endpoint for them to delete themselves from the list

Let's get on to the code for adding a new subscriber to our list. The Python code without the use of a DB or simply the code with FastAPI.

# main.py

from fastapi import FastAPI
from pydantic import BaseModel, NameEmail

app = FastAPI()

class Subscriber(BaseModel):
    full_name: str
    email: NameEmail

@app.post("/subscribe")
async def newUser(sub: Subscriber):
    name = sub.full_name
    email = sub.email

    # you could also return sub.json() or sub.dict()
    return {
        "full_name": name,
        "email": email
    }

We're using Pydantic BaseModel class to inherit our Subscriber class. Then we make the subscriber class a dict to fetch data from the request's body. NameEmail will help us get the name part of the email.

The output is a NameEmail object which has two properties: name and email. For fred.bloggs@example.com, name property would be "fred.bloggs".

Since we're using NameEmail we'll need to install email-validator to work with it. Just run pip install pydantic[email-validator]

This snippet will return the things that we have provided to it. I have entered the data by going to localhost:8000/docs if you have your server running. Our Output if we provide athul for name and athul@example.com for email will be

{
  "name": "athul",
  "email": "athul@example.com",
}

Now let's connect it with the database, Deta Base in this case. We'll be using Python's os package to get the Project Key. We have save the project key to the env vars and restart the server for it to work.

# main.py

import os
from fastapi import FastAPI
from pydantic import BaseModel, NameEmail
from deta import Deta

deta = Deta(os.getenv("DETA_PROJECT_KEY"))
db = deta.Base("mydb")  # this can be anything related to your use case

app = FastAPI()

class Subscriber(BaseModel):
    full_name: str
    email: NameEmail

@app.post("/subscribe")
async def newUser(sub:Subscriber):
    subscriber = db.put({
        "full_name": sub.full_name,
        "email": sub.email.email,
        "key": sub.email.name
    })

    return subscriber

Here we have imported the deta library and initialized it with a db name. We're specifying a key for the data. This key will be the name part of the email since most times it will be unique. If we don't specify a key, deta will generate a unique key for the record.

You can have multiple DBs with a single project key

We're using Deta's PUT method(/function) to insert our data into our DetaBase. The PUT method will generate a unique key for our data and will be stored if we don't specify a key. The output if we add ourselves a new subscriber with provide John Doe for name and john@example.com for email will be,

{
  "email": "john@example.com",
  "key": "john",
  "full_name": "John Doe"
}

So we have made an endpoint for people to subscribe to our Mailing List 🎉.

Next we'll make an endpoint for people to update a theirs info like their Name or Email.

Update a Subscriber

We'll create a new UpdateSubscriber class for updating a Subscriber's Name or Email. We'll use the get and update methods of deta to fetch and update the data. We'll also raise an error if the record is not found in the base

# main.py

from typing import Optional
from fastapi import NameEmail,HTTPException

class UpdateSub(BaseModel):
    full_name: Optional[str] = None
    email: Optional[NameEmail] = None

@app.patch("/subscriber/{email}")
async def update_sub(upsub: UpdateSub, email: NameEmail):
    sub = db.get(email.name)
    if sub is None:
        raise HTTPException(404,"User Not Subscribed")

    updates = {}
    if upsub.full_name:
        updates["full_name"] = upsub.full_name

    if upsub.email:
        updates["email"] = upsub.email.email

    db.update(updates, key=sub["key"])

    return {"msg": "User Updated"}

This endpoint will update the email or full_name of a subscriber but the caveat here is that the key cannot be changed even if the email is changed. Let's make a new user with the name John Reese(Person of Interest) with an initial email reese@gmail.com. Now let's update his email using this endpoint, for that our request URL will be as follows,

http://localhost:8080/subscriber/reese%40example.com You can see that after the /subscriber in the url, we are passing the original email and this is called a Path Parameter. We use the path paramter and get the name part of it and use it as the Key. For updating the email, we take the new email and update it in our Deta base with the update command. The updates should be in the form of a Python dict. If the operation done is correct, then we'll get a None type in return

Get all Subscribers

This endpoint will send a GET HTTP method for fetching all the data from our Deta-Base. We'll use the fetch function/method of deta to retreive all the data from our base. The fetch function will return a generator type, so we'll be using the next() function to get all the data as a List. The endpoint will be like

@app.get("/subscribers")
async def get_all_subs():
    return {"subscribers": next(db.fetch())}

The output of the above request will be

{
  "subscribers": [
    {
      "email": "athul@example.com",
      "key": "athul",
      "name": "Athuk"
    },
    {
      "email": "john@example.com",
      "full_name": "John Doe",
      "key": "john"
    },
    {
      "email": "jhreese@gmail.com",
      "full_name": "John Reese",
      "key": "reese"
    }
  ]
}

Now we have setup a Create, Update and Read endpoints. Next we'll look into the Delete operation.

Delete Subscribers

To delete a subscriber, we'll take in the full_name as the input to find the subscriber and if we there is more than one record with the same full_name, we'll fetch the data with email. Then we'll use the delete method to delete the record by getting the key from our records.

# main.py

class DelSub(BaseModel):
    email: Optional[NameEmail]
    full_name: str

@app.delete("/subscribers.del")
async def unsubscribe(unsub: DelSub):
    data = delsub.dict()
    name = data.get('full_name')
    email = data.get('email')
    sub = next(db.fetch({'full_name':name}))
    if len(sub) > 1 or len(sub) == 0:
        sub = next(db.fetch({'email':email.email}))
        if len(sub)==0:
            raise HTTPException(400,"email not found")
    db.delete(sub[0]['key'])
    return {"msg":"Record Deleted"}

The response if a record is deleted will be a Python None type so we respond with a message,

{"msg":"Record Deleted"}

So we have made a CRUD API for a personal mailing list. Now let's deploy it.

Deploy our API

For Deploying our API, we'll need Deta Micros.

Deta Micros(ervers) are a leightweight but scalable cloud runtime tied to an HTTP endpoint. They are meant to get your apps up and running blazingly fast. Focus on writing your code and Deta will take care of everything else.

You need to have Deta Micros' CLI installed in your system. You can install by following the instructions in this link. Once you've installed, we need to login to Deta via the CLI, for that execute the following command

deta login

It'll open a new browser window and will authenticate the CLI. Next we need to start a new micro. We'll also need a requirements.txt for managing our dependencies, for that either you can copy the response for the command pip freeze and paste it in a file called requirements.txt or on *nix systems, you can just execute

pip freeze > requirements.txt

This will paste the output of pip freeze to requirements.txt. Once we have done we need to deploy it to deta using the command. Once we've done this let's create a new micro, for that we can use the deta new command

$ deta new
{
        "name": "mailing-list",
        "runtime": "python3.7",
        "endpoint": "<url>",
        "visor": "enabled",
        "http_auth": "enabled"
}

It will automatically detect that we'll need a Python runtime and will deploy our code and give us a unique link for our Micro.

Yay we have deployed the API, for the details of the micro, just execute the following for details of the micro.

$ deta details
{
        "name": "mailing-list",
        "runtime": "python3.7",
        "endpoint": "<url>",
        "visor": "enabled",
        "http_auth": "enabled"
}

We need to update the environment variable to set our project_key, execute the deta update command like this

$ deta update -e .env
Updating environment variables...
Successfully updated micro's environment variables

You can disable authentication for the API to be publically available, just execute deta auth disable.

$ deta auth disable
Successfully disabled http auth

If we go to our Micro's url we'll see a Hello World response which we wrote as the / endpoint. We can see the OpenAPI spec and the Swagger Documentation, if we go to the /docs url, we'll see a page like this

Yay we have created an API with FastAPI and Deta and Deployed it to the Web.

That leads to the end of this post, here is the whole source code

import os
from typing import Optional
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, NameEmail
from deta import Deta

deta = Deta(os.getenv("DETA_PROJECT_KEY"))
db = deta.Base('mailing-list')

app = FastAPI()

class Subscriber(BaseModel):
    full_name: str
    email: NameEmail

@app.post("/subscribe")
async def newUser(sub:Subscriber):
    data = sub.dict()
    name = data.get("full_name")
    email = data.get("email")
    subscriber= db.put({
        "full_name":name,
        "email":email.email
    },key=email.name)
    return subscriber

@app.get("/subscribers")
async def getAllSubs():
    return {"subscribers":next(db.fetch())}

class UpdateSub(BaseModel):
    full_name: Optional[str] = None
    email: Optional[NameEmail] = None

@app.patch("/subscriber/{email}")
async def updateSub(subscriber:UpdateSub,email:NameEmail):
    data = subscriber.dict()
    new_name = data.get("full_name")
    new_email = data.get("email")
    sub = db.get(email.name)
    if sub is None:
        raise HTTPException(400,"User Not Subscribed")
    if new_email is None:
        update = {"full_name":new_name}
    if new_name is None:
        update = {"email":new_email.email}
    db.update(updates=update,key=sub['key'])
    return {"msg":"User Updated"}
class DelSub(BaseModel):
    email:Optional[NameEmail]
    full_name:str

@app.delete("/subscribers.del")
async def deleteSub(delsub:DelSub):
    data=delsub.dict()
    name = data.get('full_name')
    email = data.get('email')
    sub = next(db.fetch({'full_name':name}))
    if len(sub) > 1 or len(sub) == 0:
        sub = next(db.fetch({'email':email.email}))
        if len(sub)==0:
            raise HTTPException(400,"email not found")
    db.delete(sub[0]['key'])
    return {"msg":"Record Deleted"} 

@app.get("/")
async def helloworld():
    return "Hello World"

Conclusion

This is just a demo for getting started with FastAPI with Deta.sh. You can change the key for your preference or use the key generated automatically by Deta. You can also use the email address as your key but you may need to switch the @ character with something else like a - or _ for its place. If you have any queries or doubts or feedback, you can use the comments section for the same and do let me if this has helped you. Go forth and build awesome stuff ⚡️. Thank You ❤️

Connecting GitHub to Telegram with GitHub Actions

Athul Cyriac — Fri, 24 Apr 2020 16:40:38 +0000

What we'll build

We'll be making a Telegram bot with Node.js and use it with GitHub Actions for sending notifications to you about the repo. You can find a fully functioning action here

Telegram is a cloud-based mobile and desktop messaging app with a focus on security and speed. It is free to use and extensively hackable. It also has a good bot support system. The API is also easy to implement and has many wrappers for building bots with the API.

GitHub Actions

GitHub Actions is a CI/CD runtime for your GitHub repository. You can run almost anything from scripts to docker containers. You can build, test and deploy your code with GitHub Actions. All these actions are called workflows and workflows differ in the job they're doing. These maybe test workflows, build ones or deployment ones. You can find all the actions on GitHub in the marketplace

Building the Bot

Prerequisites

Basic JavaScript Knowledge
Basic GitHub Knowledge
Telegram Account

There are templates for building actions. Here we're gonna start from scratch

Environment Setup

Node, You can download node from their website
NPM comes with node, so you don't have to worry about it.
Initialize the Project

$ git init ## initialize a new git repository for version management
---
$ npm init

dotenv, Dotenv can be downloaded via

$ npm i dotenv
---
$ yarn add dotenv

node-telegram-bot-api, node-telegram-bot-api is a simple wrapper for building telegram bots. You can download it via

$ npm i node-telegram-bot-api
---
$ yarn add node-telegram-bot-api

@zeit/ncc, NCC is a Simple CLI for compiling a Node.js module into a single file, together with all its dependencies, GCC-style. It's a dev dependency and can be downloaded

yarn add --dev @zeit/ncc
---
npm i -D @zeit/ncc

Folder Structure

The dist folder will be automatically created. action.yml will be made

.
├── dist
│   └── index.js
├── index.js
├── action.yml
├── README.md
└── package.json

index.js is the file we're defining the bot
action.yml is the file we'll define the action and it's behaviours

Making the Bot

We need to get an API bot token from telegram. For that Go to Telegram and Search for Botfather. It's a bot.

Create a new bot with the /newbot command and get the API key. We'll need that, also talk to jsondump bot and get your chat id. The output may be like this, so

{
  "update_id": 143943779,
  "message": {
    "message_id": 181575,
    "from": {
      "id": 123456 // this is what we need
      "is_bot": false,
      "first_name": "Tg Name",
      "username": "TG Username",
      "language_code": "en"
    },
    "chat": {
      "id": 123456,
      "first_name": "Tg Name",
      "username": "TG Username",
      "type": "private"
    },
    "date": 1584119424,
    "text": "message"
  }
}

This will be needed for further use and We need to add it to the repo secrets which can be found in the repo settings. Be careful to add it as token and chat like as shown below

Writing the Action and Building the Bot

Fire up the terminal/cmd and make a new folder. Install the dependencies. Run the following command

$ touch index.js action.yml

Open your favourite text editor within the folder or with the file. We'll define the bot in index.js

require("dotenv").config
const Bot = require('node-telegram-bot-api');
const {
    INPUT_STATUS: ipstatus,
    INPUT_TOKEN: tgtoken,//Telegram api token
    INPUT_CHAT: chatid,// Telegram Chat ID
    INPUT_IU_TITLE: ititle,// Issue title
    INPUT_IU_NUM: inum,// Issue Number
    INPUT_IU_ACTOR: iactor,// Issue made by
    INPUT_IU_BODY: ibody,// Issue Body
    INPUT_PR_NUM: pnum,// PR Number
    INPUT_PR_STATE: prstate,// PR Opened, reponed or closed
    INPUT_PR_TITLE: ptitle,// PR Title
    INPUT_PR_BODY: pbody,// Body of the PR
    GITHUB_EVENT_NAME: ghevent,// Name of the trigger event
    GITHUB_REPOSITORY: repo,// Repository the trigger was made from
    GITHUB_ACTOR: ghactor,// User who triggered the action
    GITHUB_SHA: sha,// Commit ID
    GITHUB_WORKFLOW: ghwrkflw// Workflow Name
} = process.env;

const bot = new Bot(tgtoken)

First, we're defining the dotenv for config and initializing Telegram Bot. Here we're defining the alias variables for the environment variables. You might notice an INPUT_ for almost every environment variable, this is because GitHub Actions pass the env variable with an INPUT prefix. Other env variables are action's default environment variables. Then we initialized the bot with the API token.

GitHub actions could be triggered with Issues, Pull Request or Pushes. You can find the trigger events here. Here we're gonna get a message from the bot when an Issue or Pull Request or a Push event has happened.

const evresp = (gevent) => {
    switch (gevent) {

        case "issues":
            return `
❗️❗️❗️❗️❗️❗️

Issue ${prstate}

Issue Title and Number  : ${ititle} | #${inum}

Commented or Created By : \`${iactor}\`

Issue Body : *${ibody}*

[Link to Issue](https://github.com/${repo}/issues/${inum})
[Link to Repo ](https://github.com/${repo}/)
[Build log here](https://github.com/${repo}/commit/${sha}/checks)`
        case "pull_request":
            return `
🔃🔀🔃🔀🔃🔀
PR ${prstate} 

PR Number:      ${pnum}

PR Title:       ${ptitle}

PR Body:        *${pbody}*

PR By:          ${ghactor}

[Link to Issue](https://github.com/${repo}/pull/${pnum})
[Link to Repo ](https://github.com/${repo}/)
[Build log here](https://github.com/${repo}/commit/${sha}/checks)`
        default:
            return `
⬆️⇅⬆️⇅

ID: ${ghwrkflw}

Action was a *${ipstatus}!*

\`Repository:  ${repo}\` 

On:          *${ghevent}*

By:            *${ghactor}* 

Tag:        ${process.env.GITHUB_REF}

[Link to Repo ](https://github.com/${repo}/)
            `
    }
}

In these lines of code, we're just initializing a switch statement for the responses. We're also declaring an anonymous function to use the switch responses via a function later. We're using all the defined variables in the switch. You can check the trigger Events to get how the event is triggered and what keyword should be used.

Now for the last part of the Js file, we just take the response from the switch and assign it to a constant. Then we use the sendMessage function of the node-telegram-bot-api to send the message to the bot with the chatid and the output as the arguments.

const output = evresp(ghevent)

bot.sendMessage(chatid,output,{parse_mode : "Markdown"})

Compiling and Minifying the Js code

Since we have installed @zeit/ncc and this is used for the making the whole program with all the APIs to a single file and we need to use NCC for that. We just need to run

yarn run ncc build index.js -C -m -o dist

or you might wanna add the following to you package.json file, and run npm run test to compile and minify the code.

"scripts": {
    "test": "ncc build index.js -C -m -o dist"
  },

This will create a dist folder with and index.js file which contains the compiled code.

Making it a valid action

For making this Js file a valid action, we need to add an action.yml file. The action.yml for this action is like this

name: 'Action Name'
description: 'Action Description'
author: '<author name>'
inputs: 
  chat:
    description: 'Chat to send: chat id or @channel_name'
    required: true
  token:
    description: 'Telegram Bot token'
    required: true
  status:
    description: 'Job status'
    required: true
  iu_title: 
    description: 'Issue Title'
    default: ${{ github.event.issue.title }}
  iu_num:
    description: 'Issue Number'
    default: ${{ github.event.issue.number }}
  iu_actor: 
    description: 'Issue Triggerer'
    default: ${{ github.event.issue.user.login }}
  iu_com:
    description: 'Issue Comment'
    default: ${{github.event.comment.body}}
  pr_state:
    description: 'State of the PR'
    default: ${{ github.event.action }}
  pr_num:
    description: 'PR Number'
    default: ${{ github.event.number }}
  pr_title:
    description: 'Title of the PR'
    default: ${{ github.event.pull_request.title }}
  pr_body:
    description: 'Body/Contents of the PR'
    default: ${{ github.event.pull_request.body }}
runs:
  using: "node12"
  main: "dist/index.js"
branding:
  icon: 'repeat'  
  color: 'green'

Here we're defining the Input variables to be loaded for the action in GitHub's runtime environment. All these default data are taken from the response of the webhooks which are send by GitHub when a trigger event is occurred. You can find out more in the Action Documentation.

runs:
  using: "node12"
  main: "dist/index.js"

Here we are defining that this is a node action and should run in an environment with node, and the file which should be run, here the index.js file in the dist folder. That should do it. Create a new commit and push it to a repo. Create a new tag and this action will appear in the marketplace.

Defining a workflow to test your action

GitHub Action workflows are defined using the .yml syntax. Here is an example of a sample workflow for this action

name: <Workflow Name>

on:
  push:
  pull_request:
    types: [opened, closed]
  issues:
    types: [opened, closed, reopened]
jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - name: <AnyName>
        uses: <username>/<repo>@master
        if: always()
        with:
          chat: ${{ secrets.chat }}
          token: ${{ secrets.token }}
          status: ${{ job.status }}

The Complete code for the bot is

//Initializing dotenv and the bot
require("dotenv").config
const Bot = require('node-telegram-bot-api');
// aliasing the environment variables 
const {
    INPUT_STATUS: ipstatus, 
    INPUT_TOKEN: tgtoken, //Telegram api token
    INPUT_CHAT: chatid,// Telegram Chat ID
    INPUT_IU_TITLE: ititle,// Issue title
    INPUT_IU_NUM: inum,// Issue Number
    INPUT_IU_ACTOR: iactor, // Issue made by
    INPUT_IU_BODY: ibody, // Issue Body
    INPUT_PR_NUM: pnum, // PR Number
    INPUT_PR_STATE: prstate, // PR Opened, reponed or closed
    INPUT_PR_TITLE: ptitle, // PR Title
    INPUT_PR_BODY: pbody, // Body of the PR
    GITHUB_EVENT_NAME: ghevent, // Name of the trigger event
    GITHUB_REPOSITORY: repo, // Repository the trigger was made from
    GITHUB_ACTOR: ghactor, // User who triggered the action
    GITHUB_SHA: sha, // Commit ID
    GITHUB_WORKFLOW: ghwrkflw // Workflow Name
} = process.env;

const bot = new Bot(tgtoken)
// Function to return the response for the specific trigger
const evresp = (gevent) => {
    switch (gevent) {
//Switch statement for issues
        case "issues":
            return `
❗️❗️❗️❗️❗️❗️

Issue ${prstate}

Issue Title and Number  : ${ititle} | #${inum}

Commented or Created By : \`${iactor}\`

Issue Body : *${ibody}*

[Link to Issue](https://github.com/${repo}/issues/${inum})
[Link to Repo ](https://github.com/${repo}/)
[Build log here](https://github.com/${repo}/commit/${sha}/checks)`
// Switch statement for Pull Requests
        case "pull_request":
            return `
🔃🔀🔃🔀🔃🔀
PR ${prstate} 

PR Number:      ${pnum}

PR Title:       ${ptitle}

PR Body:        *${pbody}*

PR By:          ${ghactor}

[Link to Issue](https://github.com/${repo}/pull/${pnum})
[Link to Repo ](https://github.com/${repo}/)
[Build log here](https://github.com/${repo}/commit/${sha}/checks)`
        default:
// switch statement for Pushes
            return `
⬆️⇅⬆️⇅

ID: ${ghwrkflw}

Action was a *${ipstatus}!*

\`Repository:  ${repo}\` 

On:          *${ghevent}*

By:            *${ghactor}* 

Tag:        ${process.env.GITHUB_REF}

[Link to Repo ](https://github.com/${repo}/)
            `
    }
}
// assigning the output to a variable
const output = evresp(ghevent)
// sending the message
bot.sendMessage(chatid,output,{parse_mode : "Markdown"})

You can try out many different items using actions and this is just a sample action to get you started. Maybe sending Cat GIFs if the build succeeded on the pull request or sending a welcome message to a first time contributor. You imagination is the limit😄 and Never Stop being ⚡️

athul / telewire

Connect GitHub to Telegram

Deploy a Hugo Website to GitHub Pages

Athul Cyriac — Mon, 20 Apr 2020 04:39:16 +0000

Originally Posted on My Personal Blog

Due to the quarantine I thought of doing something good and useful for others. So I hacked up a theme for Hugo called Archie. The design was inspired from https://blog.jse.li and was hacked from the Ezhil theme for Hugo. The main reason why I built it was due to the Hugo-Ink theme built by knadh since I'm totally a fanboy of him. So deploying Archie to GitHub Pages was quite fun and easily manageable. GitHub Actions made it a lot easier since I don't have to rely on any other platforms for deployment.

Steps to Deploy the Hugo Site

There are a few steps to deploy the Hugo site to GitHub Pages and this is these steps are taken from a Discussion Thread in the Hugo Forum

Create the SSH Deploy Keys

A question arises here.... Why SSH keys? SSH keys are safe because they are secured with a SHA256 hash. It's also better to use the Deploy key than a GitHub API token as it should be confidential.

ssh-keygen -t rsa -b 4096 -C "$(git config user.email)" -f gh-pages -N ""

Execute this command and it will generate 2 files:

gh-pages.pub (public key)
gh-pages (private key)

Now we need to add these files to GitHub,for that go to repository setting and click on the Deploy Keys button, add your public key with the Allow write access

Next go to the Secrets button and add the Private key(gh-pages) as ACTION_DEPLOY_KEY.

To copy the public / private keys just run

cat gh-pages | pbcopy

and it will copy the private key to the clipboard

Defining the Workflow

A workflow is a a file which runs your action on the GitHub Environment, the Workflow is a Yaml file and it contains specific instructions on what to do and how to do it.

For our purpose we'll be defining a workflow for deploying to GitHub Pages whenever we get a push to the master branch. Paste these in the .github/workflows/gh-pages.yml file

name: Github Pages
on:
  push:
    branches:
      - master
jobs:
  deploy:
    runs-on: ubuntu-18.04
    steps:
      - uses: actions/checkout@v2
        with:
          submodules: true

      - name: Setup Hugo
        uses: peaceiris/actions-hugo@v2
        with:
          hugo-version: '0.67.1'
      - name: Build
        run: hugo --minify
      - name: Deploy
        uses: peaceiris/actions-gh-pages@v3
        with:
          deploy_key: ${{ secrets.ACTIONS_DEPLOY_KEY }}
          publish_dir: ./public

Note: Before deploying, just add the url to which you're going to deploy in the baseURL of the config.toml file with a trailing /

What I learned?

With Archie too 😁,

Hugo as a SSG
CSS styling
Hugo Templating syntax
Deploying with Deploy Keys in GitHub
Designing New Hugo Shortcodes

Get SMS notifications from GitHub

Athul Cyriac — Mon, 06 Apr 2020 03:21:04 +0000

What I built

SGS Stands for Short GitHub SMS 😁. It's an action which could be used to get updates about your GitHub repository like Issues and Pull Requests to your mobile number using twilio's SMS API. It's a simple JavaScript action which is below 70 lines of code.

Demo Link

I couldn't test it or demo it since there was no SMS twilio numbers to be available in India

Link to Code

https://github.com/athul/SGS

async function run() {
  const resultMessage = await client.messages.create({
    from,
    to,
    body: message,
  });
  tools.log.success("SMS Sent")
  return resultMessage
}

These 9 lines of code does all the sms work

How I built it (what's the stack? did I run into issues or discover something new along the way?)

The Stack is

Node.js
GitHub Actions

I've been hacking with actions for a few weeks so thought that this could be merged.
There was an official twilio action but it was limited to a specific message. I upgraded it to use a message with data from GitHub using literals and string truncation to reduce the title below 30 characters.Learned that 160 characters was the limit

Additional Resources/Info

https://github.com/JasonEtco/actions-toolkit Library for integrating actions
https://github.com/twilio-labs/actions-sms Official Twilio Action

From 0 to Go, Building a CLI for Postwoman

Athul Cyriac — Sat, 01 Feb 2020 06:38:16 +0000

"All power is within you. You can do anything and everything. Believe in that" - Swami Vivekananda

You might think why I started with a quote by Swami Vivekananda. This quote has been a driving factor in my pursuits to learn new things and building stuff.

I have been into Go for about 2 months now. I just wrote up two hacks in Go too. One was Shelby and now letme introduce you to Postwoman CLI. As the name says It's a CLI client for Postwoman 👽 in Go.

The '0' 👦

Learning either Go or Rust was in my TODO list of 2019. It was only in late 2019, I had reached the mindset to learn Go. I started out on tutorials in building servers in Go. I used to code servers in Python and fell for Backend Programming. I made a bunch of servers in Go with Gin. Go was intimidating at first since it felt more like C to me, but in a few days I got the clutch. I still don't know most of Go like using slices and goroutines that much. I learn mostly by building so, I learn when I build.

The '-1' 👶

I met Liyas Thomas, the author of Postwoman in the official Telegram Group of Postwoman. Since he too was from the state of Kerala, we had a few convos. I had opened a PR to Postwoman when it launched in August-September but it didn't get merged because of my lack of Vue/Js. But later on I had a chat with him if I did a refactor PR. He really gave me the confidence to do that and I did. The PR was reviewed and made changes by the Maintainers of PW and where it started in 1 commit it changed to 8 and the diff was +8k and -8k. The PR got merged and I was the #11 Contributor of PW 🎉. The PR didn't give me any satisfaction but later after 2 days, Liyas send me a bug and told me to fix that. I traversed over 2k lines of Vue code and found the bug. He made a commit and fixed it. This bug squashing gave me much satisfaction than all the contributions I've done. Later that week, a maintainer and My Friend James made a repo for a CLI in Js. So the idea for making another CLI was basically for learning for me.

The '1' 👨

Liyas gave a Green Flag to work on the CLI. I considered Go as first option since I didn't know how to make one in any Lang and Go was fresh in my mind. I built it in almost 2 weeks. pwcli is like httpie with displaying the Request Headers and Status Code and Output Highlighting. The Output is now limited to Json but will add support to HTML and XML in the coming days. It also has a send feature in which you can give it the Postwoman-Collection.json file and it will test all the endpoints in their respective methods and output the status code. This was actually a feature request in the Js counterpart of the CLI.

You can find PwCLI in GitHub at https://github.com/athul/pwcli. Consider trying it out and giving a star.
PwCLI is open sourced under MIT and is waiting for Feature requests and Contributions

NAME:
   Postwoman CLI - Test API endpoints without the hassle

USAGE:
   cli [global options] command [command options] [arguments...]

VERSION:
   0.0.2

DESCRIPTION:
   Made with <3 by Postwoman Team

COMMANDS:
   get      Send a GET request
   post     Send a POST Request
   put      Send a PUT Request
   patch    Send a PATCH Request
   delete   Send a DELETE Request
   send     Test all the Endpoints in the Postwoman Collection.json
   help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --help, -h     show help
   --version, -v  print the version

The '+1' 👨‍💻

In these two weeks, I've learned a lot of Go, HTTP Requests and REST APIs. The main highlights were.

Learned Go structures
How to parse Json in Go
Learned Go Slices
Learned how to use net/http package of Go
HTTP Requests
Meeting the Postwoman team, especially from Kerala
Homebrew Packaging and bits of Ruby
Building CLIs
My insecurity with starting a convo with new people
Taking a workshop (by taking one)

"Learn like your life depended on it, coz it will" - Unknown

postwoman-io / pwcli

HTTP Client like Curl or Httpie and a CLI Client for Postwoman

Thanks to

.ltag__user__id__89864 .follow-action-button { background-color: #c800ff !important; color: #ffffff !important; border-color: #c800ff !important; }

Liyas Thomas

🦄 a unicorn

liyasthomas liyasthomas https://liyasthomas.web.app

Follow me on twitter @athulcajay

Athul Cyriac

@athulcajay

Hey ya'll been building a CLI for Postwoman.io
It's now Open Sourced under MIT License and you can find it at github.com/athul/pwcli
It's written in @golang and I didn't even know how to write CLI's in Go a week ago when I started on it.

13:54 PM - 31 Jan 2020

1 18

Deploying Go Applications to GitHub Releases with GoReleaser and Travis CI

Athul Cyriac — Mon, 06 Jan 2020 13:28:17 +0000

Setting Up Goreleaser and Travis CI

All the installation Instructions of Gorelaser can be found in their Official Docs. I'm assuming you know how to set up Travis CI.

After Installing Goreleaser, you just have to cd to that go repository you want to deploy and run goreleaser init. This command will create a goreleaser.yml already filled with some commands and stuff.

You can add these to the bottom of the .gorealeaser.yml made by goreleaser init and it'll work fine.

Here is the Gorelaser part to Deploy the Application to GitHub Relases

release:
  github:
    owner: <username>
    name: <repo>

For Deploying to GitHub Releases change your .travis.yml file like this

deploy:
- provider: script
  skip_cleanup: true
  script: curl -sL https://git.io/goreleaser | bash
  on:
    tags: true

This will only deploy to GitHub releases when a tag is created and pushed. You will need a GitHub Token in the Repo scope and while adding it to Travis Secrets you must name it like GITHUB_TOKEN only.

Some tips before pushing a tags for checking Goreleaser.

You can always try goreleaser in you the terminal.

Try goreleaser release --skip-publish command and you can always test with this before you push to GitHub. I'd totally suggest this since I had simply pushed without any testing and I had to write the whole thing...again.

GoDownloader

You can install godownloader from their releases page. Follow the next steps to use it Globally

$ tar -xvf <filename>.tar.gz
$ mv godownloader /usr/local/bin/

Now go to your working folder and run

$ godownloader --repo=<user>/<repo> > godownloader.sh

This will make a godownlaoder.sh file which can be used as an installation script.

Here are my Pipelines for Shelby

.travis.yml

sudo: false
language: go

go:
  - "1.13"
before_script:
  - go get -v
  - go build
  - rm -rf shelby
#
# before we deploy, we go build for all operating systems we would like to support
deploy:
- provider: script
  skip_cleanup: true
  script: curl -sL https://git.io/goreleaser | bash
  on:
    tags: true

.goreleaser.yml


before:
  hooks:
    # you may remove this if you don't use vgo
    - go mod tidy
    # you may remove this if you don't need go generate
    - go get -v 
builds:
- 

  env:
    - CGO_ENABLED=0
  ldflags:
    - -s -w 
  goos:
    - darwin
    - linux
  goarch:
    - amd64
archives:
- replacements:
    darwin: Darwin
    linux: Linux
    amd64: x86_64
checksum:
  name_template: 'checksums.txt'
snapshot:
  name_template: "{{ .Tag }}"
changelog:
  sort: asc
  filters:
    exclude:
    - '^docs:'
    - '^test:'
release:
  github:
    owner: athul
    name: shelby