DEV Community: billylkc

Static site crawling with goq

billylkc — Thu, 09 Nov 2023 10:15:29 +0000

(You can find my original post here @ data-gulu.com)

Static site crawling with goq
1. Introduction
2. Usage
3. Static site crawling
4. Result
5. Other Demo
  1. a) Wikipedia
  2. b) OpenRice
6. Final thoughts

Static site crawling with goq

Introduction

We get it. You want the data, not the headache.

A robust web scrapers helps you to extract data from websites, saving you time on some tedious data collection tasks. It also allows you to make smarter decisions with the insights discovered from the captured data.

Here are some of the common use cases with web crawlers.

Sentiment analysis on social media
Price tracking
Content Aggregation
Product data enrichment
Brand Monitoring
etc..

In this article we will introduce a package in Go (goq) to crawl some static websites.

Usage

Basically, what you have to do is 3 things.

Study the *static page* (Not dynamic content loaded with Javascript) and identify the elements with css selector
Construct a golang struct with the goquery tag (e.g. `goquery:"div p`")
Decode into a struct with goq.NewDecorder

Static site crawling

Here I will be using a scraping friendly website (example.com), and capture the header h1 and the following paragraph div p.

package main

import (
    "encoding/json"
    "fmt"
    "log"
    "net/http"

    "astuart.co/goq"
)

// Example as a struct to capture data from example.com
type Example struct {
    Title     string `goquery:"h1"`
    Paragraph string `goquery:"div p"`
}

func main() {
    res, err := http.Get("https://example.com/")
    if err != nil {
            log.Fatal(err)
    }
    defer res.Body.Close()

    var example Example

    err = goq.NewDecoder(res.Body).Decode(&example)
    if err != nil {
            log.Fatal(err)
    }

    fmt.Println(PrettyPrint(example))
}

// PrettyPrint to print struct in a readable way
func PrettyPrint(i interface{}) string {
        s, _ := json.MarshalIndent(i, "", "\t")
        return string(s)
}

Result

{
    "Title": "Example Domain",
    "Paragraph": "This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.More information..."
}

Other Demo

The following demos are for illustration purposes. Do check the robots.txt file of each site and crawl responsibly.

a) Wikipedia

Let say we want to create a program that retrieves the first 3 paragraphs of the keyword (e.g. Linear regression) that we provide. Here are some key steps.

1. Study the css selector of the info that we need

Right Click on the paragraph -> Inspect -> Properties -> css selector

2. Create a struct with the goquery tag

We can easily check that the header is h1 and detailed descriptions div.mw-parser-output p and it should return a list of strings

type Wiki struct {
    Title   string   `goquery:"h1"`
    Content []string `goquery:"div.mw-parser-output p"`
}

3. Decode into Go struct with the GET response

// wiki gets the wikipedia link
func wiki(keyword string) Wiki {

    // construct wiki links
    link := fmt.Sprintf("https://en.wikipedia.org/wiki/%s", keyword)
    res, err := http.Get(link)
    if err != nil {
            fmt.Println(err)
    }
    defer res.Body.Close()

    var wiki Wiki
    err = goq.NewDecoder(res.Body).Decode(&wiki)
    if err != nil {
            fmt.Println(err)
    }

    return wiki
}

Print the output


import (
     "encoding/json"
     "fmt"
     "log"
     "net/http"

     "astuart.co/goq"
 )

 func main() {
     kw := "Logistic Regression"
     w := wiki(kw)

     if len(w.Content) >= 3 {
             w.Content = w.Content[0:3]
     }
     fmt.Println(w)
}

5. Demo

Full Example

You can find out the full code example - Here

b) OpenRice

And for OpenRice. You can easily check that the css selector for title would be div.cms-detail-title.or-font-family and content as div.cms-detail-body-text div.

Full code

package main

import (
    "encoding/json"
    "fmt"
    "net/http"

    "astuart.co/goq"
)
// Article as a struct to parse Openrice Article
type Article struct {
    Title   string   `goquery:"div.cms-detail-title.or-font-family"`
    Content []string `goquery:"div.cms-detail-body-text div"`
}


func main() {
        // Random promotion link from the front page. May be going to be expired
        link := "https://www.openrice.com/zh/hongkong/promo/%E3%80%90%E7%B5%82%E6%96%BC%E7%B4%84friend%E9%A3%9F%E9%A3%AF%E3%80%91%E5%B0%96%E6%B2%99%E5%92%80%E4%B8%BB%E6%89%93%E9%A3%B2%E9%85%92%E6%B5%B7%E9%AE%AE-outdoor%E4%BD%8D%E6%9C%89%E6%B0%A3%E6%B0%9B-a5816"
        result := getArticle(link)

        fmt.Println(PrettyPrint(result))
}

// PrettyPrint to print struct in a readable way
func PrettyPrint(i interface{}) string {
        s, _ := json.MarshalIndent(i, "", "\t")
        return string(s)
}

// getArticle parses the input link and extract the content of the article
// We are passing the link as argument this time
func getArticle(link string) Article {
        res, err := http.Get(link)
        if err != nil {
                fmt.Println(err)
        }
        defer res.Body.Close()

        var rice Article
        err = goq.NewDecoder(res.Body).Decode(&rice)
        if err != nil {
                fmt.Println(err)
        }

        return rice
}

Result (Partial)

{
        "Title": "【終於約Friend食飯】尖沙咀主打飲酒海鮮　Outdoor位有氣氛",
        "Content": [
                "好耐無同朋友夜晚食飯傾計喇！久違咁耐嘅晚餐，梗係想搵個舒服地方飲吓酒、食好嘢啦，尖沙咀呢間樓上餐廳Poseidon 就有個戶外大平台，可以Chill 住睇夜景食飯；室內亦有舒服卡位、高凳位、煙霧特效座位。",
                "",
                "餐廳有駐場調酒師，多款啤酒、紅白酒、雞尾酒可以飲成晚。最正係大廚會定期引入Seasonal 靚食材推新菜式，記住飲酒都要留肚食好嘢呀。",
                "",
                "同場有卡位、高凳同戶外位揀。",
                "",
                "",
                "煙霧特效座位好有氣氛。",
                "",
                "人氣招牌菜　海鮮拼盤",
                "呢度一向主打靚食材，食靚嘢點少得海鮮，所以呢個「海鮮拼盤」一直好有人氣！今次新Menu 繼續有新鮮即開生蠔、雞尾凍蝦、翡翠螺、青口、蝦，新引入超鮮甜花甲，任沾泰式辣醬、紅酒醋、雞尾酒醬3款醬汁，有2人份或4人份揀。",
                "",
                "想食得盡興啲，都可以追加特級鱘魚魚子醬、波士頓龍蝦，會同其他海鮮一齊放喺貝殼大碟上枱，影相一流。",
                "",
                "貝殼大碟上枱超靚。"
      ]
}

Here is the source code - Here

Final thoughts

I have used many packages in Python and Go, this is by far the easiest package for static site crawling. All you have to do is to define a goquery tag in the struct, then Voila, you get your content.

Happy Coding!

You can find my original post here @ data-gulu.com

Hosting a Keyword Extraction Model with Flask and FastAPI

billylkc — Mon, 06 Nov 2023 09:09:36 +0000

[Hosting a Keyword Extraction Model with Flask and FastAPI Api Model
1. Introduction
2. Overview
3. Yet Another Keyword Extractor
  1. Main Features
  2. Heuristic Approach
4. Our ML model
  1. Installation
  2. Our model
5. Flask API
  1. Installation
  2. Serve with /keywords endpoint
  3. Host the server with port 5001 app.run(host="0.0.0.0", port=5001, debug=True)
6. Testing with curl
  1. Demo
7. FastAPI
  1. Key steps
  2. Code
  3. Host
  4. Documentation
8. Testing with Postman
  1. Demo
9. Complete example
10. Final thoughts

Hosting a Keyword Extraction Model with Flask and FastAPI API Model

Introduction

When you noticed learning how to build a machine learning model is not enough, you are graduating from the Data Science bootcamp.

Here is an introduction of ML model serving with Restful API endpoint.

Overview

In this post, we will be covering the following topics

Yet Another Keyword Extractor
Our ML model
Flask API
Testing with curl
FastAPI
Testing with Postman

At the end of this post, we will have an API endpoint (from localhost) to extract 20 keywords from a text paragraph with a simple POST requests.

Yet Another Keyword Extractor

YAKE (Yet Another Keyword Extractor) is a light-weight unsupervised automatic keyword extraction method. It rests on text statistical features extracted from single document to select the most important keywords.

It is quite useful to extract details from a text paragraph or use it as an alternatives to labeled data (When you don't have any).

Main Features

Unsupervised approach
Single-Document
Corpus-Independent
Domain and Language Independent

Heuristic Approach

As opposed to other keyword extraction algorithms like tf-idf, one of the strong selling points of YAKE is its ability to extract keywords within a single document. It can be easily applied to single paragraph or document, without the existence of a corpus, dictionary or other external collections.

Here are some components of the method outlined in the paper.

Text preprocessing
Feature extraction
*Individual terms score*
Candidate keywords list generation
Data deduplication
Ranking

Some interesting ideas during the term scoring process.

Casing

Reflects the casing aspect of a word.
Word Positions

Values more on the words occurring at the beginning of the documents, on the assumption that relevant keywords often concentrate more at the beginning.
Word Relatedness to Context

Computes the number of different terms that occur to the left and right side of the candidate word. The more the *number of different terms co-occur with the candidate word* (on both sides), the more the meaningness it is likely to be. Similar idea applies to different sentences as well.

For more details, you can check out the paper in the reference page.

Reference: Github and Paper

Our ML model

As the main purpose of this post is to demonstrate how to deploy a model with API endpoints, I will use a simple wrapper of yake.KeywordExtrator function to act as our machine learning model.

Installation

pip install yake

### Our model
import yake
def ExtractKeywords(text):
    """ Extract 20 keywords from the input text """

    language = "en"
    max_ngram_size = 2
    deduplication_thresold = 0.3
    deduplication_algo = "seqm"
    window_size = 1
    num_of_keywords = 20

    custom_kw_extractor = yake.KeywordExtractor(
        lan=language,
        n=max_ngram_size,
        dedupLim=deduplication_thresold,
        dedupFunc=deduplication_algo,
        windowsSize=window_size,
        top=num_of_keywords,
        features=None,
    )
    kws = custom_kw_extractor.extract_keywords(text)
    keywords = [x[0] for x in kws]  # kws is in tuple format, extract the text part

    return keywords

Flask API

Having a ML model ready is only half the job done. A model is useful only when someone is able to use it.

Now we are going to serve our model with a Restful API endpoint using Flask. The package uses a simple decorator format for you to define an endpoint, e.g. @app.route('/keywords', methods = ['POST', 'GET']).

Here we specify our endpoint to accept both GET and POST requests. The GET request will print a curl statement, and the POST request will extract the keywords.

Installation

pip install flask

Serve with `/keywords` endpoint

    from flask import Flask, request
import yake

app = Flask(__name__)

def ExtractKeywords(text):
    """ Extract 20 keywords from the input text """

    language = "en"
    max_ngram_size = 2
    deduplication_thresold = 0.3
    deduplication_algo = "seqm"
    window_size = 1
    num_of_keywords = 20

    custom_kw_extractor = yake.KeywordExtractor(
        lan=language,
        n=max_ngram_size,
        dedupLim=deduplication_thresold,
        dedupFunc=deduplication_algo,
        windowsSize=window_size,
        top=num_of_keywords,
        features=None,
    )
    kws = custom_kw_extractor.extract_keywords(text)
    keywords = [x[0] for x in kws]  # kws is in tuple format, extract the text part

    return keywords

@app.route('/keywords', methods = ['POST', 'GET'])
def keywords():
    if request.method == "POST":
        json_data = request.json
        text = json_data["text"]
        kws = ExtractKeywords(text)

        # return a dictionary
        response = {"keyowrds": kws}
        return response

    elif request.method == "GET":
        response = """
            Extract 20 keywords from a long text. Try with curl command. <br/><br/><br/>

            curl -X POST http://127.0.0.1:5001/keywords -H 'Content-Type: application/json' \
            -d '{"text": "Logistic regression is a statistical model that in its basic form uses a logistic function to model a binary dependent variable, although many more complex extensions exist. In regression analysis, logistic regression[1] (or logit regression) is estimating the parameters of a logistic model (a form of binary regression). Mathematically, a binary logistic model has a dependent variable with two possible values, such as pass/fail which is represented by an indicator variable, where the two values are labeled 0 and 1. In the logistic model, the log-odds (the logarithm of the odds) for the value labeled 1 is a linear combination of one or more independent variables (predictors); the independent variables can each be a binary variable (two classes, coded by an indicator variable) or a continuous variable (any real value). The corresponding probability of the value labeled 1 can vary between 0 (certainly the value 0) and 1 (certainly the value 1), hence the labeling; the function that converts log-odds to probability is the logistic function, hence the name. The unit of measurement for the log-odds scale is called a logit, from logistic unit, hence the alternative names. Analogous models with a different sigmoid function instead of the logistic function can also be used, such as the probit model; the defining characteristic of the logistic model is that increasing one of the independent variables multiplicatively scales the odds of the given outcome at a constant rate, with each independent variable having its own parameter; for a binary dependent variable this generalizes the odds ratio."}'
            """
        return response

    else:
        return "Not supported"

if __name__ == '__main__':
    app.run(host="0.0.0.0", port=5001, debug=True)

Host the server with port 5001 `app.run(host="0.0.0.0", port=5001, debug=True)`

`python main.py`

Reference - Flask

Testing with curl

Let's use a paragraph from wikipedia of the Logistic Regression page as an input of our curl command and pass it as an argument text (Double quote removed) to the model.

curl -X POST http://127.0.0.1:5001/keywords -H 'Content-Type: application/json' \
      -d '{"text": "Logistic regression is a statistical model that in its basic form uses a logistic function to model a binary dependent variable, although many more complex extensions exist. In regression analysis, logistic regression[1] (or logit regression) is estimating the parameters of a logistic model (a form of binary regression). Mathematically, a binary logistic model has a dependent variable with two possible values, such as pass/fail which is represented by an indicator variable, where the two values are labeled 0 and 1. In the logistic model, the log-odds (the logarithm of the odds) for the value labeled 1 is a linear combination of one or more independent variables (predictors); the independent variables can each be a binary variable (two classes, coded by an indicator variable) or a continuous variable (any real value). The corresponding probability of the value labeled 1 can vary between 0 (certainly the value 0) and 1 (certainly the value 1), hence the labeling; the function that converts log-odds to probability is the logistic function, hence the name. The unit of measurement for the log-odds scale is called a logit, from logistic unit, hence the alternative names. Analogous models with a different sigmoid function instead of the logistic function can also be used, such as the probit model; the defining characteristic of the logistic model is that increasing one of the independent variables multiplicatively scales the odds of the given outcome at a constant rate, with each independent variable having its own parameter; for a binary dependent variable this generalizes the odds ratio."}'

Demo

Results

    {
      "keywords": [
        "logistic model",
        "variable",
        "regression",
        "binary dependent",
        "labeled",
        "form",
        "odds",
        "exist",
        "basic",
        "complex",
        "indicator",
        "probability",
        "log-odds scale",
        "sigmoid function",
        "converts log-odds",
        "Mathematically",
        "scales",
        "alternative",
        "defining",
        "constant"
      ]
    }

The result is actually quite good given its unsupervised nature. We can see some important keywords like log-odds, sigmoid function and binary in the result.

FastAPI

Apart from Flask that we just introduced, there is another popular package to host API endpoints - FastAPI.

FastAPI is a modern, fast and popular web framework for building APIs based on standard Python type hints. It is a high performant package, and it is on par with some popular framework written in *NodeJS* and *Go*.

Let's try to host our keywords model again with FastAPI.

Key steps

Both Input and Output Object inherit pydantic.Basemodel object
Use python type hints str (input) and List[str] (output) to define field types of the objects
Use Objects as input/output parameter Response/Paragraph

# Input object with a text field
class Paragraph(BaseModel):
    text: str

# Output object with keywords as field
class Response(BaseModel):
    keywords: List[str]

@app.post("/keywords", response_model=Response)
def keywords_two(p: Paragraph):
    ...
    return Response(keywords=kw)

Code

from fastapi import FastAPI
from pydantic import BaseModel
from typing import List
import yake

# Input
class Paragraph(BaseModel):
    text: str

# Output
class Response(BaseModel):
    keywords: List[str]

app = FastAPI()

def ExtractKeywords(text):
    """ Extract 20 keywords from the input text """

    language = "en"
    max_ngram_size = 2
    deduplication_thresold = 0.3
    deduplication_algo = "seqm"
    windowSize = 1
    numOfKeywords = 20

    custom_kw_extractor = yake.KeywordExtractor(
        lan=language,
        n=max_ngram_size,
        dedupLim=deduplication_thresold,
        dedupFunc=deduplication_algo,
        windowsSize=windowSize,
        top=numOfKeywords,
        features=None,
    )
    kws = custom_kw_extractor.extract_keywords(text)
    keywords = [x[0] for x in kws]  # kws is in tuple format, extract the text part

    return keywords


  @app.post("/keywords", response_model=Response)
  def keywords(p: Paragraph):
      kw = ExtractKeywords(p.text)
      return Response(keywords=kw)

Host

a) Install fastapi and uvicorn

    pip install fastapi
    pip install uvicorn

b) Host FastAPI with uvicorn

uvicorn main:app --host 0.0.0.0 --port 5001 --reload --debug --workers 3

Documentation

FastAPI creates a documentation page for you by default using the Swagger UI. You can open the documentation page with http://localhost:5001/docs. If you follow the schema definition, you can have a nice looking API documentation with some examples as well.

Reference - FastAPI and Declare Request Example

Testing with Postman

Demo

Complete example

You can find the complete examples here - Flask and FastAPI

Final thoughts

Here we introduced two different frameworks (Flask and FastAPI) to serve our keyword extraction model on our local machine. While Flask being more popular among web developers, and FastAPI being more performant, it is both pretty easy to use.

Hopefully you can see how easy it is for you to host the model using the frameworks. If you have any questions or feedback, feel free to leave a comment.

I also write on my own blog (https://data-gulu.com). You can find more articles about python and machine learning there.

Happy Coding!

Reference:

Photo by Ilyuza Mingazova on Unsplash
YAKE - Github and Paper
Flask and FastAPI - Here and Here

Introduction to Jinja Template and Google Chart

billylkc — Mon, 06 Nov 2023 05:13:34 +0000

Introduction

In this post you will learn
Why we need a template system
Basic syntax of Jinja Template (Python Jinja2)
Generate a simple HTML report with Jinja and Google Chart (Google Chart)

Why we need a template system

While string formatting in Python is pretty straight forward e.g. print(f"Hello {name}"), it could be quite troublesome when we are writing some long messages with many variables, or sometimes we want to use Python function like if or for loop in the paragraph.

A template system in Python, or any programming language, is a tool that allows you to separate the presentation logic (Template) from business logic (Code) in your applications. It provides a way to define and generate content, such as HTML, XML, or plain text, by combining static template files with dynamic data. In this post, we will demonstrate how to generate some simple HTML report in the following section.

Basic Syntax of Jinja Template

Render your first jinja template

It looks very similar to string formatting at the first glance. We will first define a template t = Template(...) with variable {{ var_name }} , and then render it with the python variables t.render(var_name="sth").

from jinja2 import Template

t = Template('Hello, {{ name }}!')
print(t.render(name='John Doe'))

# Output:
"Hello, John Doe!"

Basic syntax

The syntax is almost the same as python with a little bit of syntactic sugar, we can use some conditional block via {% %}, and reference to the variable with {{ }}. And another useful thing to note is, we can add a dash - to the operator, so Jinja knows we dont want an additional line break, e.g. {%- -%} (No line break) vs {% %} (With line break).

a) Variables

Syntax: {{ foo }}, {{ foo.bar }}, {{ foo["bar"] }}

You can use a dot (.) to access attributes of a variable in addition to the standard Python __getitem__ "subscript" syntax ([]).

from jinja2 import Template

baz = "a"
foo = {}
foo["bar"] = "b"

t = Template("""
{{ baz }}
{{ foo.bar }}
{{ foo["bar"]}}
""")

print(t.render(
    foo=foo,
    baz=baz,
))

# Output
a
b
b

b) For loop

Syntax: {% for i in some_list %} {{ i }} {% endfor %} with line break and {%- for i in some_list -%} {{ i }} {% endfor %} without line break

from jinja2 import Template

some_students = ["john", "terry", "ken"]

t = Template("""
{%- for student in students -%}
<li> {{ student }} </li>
{% endfor %}
""")
print(t.render(students=some_students))

# output
<li> john </li>
<li> terry </li>
<li> ken </li>

c) If.. elif.. else

Syntax: {% if x = "a" %} {% elif x = "b" %} {% else %} {% endif %}

from jinja2 import Template

x = 40

# Additional spacing is added for readability
t = Template("""
{% if x >= 0 and x < 30 %}
x is larger than 0

{% elif x >= 30 %}
x is larger than 30

{% else %}
x is smaller than 30

{% endif %}
""")
print(t.render(x=x))

# output
x is larger than 30

d) Comments

Syntax: {# #} (with new line) or {#- -#} (without new line)

from jinja2 import Template

t = Template("""
{#- This is a comment. And it is not displayed in the output. -#}
Hello, {{ name }}!
""")
print(t.render(name='John Doe'))

# Output
Hello, John Doe!

e) Read template from a file

For better file management, we usually put the files into a ./templates folder. It helps to separate the presentation layer (template) and logical layer (code). The file structure would be something like this.

.
├── main.py
└── templates
    └── simple_report.html

First we put the template file under ./templates/simple<sub>report.html

{#- <!-- templates/simple_report.html --> -#}
<!DOCTYPE html>
<html>
  <body>
    <h1>Hi {{ name }}!</h1>
  </body>
</html>

Then we can read the templates with open("some_file.html", "r") as f: ... Template(f.read()) in main.py

# May mess up Jinja Template Hierarchy tho. In our simple case, it wont matter much
from jinja2 import Template

name = "John"

with open("templates/simple_report.html", "r") as f:
    template = Template(f.read())

rendered = template.render(name=name)
print(rendered)

Simple Reporting Templates with External Javascript Packages

After the long syntax introduction, we will jump straight to the use cases.

Sometimes when the buisness users are asking for some advance/deep-dive adhoc reports, there are not much report choices for a quick study, you can use

Excel file
Commercial products (e.g. Tableau, PowerBI, etc..)
Self host a Python server (e.g. matplotlib, seaborn, etc..)
A simple HTML file generated with Template and some Javascript Library (Below example)

Each of the option has it's pros and cons, a static file (like HTML) provides you an other options, if other commercial tools or hosting a python server is not available.

Google Chart Library

Google chart tools are powerful, simple to use, and free. And usually these kinds of Javascript chart library provides a richer (and customizable) analytic gallery for you to use, which sometimes it is quite difficult to do it in excel, like Sankey Diagram, Interval Plots and Tree Map.

Here I will use a simple bar chart as an example. And you will see how we can create a simple HTML file with bar chart (or other graphs) easily if you know how to use Jinja Template.

This is the outline of the steps to create a standalone HTML file with Google Chart

Copy the minimal example of Stacked bar chart (Google Charts)
Replace the data part with Jinja Template variable
Try it with dummy data
Replace it with real data. Here we will use the Daily Passenger Traffic for different Control Points from HK Government Open Data (data.gov.hk)
Render the Template, and export as HTML file

a) Stacked Bar Chart

Let's copy the minimal example of stack bar chart. I rename some variables name from the original example here (e.g. materialChart to chart, etc..)

The code is pretty easy to read. We import some chart library with <script type="text/javascript" src="https://www.gstatic.com/charts/loader.js"></script> and initialize the google chart object and callback function with with google.charts.setOnLoadCallback(drawChart);. It's quite straight forward even if you do not have much experience with Javascript.

<html>
  <head>
    <script type="text/javascript" src="https://www.gstatic.com/charts/loader.js"></script>
    <script type="text/javascript">

     google.charts.load('current', {packages: ['corechart', 'bar']});
     google.charts.setOnLoadCallback(drawChart);

     function drawChart() {
       var data = google.visualization.arrayToDataTable([
         ['City', '2010 Population', '2000 Population'],
         ['New York City, NY', 8175000, 8008000],
         ['Los Angeles, CA', 3792000, 3694000],
         ['Chicago, IL', 2695000, 2896000],
         ['Houston, TX', 2099000, 1953000],
         ['Philadelphia, PA', 1526000, 1517000]
       ]);

       var options = {
         chart: {
           title: 'Population of Largest U.S. Cities'
         },
         hAxis: {
           title: 'Total Population',
           minValue: 0,
         },
         vAxis: {
           title: 'City'
         },
         bars: 'horizontal'
       };
       var chart = new google.charts.Bar(document.getElementById('chart_div'));
       chart.draw(data, options);
     }
    </script>
  </head>

  <body>
    <div id="chart_div"></div>
  </body>
</html>

b) Replace with Jinja Template variables

Then we can replace some of the values with Template variable like {{ data }}, {{ title }}, {{ h_axis }}, etc..

from jinja2 import Template

t = Template(
"""
<html>
  <head>
    <script type="text/javascript" src="https://www.gstatic.com/charts/loader.js"></script>
    <script type="text/javascript">

     google.charts.load('current', {packages: ['corechart', 'bar']});
     google.charts.setOnLoadCallback(drawChart);

     function drawChart() {
       var data = google.visualization.arrayToDataTable(
         {{ data }}
       );

       var options = {
         chart: {
           title: '{{ title }}'
         },
         hAxis: {
           title: '{{ h_axis }}',
           minValue: 0,
         },
         vAxis: {
           title: '{{ v_axis }}'
         },
         bars: 'horizontal'
       };
       var chart = new google.charts.Bar(document.getElementById('chart_div'));
       chart.draw(data, options);
     }
    </script>
  </head>

  <body>
    <div id="chart_div"></div>
  </body>
</html>
"""
)

c) Test with Dummy Data

If we go back and study the embeded data in the json, we will see the data we need is a list of list, with 3 columns (One categorical and two numerics). So we we create the dataframe accordingly, with ["Control Point", "Arrival", "Departure"] as our column for our traffic data.

# The embeded data in the original json. A list of list with each record as an element.
[
  ['City', '2010 Population', '2000 Population'],
  ['New York City, NY', 8175000, 8008000],
  ['Los Angeles, CA', 3792000, 3694000],
  ['Chicago, IL', 2695000, 2896000],
  ['Houston, TX', 2099000, 1953000],
  ['Philadelphia, PA', 1526000, 1517000]
]

Let's create a dummy dataframe, and convert it to list of list object.

import pandas as pd

df = pd.DataFrame({
    "Control Point": ["Airport", "Lok Ma Chau", "Lo Wu"],
    "Arrival": [154, 120, 40],
    "Departure": [21, 40, 32],
})
print(df.head())

# Convert dataframe to List of list, and insert to column header as the first element.
data = df.values.tolist()
data.insert(0, df.columns.tolist())
print(data)

# df
  Control Point  Arrival  Departure
0       Airport      154         21
1   Lok Ma Chau      120         40
2         Lo Wu       40         32

# data
[
  ['Control Point', 'Arrival', 'Departure'], 
  ['Airport', 154, 21], 
  ['Lok Ma Chau', 120, 40], 
  ['Lo Wu', 40, 32]
]

d) Replace with real data

Here we will get the 2022 passenger traffic data from HK Government Open Data (data.gov.hk). And aggregate the data from daily records to yearly records. For the df operation, you can check out my previous post on Common Pandas Functions.

import pandas as pd

# Get the 2022 arrival/departure data
url = "https://www.immd.gov.hk/opendata/eng/transport/immigration_clearance/statistics_on_daily_passenger_traffic.csv"
df = pd.read_csv(url)
df = df[["Date", "Control Point", "Arrival / Departure", "Total"]]
df['Date'] = pd.to_datetime(df['Date'], format="%d-%m-%Y")
df_subset = df[(df['Date'] >= '2022-01-01')
               & (df['Date'] <= '2022-12-31')].reset_index(drop=True)  # 2022 data only, for simplicity
print(df_subset)

# Group by Control Point. We can safely ignore date, as we are only using 2022 data
df_agg = df_subset.groupby(["Control Point", "Arrival / Departure"]).agg(Total=("Total", "sum"),).reset_index()
df_agg = df_agg[df_agg.Total > 0].reset_index(drop=True)
print(df_agg)

# Long to wide. pivot_table is used here, just for the fill_value argument
df_wide = df_agg.pivot_table(
    index=["Control Point"],
    columns="Arrival / Departure",
    values=["Total"],
    aggfunc="sum",
    fill_value=0,
)

df_wide_flattened = df_wide.copy()
df_wide_flattened.columns = ["_".join(x) for x in df_wide_flattened.columns.to_flat_index()]
df_wide_flattened.reset_index(inplace=True)
print(df_wide_flattened)

data = df_wide_flattened.values.tolist()
data.insert(0, df_wide_flattened.columns.tolist())  # Insert back the column header
print(data)



# df_subset (2022 Data)
            Date                   Control Point Arrival / Departure  Total
0     2022-01-01                         Airport             Arrival    628
1     2022-01-01                         Airport           Departure    778
2     2022-01-01  Express Rail Link West Kowloon             Arrival      0
3     2022-01-01  Express Rail Link West Kowloon           Departure      0
...

# df_agg (Aggregate to Year level)
                    Control Point Arrival / Departure    Total
0                         Airport             Arrival  1957891
1                         Airport           Departure  2175626
8                  Heung Yuen Wai             Arrival      301
10  Hong Kong-Zhuhai-Macao Bridge             Arrival    77075
...

# df_wide_flattened (Long to wide table)
                   Control Point  Total_Arrival  Total_Departure
0                        Airport        1957891          2175626
1                 Heung Yuen Wai            301                0
2  Hong Kong-Zhuhai-Macao Bridge          77075           114504
3        Kai Tak Cruise Terminal           8233             3610
4                   Shenzhen Bay         485248           439669

# data (Final List of List object for input of the graph)
[['Control Point', 'Total_Arrival', 'Total_Departure'], ['Airport', 1957891, 2175626], ['China Ferry Terminal', 0, 0], ..]

e) Render with Jinja Template, and export to html file

Finally, we can render the variables with Jinja, and export to the HTML file.

from jinja2 import Template
import pandas as pd

# Get the 2022 arrival/departure data, hopefully the URL will not break
url = "https://www.immd.gov.hk/opendata/eng/transport/immigration_clearance/statistics_on_daily_passenger_traffic.csv"
df = pd.read_csv(url)
df = df[["Date", "Control Point", "Arrival / Departure", "Total"]]
df['Date'] = pd.to_datetime(df['Date'], format="%d-%m-%Y")
df_subset = df[(df['Date'] >= '2022-01-01')
               & (df['Date'] <= '2022-12-31')].reset_index(drop=True)  # 2022 data only, for simplicity

# Group by Control Point. We can safely ignore date, as we are only using 2022 data
df_agg = df_subset.groupby(["Control Point", "Arrival / Departure"]).agg(Total=("Total", "sum"),).reset_index()
df_agg = df_agg[df_agg.Total > 0].reset_index(drop=True)

# Long to wide
df_wide = df_agg.pivot_table(
    index=["Control Point"],
    columns="Arrival / Departure",
    values=["Total"],
    aggfunc="sum",
    fill_value=0,
)

df_wide_flattened = df_wide.copy()
df_wide_flattened.columns = ["_".join(x) for x in df_wide_flattened.columns.to_flat_index()]
df_wide_flattened.reset_index(inplace=True)

# List of List
data = df_wide_flattened.values.tolist()
data.insert(0, df_wide_flattened.columns.tolist())  # Insert back the column header

# Create template
t = Template("""
<html>
  <head>
    <script type="text/javascript" src="https://www.gstatic.com/charts/loader.js"></script>
    <script type="text/javascript">

     google.charts.load('current', {packages: ['corechart', 'bar']});
     google.charts.setOnLoadCallback(drawChart);

     function drawChart() {
       var data = google.visualization.arrayToDataTable(
       {{ data }}
       );

       var options = {
         chart: {
           title: '{{ title }}'
         },
         hAxis: {
           title: '{{ h_axis }}',
           minValue: 0,
         },
         vAxis: {
           title: '{{ v_axis }}'
         },
         bars: 'horizontal'
       };
       var chart = new google.charts.Bar(document.getElementById('chart_div'));
       chart.draw(data, options);
     }
    </script>
  </head>

  <body>
    <div id="chart_div"></div>
  </body>
</html>
""")

# Render template, and save to a html file
output = t.render(
    data=data,
    title="Inbound and Outbound Passenger at different Control Points (Year 2022)",
    h_axis="Passengers",
    v_axis="Control Point",
)
with open('output.html', 'w') as f:
    f.write(output)

Output - output.html

Final Thoughts

There is a lot of other Javascript chart packages (e.g. Vega Chart, HighCharts, etc..), you can easily create some pretty charts with the techniques here with Jinja Template. As it is in HTML format, you can easily include some Insight session as well for some quick comments for your report.

Remarks: Just be careful for not leaking any sensitive data, as a static report is difficult to do access control or audit logging.

I also write on my own blog (https://data-gulu.com). You can find more articles about python and machine learning there.

Happy Coding!

Parse JSON API response in Go

billylkc — Tue, 06 Jul 2021 15:40:45 +0000

(You can find my original post here - data-gulu.com)

Introduction

When you are hosting your model result as a micro-services, or scrapping data from websites, you will often find yourself working on some Restful API JSON object. In this post, I will show you how easily we can work with JSON data in Go.

Overview

Get JSON response from sample API hosting site - reqres
Generate go struct from the response - json-to-go
Unmarshal JSON response to go struct
Loop through the struct and print data from the result

Get Requests



package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
)

func main() {
    resp, err := http.Get("https://reqres.in/api/users?page=2")
    if err != nil {
        fmt.Println("No response from request")
    }
    defer resp.Body.Close()
    body, err := ioutil.ReadAll(resp.Body) // response body is []byte
    fmt.Println(string(body))              // convert to string before print
}

Result JSON



{"page":2,"per_page":6,"total":12,"total_pages":2,"data":[{"id":7,"email":"michael.lawson@reqres.in","first_name":"Michael","last_name":"Lawson","avatar":"https://reqres.in/img/faces/7-image.jpg"},{"id":8,"email":"lindsay.ferguson@reqres.in","first_name":"Lindsay","last_name":"Ferguson","avatar":"https://reqres.in/img/faces/8-image.jpg"},{"id":9,"email":"tobias.funke@reqres.in","first_name":"Tobias","last_name":"Funke","avatar":"https://reqres.in/img/faces/9-image.jpg"},{"id":10,"email":"byron.fields@reqres.in","first_name":"Byron","last_name":"Fields","avatar":"https://reqres.in/img/faces/10-image.jpg"},{"id":11,"email":"george.edwards@reqres.in","first_name":"George","last_name":"Edwards","avatar":"https://reqres.in/img/faces/11-image.jpg"},{"id":12,"email":"rachel.howell@reqres.in","first_name":"Rachel","last_name":"Howell","avatar":"https://reqres.in/img/faces/12-image.jpg"}],"support":{"url":"https://reqres.in/#support-heading","text":"To keep ReqRes free, contributions towards server costs are appreciated!"}}

Convert JSON response

You can go to this site - JSON to go to convert JSON response to Go struct very easily.

Unmarshal JSON to Go struct

You can then unmarshal the []byte from the GET response to the Response struct that we just auto-generated



// Generated go struct
type Response struct {
    Page       int `json:"page"`
    PerPage    int `json:"per_page"`
    Total      int `json:"total"`
    TotalPages int `json:"total_pages"`
    Data       []struct {
        ID        int    `json:"id"`
        Email     string `json:"email"`
        FirstName string `json:"first_name"`
        LastName  string `json:"last_name"`
        Avatar    string `json:"avatar"`
    } `json:"data"`
    Support struct {
        URL  string `json:"url"`
        Text string `json:"text"`
    } `json:"support"`
}

// snippet only
var result Response
if err := json.Unmarshal(body, &result); err != nil {   // Parse []byte to go struct pointer
    fmt.Println("Can not unmarshal JSON")
}
fmt.Println(PrettyPrint(result))

Response struct preview (partial)



{
    "page": 2,
    "per_page": 6,
    "total": 12,
    "total_pages": 2,
    "data": [
    {
        "id": 7,
        "email": "michael.lawson@reqres.in",
        "first_name": "Michael",
        "last_name": "Lawson",
        "avatar": "https://reqres.in/img/faces/7-image.jpg"
    },
        {
        "id": 8,
        "email": "lindsay.ferguson@reqres.in",
        "first_name": "Lindsay",
        "last_name": "Ferguson",
        "avatar": "https://reqres.in/img/faces/8-image.jpg"
    }
    ]
}

Finally loop through the data node and print out the FirstName of the result



// Loop through the data node for the FirstName

for _, rec := range result.Data {

    fmt.Println(rec.FirstName)

}

Complete code



package main

import (

    "encoding/json"

    "fmt"

    "io/ioutil"

    "net/http"

)

type Response struct {

    Page       int json:"page"

    PerPage    int json:"per_page"

    Total      int json:"total"

    TotalPages int json:"total_pages"

    Data       []struct {

        ID        int    json:"id"

        Email     string json:"email"

        FirstName string json:"first_name"

        LastName  string json:"last_name"

        Avatar    string json:"avatar"

    } json:"data"

    Support struct {

       URL  string json:"url"

       Text string json:"text"

    } json:"support"

}

func main() {

<span class="c">// Get request</span>
<span class="n">resp</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">http</span><span class="o">.</span><span class="n">Get</span><span class="p">(</span><span class="s">"https://reqres.in/api/users?page=2"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
    <span class="n">fmt</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="s">"No response from request"</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">defer</span> <span class="n">resp</span><span class="o">.</span><span class="n">Body</span><span class="o">.</span><span class="n">Close</span><span class="p">()</span>
<span class="n">body</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">ioutil</span><span class="o">.</span><span class="n">ReadAll</span><span class="p">(</span><span class="n">resp</span><span class="o">.</span><span class="n">Body</span><span class="p">)</span> <span class="c">// response body is []byte</span>

<span class="k">var</span> <span class="n">result</span> <span class="n">Response</span>
<span class="k">if</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">json</span><span class="o">.</span><span class="n">Unmarshal</span><span class="p">(</span><span class="n">body</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">result</span><span class="p">);</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>  <span class="c">// Parse []byte to the go struct pointer</span>
    <span class="n">fmt</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="s">"Can not unmarshal JSON"</span><span class="p">)</span>
<span class="p">}</span>

<span class="c">// fmt.Println(PrettyPrint(result))</span>

<span class="c">// Loop through the data node for the FirstName</span>
<span class="k">for</span> <span class="n">_</span><span class="p">,</span> <span class="n">rec</span> <span class="o">:=</span> <span class="k">range</span> <span class="n">result</span><span class="o">.</span><span class="n">Data</span> <span class="p">{</span>
    <span class="n">fmt</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="n">rec</span><span class="o">.</span><span class="n">FirstName</span><span class="p">)</span>
<span class="p">}</span>



}

// PrettyPrint to print struct in a readable way

func PrettyPrint(i interface{}) string {

    s, _ := json.MarshalIndent(i, "", "\t")

    return string(s)

}

Demo

(You can find my original post here - data-gulu.com)

DEV Community: billylkc

Static site crawling with goq

Table of Contents

Static site crawling with goq

Introduction

Usage

Static site crawling

Result

Other Demo

a) Wikipedia

1. Study the *css selector* of the info that we need

2. Create a *struct* with the *goquery tag*

3. *Decode* into Go struct with the GET response

Print the output

5. Demo

b) OpenRice

Full code

Final thoughts

Hosting a Keyword Extraction Model with Flask and FastAPI

Table of Contents

Hosting a Keyword Extraction Model with Flask and FastAPI API Model

Introduction

Overview

Yet Another Keyword Extractor

Main Features

Heuristic Approach

Our ML model

Installation

Flask API

Installation

Serve with /keywords endpoint

Host the server with port 5001 app.run(host="0.0.0.0", port=5001, debug=True)

Testing with curl

Demo

FastAPI

Key steps

Code

Host

Documentation

Testing with Postman

Demo

Complete example

Final thoughts

Introduction to Jinja Template and Google Chart

Introduction

Why we need a template system

Basic Syntax of Jinja Template

Basic syntax

a) Variables

b) For loop

c) If.. elif.. else

d) Comments

e) Read template from a file

Simple Reporting Templates with External Javascript Packages

Google Chart Library

a) Stacked Bar Chart

b) Replace with Jinja Template variables

c) Test with Dummy Data

d) Replace with real data

e) Render with Jinja Template, and export to html file

Final Thoughts

Parse JSON API response in Go

Introduction

Overview

Get Requests

Result JSON

Convert JSON response

Unmarshal JSON to Go struct

Complete code

Demo

1. Study the css selector of the info that we need

2. Create a struct with the goquery tag

3. Decode into Go struct with the GET response

Serve with `/keywords` endpoint

Host the server with port 5001 `app.run(host="0.0.0.0", port=5001, debug=True)`