DEV Community: Effy L.H.

'Queendom2Data' -A data visualization web app built by React+Flask

Effy L.H. — Mon, 09 May 2022 14:52:38 +0000

Introduction

Queendom2data is a data visualization app displays the YouTube views count of the ongoing K-pop reality survival show ‘Queendom2’.

App Link: www.qd2data.xyz
GitHub Repo: https://github.com/monkeyapple/queendom2data

Techstack

Programming Languages: Python, Javascript,SQL,CSS,HTML
Frameworks: React(Front-end),Flask(Back-end),Bootstrap(CSS)
Database: PostgreSQL
Automation: HerokuScheduler
Data API: YouTube Data API

WorkFlow

Preview

Current Version

v1.0.0

Technical details about this project

I also wrote two articles for this project:

New Features Updated: Udemy Syllabus Scraper v1.0.2

Effy L.H. — Wed, 06 Apr 2022 08:50:29 +0000

Introduction

Syllabus-Scraper is a web application for generating the Markdown syllabus of Udemy courses by using Flask, jQuery, UdemyAPI, and PostgreSQL.

App link: www.syllabuscraper.com

Preview

Current Version

v1.0.2

New Features

A data table added at the bottom of the page, which is a convenient way for users to check the 10 most recent searched courses by other users. Users can choose interested courses and copy the MarkDown syllabus directly inside the table.

Technical details

GitHub Repo: https://github.com/monkeyapple/Syllabus-Scraper
I also wrote a article with all the details about this Web App, please check: link

Display recent queries on Web Page from PostgreSql database with Flask+psycopg2+Bootstrap

Effy L.H. — Wed, 06 Apr 2022 08:25:12 +0000

This is a simple tutorial for people who is just stepping into Flask framework and trying to display database queries on the screen.

GitHub repo for this tutorial:Display recent queries on Web Page from PostgreSql database with Flask+psycopg2+Bootstrap

Sample dataset

Here we have a sample database contains a table called ‘game’ which stores 18 games and their last visit records.

Goal

Our goal is to display the 5 most recent queries on the screen

Preview of the result

Preparations

Create an HTML file called index.html

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="utf-8">
    <title>Display recent database queries</title>
<body>
    <h1>Hello</h1>

</body>
</html>

Create a file called run.py, then define a route to render the index.html file we just created

from flask import Flask,render_template

app=Flask(__name__)

############################################
        # Routes
##########################################
@app.route('/')
def index():
    return render_template('index.html')

if __name__=='__main__':
    app.run(debug=True)

Run in the command line and the result is shown as below:

python3 run.py

Let’s get started!

Back-End

Create a file called **database.ini **in order to safely store database configurations:

[postgresql]
host=localhost
database=games
user=postgres
password=123456
port=5432

Create a file called config.py in order to safely call the configurations defined inside the database.ini

from configparser import ConfigParser

def config(filename='/Users/Project/DisplayQueries_With_Flask_jQuery_Bootstrap_PostgreSQL/database.ini', section='postgresql'):
    parser = ConfigParser()
    parser.read(filename)

    db = {}
    if parser.has_section(section):
        params = parser.items(section)
        for param in params:
            db[param[0]] = param[1]
    else:
        raise Exception('Section {0} not found in the {1} file'.format(section, filename))

    return db

Let’s update the run.py

Import needed modules: config(we just created in step 1) , psycopg2 (psycopg2 is a popular PostgreSQL database adapter )
Connect with our local database by using the config module we defined before, then execute a SQL statement to extract the 5 most recent rows from the database(sort the records according to the ‘last_visit’ column in descending order)

from flask import Flask,render_template
#1.
from config import config
import psycopg2
app=Flask(__name__)


###########################################
        #Route
#########################################

@app.route('/')
def index():
    conn=None
    rowResults=None
    #2.
    try:
        params = config()
        conn = psycopg2.connect(**params)
        cur=conn.cursor()
        order_statement='SELECT * FROM game ORDER BY last_visit DESC LIMIT 5;'
        cur.execute(order_statement)
        rowResults=cur.fetchall()
        print(rowResults)
    except(Exception, psycopg2.Error) as error:
        print(error)

    finally:
        if conn is not None: 
            cur.close()
            conn.close()
            print("database connection is now closed")
    return render_template('index.html')

if __name__=='__main__':
    app.run(debug=True)

Check the results we got

The 5 records we need are as below:

The 5 records we actually got:

(Which means our SQL operations successfully executed!)

Outputs:
[
(4, 'The Legend of Zelda: Breath of the Wild', datetime.datetime(2021, 4, 14, 22, 6, 12)), 
(3, 'Horizon Forbidden West', datetime.datetime(2021, 4, 5, 2, 43, 2)), 
(8, 'The Elder Scrolls V: Skyrim', datetime.datetime(2021, 3, 27, 21, 31, 4)), 
(5, 'The Witcher 3: Wild Hunt', datetime.datetime(2021, 3, 25, 6, 8, 23)), 
(16, 'Final Fantasy', datetime.datetime(2021, 3, 21, 17, 52, 47))
]

Since now, we’ve finished all the back-end scripts, let’s move on to the front-end and display the results on the web page:

Front-End

Add a line at the end of the index() function in the run.py, the render_template method will help us to render the defined index.html on the web page, meanwhile, we pass in a variable called “recentRecords” and assign the ‘rowResults’ (query results we retreived from the databse) onto it.

from flask import Flask,render_template
from config import config
import psycopg2
app=Flask(__name__)


###########################################
        #Route
#########################################

@app.route('/')
def index():
    conn=None
    rowResults=None
    try:
        params = config()
        conn = psycopg2.connect(**params)
        cur=conn.cursor()
        order_statement='SELECT * FROM game ORDER BY last_visit DESC LIMIT 5;'
        cur.execute(order_statement)
        rowResults=cur.fetchall()

    except(Exception, psycopg2.Error) as error:
        print(error)

    finally:
        if conn is not None: 
            cur.close()
            conn.close()
            print("database connection is now closed")
    return render_template('index.html',recentRecords=rowResults)

if __name__=='__main__':
    app.run(debug=True)

Render query results “recentRecords” by using Jinja’s for loops syntax
Control flow: if the “recentRecords” is equals to an empty array [], displaying a text “Not enough data” on the screen. Otherwise, displaying a bootstrap table with values we got from the database.
About the style of table, I just simply copy&paste the bootstrap default CSS classes: “table-hover table-dark table-striped”, or you can reference the official document to choose the style you like: https://getbootstrap.com/docs/5.1/content/tables/

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="utf-8">
    <title>Display recent database queries</title>
    <!-- CSS stylesheet -->
    <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.1/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-+0n0xVW2eSR5OomGNYDnhzAbDsOXxcvSN1TPprVMTNDbiYZCxYbOOl7+AMvyTG2x" crossorigin="anonymous">
<body>
    <table class="table table-hover table-dark table-striped" cellspacing="0">
        <thead>
        <tr>
            <th scope="col">#</th>
            <th scope="col" style="text-align:left">Game</th>
            <th scope="col">Visit Date</th>
        </tr>
        </thead>
        <tbody>
        {% if recentRecords==[] %}
        <p>
            Not enough data
        </p>
        {% else %}
        {% for i in range(0,5) %}
        <tr>
            <th scope="row">{{i+1}}</th>
            <td align="left">
                <p>{{recentRecords[i][1]}}</p>
            </td>
            <td>
                <p>{{recentRecords[i][2]}}</p>
            </td>
        </tr>
        {% endfor %}
        {% endif %}

        </tbody>
    </table>
</body>
</html>

Local Test

Let’s run the script and access the address http://127.0.0.1:5000 on your browser:
python3 run.py

It works!🥳

The Real Project Example

Another motivation I wrote this tutorial is: I just extended my launched Web App with the same functionality mentioned in this tutorial, which you can check it at:

App link: link
And I wrote an article about the Web App: link

That's all for this tutorial. Thanks!
If you have any questions, feel free to comment.

Udemy Syllabus Scraper Released

Effy L.H. — Sun, 02 Jan 2022 08:29:58 +0000

Introduction

Syllabus-Scraper is a web application for generating the Markdown syllabus of Udemy courses by using Flask, jQuery, UdemyAPI, and PostgreSQL.

App link: www.syllabuscraper.com
GitHub repo:https://github.com/monkeyapple/Syllabus-Scraper

Current Version

V1.0

Features

Generating Markdown syllabus of Udemy courses

How to use

Copy the course link from Udemy
Paste the course link in the search box

Preview

Technical details about this project

I wrote a article for this project:link

Internal Implementations of List.extend() and List.append() operations in Python

Effy L.H. — Sun, 09 Aug 2020 12:28:26 +0000

Let’s take a deep dive into the internal implementation of List.extend() & List.append() operations:

Suppose the list nums[] contains 7 integer elements, and we're about to execute lines as below:

Sample 1

result=[]
    for i in range(n):
        result.append(nums[0])
        result.append(nums[1])
return result

Sample 2

result=[]
    for i in range(n):
        result.extend([nums[0],nums[1]])
return result

Analysis of Internal Implementation

Sample 1

When we execute result.append(nums[0]),result.append(nums[1]), the internal implementation will be like as below:

List result get reference to num[0]which actually pointed to the integer instance 'int1'
Then list result get reference to num[1]which actually pointed to the integer instance 'int2'

Sample 2

When we execute result.extend([nums[0],nums[1]]), the internal implementation will be like as below:

Contents inside the parentheses of .extend() function can be treated as a temp list which actually pointed to the same integer objects as list nums
Then, the list result receive those references to those integers through that temp list

This post is also a complement to my previous post:
https://dev.to/effylh/leetcode-1470-shuffle-the-array-3ok5

LeetCode:442. Find All Duplicates in an Array

Effy L.H. — Sun, 09 Aug 2020 04:20:38 +0000

https://leetcode.com/problems/find-all-duplicates-in-an-array/

Solution

set is an collection of elements without duplicates, we use this property to solve the problem:

Execute an iteration cross the given array, meanwhile:

check the length of current set
add current element from the array into the set
go back to 1., if there is no change in the length, then it indicates that the added element is duplicated

Submissions

faster than 87.10%

class Solution:
    def findDuplicates(self, nums: List[int]) -> List[int]:
        myset=set()
        result=[]
        for i in nums:
            length=len(myset)
            myset.add(i)
            if length==len(myset):
                result+=[i]
        return result

LeetCode:680. Valid Palindrome II

Effy L.H. — Sun, 09 Aug 2020 03:38:51 +0000

https://leetcode.com/problems/valid-palindrome-ii/submissions/

Solution:Array

The goal of this problem is to find the 'the character which needs to be deleted',let's denote it as 'key'. The given string will be shown as a pattern of Palindrome after located and deleted the 'key',

Find the 'key'

We try to check the comparison of head & tail elements of the string, if they are identical characters, check the next pair, otherwise, one of them will be the 'key' we want to find.

(if the length of string is n, in this way, the time complexity can be reduced to n/2 )

Exception:but here is a special scenario will break the algorithm of 'head-tail- comparison',for example: 'abbaa'.

thus we design flow control as:

if the 'key' located at head or tail, return True
else we begin to execute our 'head-tail-comparison'.

Details of head-tail-comparison

(psuedocode)
if head!=tail:
  Construct a named 'List1'(removded head) 
  Construct a named 'List2'(removded tail)

then:
if ConstructList1==(in reversed ordered)ConstructList1-->return True
if ConstructList2==(in reversed ordered)ConstructList2-->return True

Submission

faster than 91.08%

class Solution:
    def validPalindrome(self, s: str) -> bool:
#Step1:
        if s[1:] == s[1:][::-1] or s[0:-1] == s[0:-1][::-1]:
            return True
#Step2:
        else:
            for i in range((len(s) + 1) // 2):
                if s[i] != s[-(i + 1)]:
                    check1 = s[0:i] + s[(i + 1):]
                    check2 = s[0: -(i + 1)] + s[-i:]
                    if check1 == check1[::-1] or check2 == check2[::-1]:
                        return True
                    else:
                        return False
            return True

LeetCode:125. Valid Palindrome

Effy L.H. — Fri, 07 Aug 2020 12:53:34 +0000

https://leetcode.com/problems/valid-palindrome/

Solutions

step1: apply .isalnum() on each character for filtering out all the alphanumeric characters in the given string .
step2: compare filtered list with the one in reversed order, if they're identical with each other, we got the string with pattern of palindrome

Submissions

faster than 98.95%

class Solution:
    def isPalindrome(self, s: str) -> bool:
        result=''
        for i in s:
            if i.isalnum():
                result+=i.lower()
        if result==result[::-1]:
            return True
        return False

TOC-generator of Bear Note

Effy L.H. — Tue, 04 Aug 2020 14:04:22 +0000

I'm a person who relies deeply on notes app, mostly I use Bear Note for taking notes for tech books or courses. When a note became huge and included enormous topics and headings inside, TOC(Table Of Contents) will be very useful for an efficient searching. So I created this simple tool for generating TOC in Bear Note.

User Interface

Cause it's just a simple tool, I quickly built the UI with simple settings in QtDesigner without any further edits.

Dev Workflow

Step1: Export identifier & html page of that note.[operations inside Bear Note]

Reference to Bear Note API document, we define a method to generate TOC with valid jump links inside Bear Note.
Ref:https://bear.app/faq/X-callback-url%20Scheme%20documentation/

def textFormat(self, aster, insert, factor):
    return '\t' * factor+aster+' '+'[' + insert + ']' + '(' + self.bearNoteUrl(insert) + ')\n'

Step2: Hit 'Run' button to trigger the scraping process

Use beautifulSoup to extract Headings from HTML file:

Cause the HTML code exported from Bear Note is very clean, so html.parser is enough to reach our goal, and no need to import extra module library such as 'lxml'
Search all lines of the html code and generate appropriate links which matched with the conditions (here conditions are all types of Headings,from'h1'to'h5')

def scraping(self): 
    self.ui.textBrowser.clear()
    self.scrapResult = ''
    self.asteriskAdd()
    if self.htmlFilePath[0] != '' and len(self.ui.identifier.text()) != 0:
        url = open(self.htmlFilePath[0], encoding='utf-8', errors='ignore')
        bs = BeautifulSoup(url, 'html.parser')
        self.collectList = bs.find_all(['h1', 'h2', 'h3', 'h4'])
        self.scrapResult = "# Table of Contents\n"

        for i in self.collectList:
            if i.name == 'h1':
                insert = i.get_text()
                self.scrapResult+=self.textFormat(self.aster1,insert,0)
            elif i.name == 'h2':
                insert = i.get_text()
                self.scrapResult+=self.textFormat(self.aster2,insert,1)
            elif i.name == 'h3':
                insert = i.get_text()
                self.scrapResult+=self.textFormat(self.aster3,insert,2)
            elif i.name == 'h4':
                insert = i.get_text()
                self.scrapResult+=self.textFormat(self.aster4,insert,3)
            elif i.name == 'h5':
                insert = i.get_text()
                self.scrapResult+=self.textFormat(self.aster5,insert,4)
        self.scrapResult += '***\n'

        self.ui.textBrowser.append(self.scrapResult)

    else:
        QMessageBox.warning(self, 'Warning', 'identifier or html file needed!')

Step3: Markdown texts of TOC will be shown in text browser at the bottom, then users could copy & paste them into Bear Note

Operation Screenshot

Results

Sample of generated TOC in Bear Note

Deployment

I use Pyinstaller to create distributable .EXE file for Windows users and .app for Mac users(find them in "dist" folder). But there is one thing we needs to pay attention: The default .spec setting generated from Pyinstaller will lead to low resolution rendering in Mac Retina. We need to add one more line into .spec file to solve this problem.
Reference:https://pyinstaller.readthedocs.io/en/stable/spec-files.html

info_plist={'NSHighResolutionCapable': 'True'}

Project link

You can find the source code and executable files here:
https://github.com/monkeyapple/TOC-Generator-for-Bear-Note

What's next

provide more customization options like:ordered Lists
seek better solution for packaging: Pyinstaller generated 70M executable file for Mac seems too much, compare with the Windows version which is 33M.

LeetCode 1470. Shuffle the Array

Effy L.H. — Fri, 24 Jul 2020 13:46:39 +0000

Solution

link
The logic of this problem is quite simple:

treat the given array which is divided at the position n as two parts.
use n as the iterator(reduce the cost of For Loop to O(len(array)/2)), then add each element into the result array

I made two submissions to solve this problem, all the lines are the same except one line inside the For Loop , which made a huge difference in the time complexity.

Submissions

faster than 39.93%:

class Solution:
    def shuffle(self, nums: List[int], n: int) -> List[int]:
        result=[]
        for i in range(n):
            result+=[nums[i],nums[i+n]]
        return result

faster than 95.78%:

class Solution:
    def shuffle(self, nums: List[int], n: int) -> List[int]:
        result=[]
        for i in range(n):
            result+=[nums[i]]+[nums[i+n]]
        return result

Analysis

result+=[nums[i],nums[i+n]] has the same effect as:

result.extend([nums[i],nums[i+n]])
# O(k),k is the number of elements inside the extending list

result+=[nums[i]]+[nums[i+n]]has the same effect as:

result.append(nums[i]) # O(1)
result.append(nums[i+1]) # O(1)

I wrote a complement post to this topic:
https://dev.to/effylh/internal-implements-of-list-extend-and-list-append-operations-in-python-4p8j

Summary

We're familiar with those methods, but we still need to pay attention about the time complexity and make the most appropriate choice.
Method Time Complexity:
append() O(1)
insert() O(n)
extend() O(k)