DEV Community: Sam Zhang

I made a new web-components framework

Sam Zhang — Mon, 12 Sep 2022 11:55:03 +0000

Recently I've made a frontend framework based on Web Components called Leafjs.

GitHub: https://github.com/samzhangjy/leafjs
Docs: https://leafjs.samzhangjy.com/

What is Leafjs?

Leafjs is a lightweight frontend framework built based on web components.

Using web components allows you to create components natively - supported by most modern browsers. Unlike React and Vue, Leafjs's component system is totally built on top of the web components system, meaning that you can even use your components through native tags inside a plain HTML file as long as importing your bundled code.

And for sure, Leafjs implements a simple reactivity system that allows two-way binding, just like most frameworks do.

Getting Started

Leafjs provides a out-of-the-box bundle tool for you to use. Run the following command in your shell to create a new Leafjs app:

$ npx create-leafjs-app my-app

Then install the dependencies using either Yarn or NPM. After that, run:

$ npm run dev

To start a hot-reloading development server.

Or if you want to play Leafjs online, here's a CodeSandbox:

For more information please refer to the documentation.

How do you think about Leafjs? Tell me and I'll keep improving it!

I built a terminal-like portfolio with Vue 3

Sam Zhang — Sat, 04 Jun 2022 06:13:42 +0000

Hey guys, I'm Sam Zhang.

Edit: since a lot of you are commenting about it, I would like to mention one thing though that this "portfolio" is just made for fun and not recommended to use in a real world application if you're really looking for a job.

Recently I've updated my personal website. Originally I used LogTure as my primary website, but I think it's not "cool" enough.

So I started building a cooler one. After Googling around, I decided to make a terminal-like site similar to yasfu.net with Vue 3.

Vue 3 is still new to me though. I mainly work on React projects recently and didn't explore the new features introduced in Vue 3. So this time I decided to give it a try. And after struggling with errors, I finally finished this project.

Demo: https://samzhangjy.com/

GitHub: https://github.com/samzhangjy/VueTerm

And of course, feel free to report issues to me!

However there're some downsides about this project. One is that the current bash-like command system has some trouble handling complicated paths. So for example, when you type in cd ../folder/subfolder will cause an inner failure of cd.

Recursive might be a good way to solve this problem and I'll try to implement it into this project in the near future.

And here is the end of the article. Feel free to play around with my project!

Restful CRUD with Golang for beginners

Sam Zhang — Thu, 02 Jun 2022 13:27:40 +0000

Hey guys, it's Sam Zhang.

So in the previous post we finished a simple Hello World request using Golang and Gin. Today it's time for us to build something more complex - CRUD with Gin!

Getting started
Installing GORM
Defining database models
Connecting to database
RESTful APIs
- Create
- Read
- Update
- Delete
Conclusion

Getting started

This tutorial is made for developers who have some experience in programming but relatively new to Go/Gin. You might want to learn some common programming concepts if you're a total beginner.

So to start up, you'll need to have the hello world program finished from our last post. Or of course you could clone it from GitHub to continue.

Then let's get started!

CRUD stands for Create, Read, Update and Delete. Those are the four basic operations in the database and we will be implementing it into our Go app today.

When dealing with databases, you could just write some plain old SQL commands and execute directly it using the database drivers. But there's a problem with it: SQL injection attack. So using ORMs is normally a better option. We will be using GORM throughout this series just because it's popular and easy to get started with.

Installing GORM

As it described in the docs, simply use go get gorm.io/gorm to install it.

However, GORM needs database drivers in order to connect to databases and do operations. I will be using Postgres for now and you can use whatever database you wanted.

Note: Sqlite is not recommended since it doesn't support some complex operations natively. But for now, you can use it since we don't have the need of complex operations and you can migrate to others in the future.

So let's install the database driver too:

$ go get -u gorm.io/driver/postgres  # or other database provider

...and GORM is ready to use!

Defining database models

Like other ORMs, GORM defines a table using models.

To define a model, you need to declare a struct containing the information about the table. For example:

type <name> struct {
    <field>  <field_type>
}

is the most basic form of defining a struct in Go. I assume that you have some basic knowledge about relational database storage so we won't discuss it very much here.

In order to create a GORM model, we simply need to fill in the information required. If we wanted to store a blog post in the database, then the following fields might be helpful:

ID (unsigned integer, primary key, auto increment, required)
Title (string, required)
Content (string, required)
Created at (time, defaults to current time)

So let's create a model for blog posts based on the above fields:

// models/post.go
package models

import "time"

type Post struct {
    ID        uint      `json:"id" gorm:"primaryKey"`
    Title     string    `json:"title"`
    Content   string    `json:"content"`
    CreatedAt time.Time `json:"created_at"`
}

Okay so let me explain all these.

We defined a GORM model using a struct and declared several fields. Here uint is unsigned int in other languages and time.Time is the datetime format in Golang.

But the backticks after the field type might be a little weird for new Go users. Those strings are called tags. They are using backtick annotation to define key-value pairs.

Struct tags are small pieces of metadata attached to fields of a struct that provide instructions to other Go code that works with the struct.¹

The json defines with key should the JSON encoder use when serializing the current field into JSON format. And the gorm key will let GORM know some extra information about this field. For example, here we defined ID as a primary key for the model.

Connecting to database

So now we successfully created the database schema, let's connect it to a real database:

// models/setup.go
package models

import (
    "gorm.io/driver/postgres"
    "gorm.io/gorm"
)

var DB *gorm.DB

func ConnectDatabase() {
    dsn := "host=localhost user=postgres dbname=go_blog port=5432 sslmode=disable timezone=Asia/Shanghai"
    database, err := gorm.Open(postgres.Open(dsn), &gorm.Config{})  // change the database provider if necessary

    if err != nil {
        panic("Failed to connect to database!")
    }

    database.AutoMigrate(&Post{})  // register Post model

    DB = database
}

In function ConnectDatabase(), we first defined our data source name and established the database connection. Here I used the Postgres driver and you might change it to fit your own need.

If there's any problems when connecting to our database, err will point to the error. In this case, we will call panic() to terminate the whole process. panic() is a builtin function that acts similar to raise in Python and throw in JavaScript.

Then we registered our Post model to the database and "exported" the database variable.

Note that DB is a global variable that is accessible in every file of package models, making operations with database easier without importing everything.

...and then let's call the connection function in our main.go:

// main.go
package main

import (
    "samzhangjy/go-blog/models"

    "github.com/gin-gonic/gin"
)

func main() {
    router := gin.Default()

    models.ConnectDatabase()  // new!

    // ...

    router.Run("localhost:8080")
}

And now we've finally made a connection to the database. Time to code the request controllers!

RESTful APIs

Unlike the first post, we are going to put all of our request logic into a separate folder called controllers and import them later in main.go to define the routes.

Create

Let's start by adding a create method:

// controllers/post.go
package controllers

import (
    "net/http"
    "samzhangjy/go-blog/models"

    "github.com/gin-gonic/gin"
)

type CreatePostInput struct {
    Title   string `json:"title" binding:"required"`
    Content string `json:"content" binding:"required"`
}

func CreatePost(c *gin.Context) {
    var input CreatePostInput
    if err := c.ShouldBindJSON(&input); err != nil {
        c.AbortWithStatusJSON(http.StatusBadRequest, gin.H{"error": err.Error()})
        return
    }

    post := models.Post{Title: input.Title, Content: input.Content}
    models.DB.Create(&post)

    c.JSON(http.StatusOK, gin.H{"data": post})
}

You might be wondering, why is there struct again? Well structs is an important concept in Go and you will see it a lot throughout this series. In this case, our struct CreateBlogInput defines body schema for request CreateBlog. The new tag binding is a Gin validation tag based on Validator. If you wanted to know something about Gin bindings, here's a great read: https://blog.logrocket.com/gin-binding-in-go-a-tutorial-with-example.

Then let's focus on CreateBlog(). We first validated the request body (variable input here) using context.ShouldBindJSON(). If the body is invalid, err would contain some error messages. If err contains something, then we will simply return a 400 HTTP status code and abort the request. This if err = statement; err != nil {} statement is a commonly used error handling technique in Go.

If the input is valid, we will first create a Post model with data given from the input. Then we will call database.Create() to put this record into the Post table.

Finally, we will return HTTP 200 with the newly created post schema if everything goes as expected.

And let's bind our controller to a route:

// main.go
package main

import (
    "samzhangjy/go-blog/controllers"
    "samzhangjy/go-blog/models"

    "github.com/gin-gonic/gin"
)

func main() {
    router := gin.Default()

    models.ConnectDatabase()

    router.POST("/posts", controllers.CreatePost)  // here!

    router.Run("localhost:8080")
}

Notice that we're passing the controller function itself to router.POST(), without parentheses.

Run your app with air and use tools like Postman to play around with this endpoint!

Read

Then let's quickly add an endpoint to view every post created:

// controllers/post.go

// ...

func FindPosts(c *gin.Context) {
    var posts []models.Post
    models.DB.Find(&posts)

    c.JSON(http.StatusOK, gin.H{"data": posts})
}

Unlike the previous request, this one has no request body. We defined array posts to store the posts created with type models.Post. DB.Find(&posts) means to find every entry that exists in the database and store the fetched result to posts. Remember to pass in the pointer instead of the actual variable!

And then quickly bind it to our router:

// main.go
package main

import (
    "samzhangjy/go-blog/controllers"
    "samzhangjy/go-blog/models"

    "github.com/gin-gonic/gin"
)

func main() {
    router := gin.Default()

    models.ConnectDatabase()

    router.POST("/posts", controllers.CreatePost)
    router.GET("/posts", controllers.FindPosts)

    router.Run("localhost:8080")
}

And now you could see the posts you created using CreateBlog!

Then let's create a route that fetches only one specified post by URL param:

// controllers/post.go

// ...

func FindPost(c *gin.Context) {
    var post models.Post

    if err := models.DB.Where("id = ?", c.Param("id")).First(&post).Error; err != nil {
        c.AbortWithStatusJSON(http.StatusNotFound, gin.H{"error": err.Error()})
        return
    }

    c.JSON(http.StatusOK, gin.H{"data": post})
}

The new methods here is DB.Where and DB.First. DB.Where() lets you write SQL query commands, replacing the dynamic data with ? and passing the actual data as the second argument. DB.First(&post), like its name, selects the first name of the given collection of data and stores the result inside post.

context.Param("<param-name>") is a Gin method to fetch the URL parameter by param name. The param name is defined like:

// main.go
package main

import (
    "samzhangjy/go-blog/controllers"
    "samzhangjy/go-blog/models"

    "github.com/gin-gonic/gin"
)

func main() {
    router := gin.Default()

    models.ConnectDatabase()

    router.POST("/posts", controllers.CreatePost)
    router.GET("/posts", controllers.FindPosts)
    router.GET("/posts/:id", controllers.FindPost)  // here!

    router.Run("localhost:8080")
}

The slug starting with a colon : is defined as url parameters in Gin. The string after the colon is the parameter's name, which we'll use to fetch the param value.

A route /posts/:id will match the following:

/posts/1
/posts/1/

But won't match:

/posts/1/abcd
/posts/

Update

Then it's time for updating posts:

// controllers/post.go

// ...

type UpdatePostInput struct {
    Title   string `json:"title"`
    Content string `json:"content"`
}

func UpdatePost(c *gin.Context) {
    var post models.Post
    if err := models.DB.Where("id = ?", c.Param("id")).First(&post).Error; err != nil {
        c.AbortWithStatusJSON(http.StatusNotFound, gin.H{"error": "record not found"})
        return
    }

    var input UpdatePostInput

    if err := c.ShouldBindJSON(&input); err != nil {
        c.AbortWithStatusJSON(http.StatusBadRequest, gin.H{"error": err.Error()})
        return
    }

    updatedPost := models.Post{Title: input.Title, Content: input.Content}

    models.DB.Model(&post).Updates(&updatedPost)
    c.JSON(http.StatusOK, gin.H{"data": post})
}

So we defined a struct containing request body schema again, but this time without any validation, since everything is optional. We copied the code to validate if the current post exists from the previous FindPost() method.

And if the post exists and the request body is valid, we will define a new model containing the contents of the newly generated post data. In this case, its name is updatedPost. Then we will fetch the model for the original post using DB.Model(&post) and update is using model.Updates(&updatedPost).

model.Updates() will update multiple fields and won't modify the fields that didn't defined in the updated schema (updatedPost). model.Update() will only update one field at a time.

And bind it to the router:

// main.go
package main

import (
    "samzhangjy/go-blog/controllers"
    "samzhangjy/go-blog/models"

    "github.com/gin-gonic/gin"
)

func main() {
    router := gin.Default()

    models.ConnectDatabase()

    router.POST("/posts", controllers.CreatePost)
    router.GET("/posts", controllers.FindPosts)
    router.GET("/posts/:id", controllers.FindPost)
    router.PATCH("/posts/:id", controllers.UpdatePost)

    router.Run("localhost:8080")
}

Delete

And finally here comes the DELETE operation at last:

// controllers/post.go

// ...

func DeletePost(c *gin.Context) {
    var post models.Post
    if err := models.DB.Where("id = ?", c.Param("id")).First(&post).Error; err != nil {
        c.AbortWithStatusJSON(http.StatusNotFound, gin.H{"error": "record not found"})
        return
    }

    models.DB.Delete(&post)
    c.JSON(http.StatusOK, gin.H{"data": "success"})
}

And we need to ensure that the currently given post ID is valid. Then we will call DB.Delete(&post) to delete the post entry from our database.

Then finally, bind it to our router:

// main.go
package main

import (
    "samzhangjy/go-blog/controllers"
    "samzhangjy/go-blog/models"

    "github.com/gin-gonic/gin"
)

func main() {
    router := gin.Default()

    models.ConnectDatabase()

    router.POST("/posts", controllers.CreatePost)
    router.GET("/posts", controllers.FindPosts)
    router.GET("/posts/:id", controllers.FindPost)
    router.PATCH("/posts/:id", controllers.UpdatePost)
    router.DELETE("/posts/:id", controllers.DeletePost)

    router.Run("localhost:8080")
}

Congrats! You've successfully built a collection of simple but working CRUD restful API routes! Play it around and try to modify some parts or add some new operations!

Conclusion

This is Part 2 of my learning Go web development. I actually learned a lot from writing this series and if there's any mistakes in the post, plz point them out!

I uploaded all the source code used in this post to GitHub. Feel free to clone it and play around with it!

I'm Sam Zhang and I'll see you guys next time!

From [Golang] Struct Tags explained. ↩

Learning Go Web Development From Zero

Sam Zhang — Thu, 26 May 2022 10:34:09 +0000

Hey guys, I'm Sam Zhang.

Today I wanted to explore the field of backend development - so I chose Go. The main reason I chose Go instead of other languages (such as Java), is because Go is kinda a new language (at least in my opinion) .

And sadly, I have absolutely no experience in Go development before. So I started this series of blog in order to keep track on myself. I will try to "record" what I've done and write some main concepts in my posts.

Of course, because I'm still a beginner (or newbie?) in Go, so there might be some incorrect stuff here. If so, please leave me a comment to help me improve!

Then let's get started!

Installing Go

Unlike many other languages, installing Go is quite simple, at least for me.

Go ahead and download the official installer at the Go download page. Run it, then you're all set!

Go comes with its command line tool go by default. It includes a set of commands such as run, get, mod, etc. For sure, I'm going to use those command a lot in this series. Check out the full list of commands here.

Installing Gin

I Googled about which Go web framework is the best for beginners, but turned out to be not that helpful. However, a lot of them mentioned Gin and has almost 60k stars on GitHub. So I chose Gin as my framework to get started.

Create a new folder to contain your project and follow the instructions of installation.

Troubleshooting

If you're a total beginner like me, you might be facing the following problem when running go get:

$ go get -u github.com/gin-gonic/gin
go: go.mod file not found in current directory or any parent directory.
        'go get' is no longer supported outside a module.
        To build and install a command, use 'go install' with a version,
        like 'go install example.com/cmd@latest'
        For more information, see https://golang.org/doc/go-get-install-deprecation
        or run 'go help get' or 'go help install'.

This means that you don't have a module created in the current folder. go mod is a command for managing Go modules. We'll use go mod init command to generate a new module.

The go mod init command accepts one argument, module-path . I'm not so sure about this argument right now and I understand it like the name of the module. The format of module-path should be as follows:

lorem/ipsum/dor

Then run go mod init <module-path> , replacing <module-path> with your own module path. After that, you should be able to install gin-gonic correctly.

Writing the first Gin request

So now we're all set with environments. Let's write a simple "Hello World" request in Go.

Create a new file named main.go in the root folder and fill in the following code:

package main

import (
    "net/http"

    "github.com/gin-gonic/gin"
)

func main() {
    router := gin.Default()  // router with default middleware installed
    // index route
    router.GET("/", func(c *gin.Context) {
        c.JSON(http.StatusOK, gin.H{
            "message": "Hello World",
        })
    })
    // run the server
    router.Run()
}

Then run go run main.go and open up http://localhost:8080 . Hopefully a Hello World will come up all right:

{
    "message": "Hello World"
}

Understanding the Hello World code

So... what the hell is going on here? Let's take a look at our code.

We've declared package main and imported some useful packages to our code. I'll try my best to explain code in main() function line by line.

So on line 10 we've declared a variable called router. gin.Default returns a router with default middleware (stuff like Logger and Recovery middleware) attached. You might be wondering, what in the world is := ? Well, it's similar to = in other programming languages. In Golang, a single = means assignment and := means declare and assignment. E.g:

var router = gin.Default()

Is equivalent to:

router := gin.Default()

The := operator is called Short variable declarations. More information see this StackOverflow question.

On line 12 ~ 16, we defined a GET request mapping to / and returns a JSON response. router.GET(), of course, defines a GET request. Its second argument accepts a callable function whereas that function will receive an argument with type gin.Context.

The context variable is named c in this case. *gin.Context is a pointer to the gin.Context value. The contexts in Gin provides a lot of features such as middleware and responses. For Chinese readers, you can take a look at this article for a more in-depth view of gin.Context.

On line 13 we called c.JSON() to response this request with content whose MIME type is application/json. http.StatusOK is a constant variable for the HTTP status code 200 success, which is from the builtin package net/http.

gin.H is a short-handed version of defining type for JSON data. If you'd like to nest JSON objects inside JSON objects, you can write something like:

gin.H{
    "some-data": gin.H{
        "message": "success"
    }
}

And finally on line 18, we run the server on the default port 8080.

Live reloading

Now if you wanted to change the content of your Go application, you need to restart the development server every time you change something in order to test the newly added code.

So some web frameworks (such as Python Flask) has a builtin hot reloader for the development server. Unfortunately, Gin doesn't come with that. I chose to use air as my auto reloader and of course you can use others.

Setting up Air is quite simple. Run the following command to install it:

curl -sSfL https://raw.githubusercontent.com/cosmtrek/air/master/install.sh | sh -s -- -b $(go env GOPATH)/bin

This will install Air to your $GOPATH, which is default to ~/go . Then append it to your $PATH by adding the following line to your .bashrc (or .zshrc):

export PATH=$PATH:$(go env GOPATH)/bin

...and you're all set! Run air init to generate a default Air configuration.

Using Air is even simpler. Just run air in your command line and Air run main.go in the current folder and will automatically watch for changes in your source code.

Tip for macOS users: if you don't want to see the annoying popup every time Air reloads your code, simply replace

router.Run()

with

router.Run("localhost:8080")

Conclusion

So... I finally finished the Gin hello world! I put all my source code to GitHub and hopefully I will be updating it. Clone it if you found something useful or just to play around!

I'm Sam Zhang and I'll see ya next time!

VueBlogger: A blogging site generator for Vue

Sam Zhang — Thu, 07 Oct 2021 08:50:45 +0000

What is VueBlogger?

VueBlogger is a light-weight blogging site generator for Vue.js, built for geeks who wanted to write their blog site in Vue and write posts in Markdown.

I developed it for a reason: there isn't really a simple blogging tool for Vue. VuePress works, but it's to complicated. So for that purpose, I developed this light-weight blogging site for Vue: VueBlogger.

You can host it on any server that has Nodejs and Vue installed. Actually, you even don't need them if you already built your blog on your own laptop: just host the HTML and JavaScript files directly!

Behind VueBlogger

At first, I just wanted to build a lightweight blog for myself. However, I found that if I publish the source code and write a tutorial on how to use it, developers can spend less time on building the blog but more time writing the posts.

So I decided to make VueBlogger open source. And that's why this post appeared.

Of course, VueBlogger is not quite complete: there might be bugs, and the UI might not work properly. However, from my own usage, there aren't any issues till now. I will try to add some more features and make VueBlogger more customizable in the future.

How to use VueBlogger?

Note: Make sure you have Node.js and Vue 2 installed before continuing.

Configuration

You need to config your blog before using.

General Configuration

Write your configuration for the entire blog in /posts/data/config.json. For example:

{
    "config": {
        "username": "Your username",
        "avatarPath": "Your avatar image url",
        "description": "Your blog description",
        "name": "Your name",
        "blogTitle": "Your blog title",
        "blogStartYear": 2019  // The year your blog started. Used to generate the copyrights in the footer.
    },
    "contacts": [
        {
            "name": "Contact method",
            "value": "Contact information, such as your email address",
            "link": "Contact link, such as `mailto:email@example.com`"
        }
    ]
}

Please fill in all information above, or VueBlogger might crash.

Projects Configuration

First, provide a list of your projects needed to be shown on your blog. Configure them in /posts/data/projects.json. E.g.:

{
    "projects": [
        {
            "name": "Project name",
            "des": "Project description",
            "img": "Project cover image url",
            "link": "Project homepage url",
            "github": "Project GitHub url"
        }
    ]
}

Also, please fill in all information above.

Then, please create a file called project-des.md under /posts/data/. In this markdown file, you can write some text which will be shown in the projects page above the projects. For example:

Below are my personal projects made in my free time.

You can write in MarkDown format.

About Yourself Configuration

You needed to provide a MarkDown file to display on the about page. Write something about your self in the Markdown file /posts/data/about.md.

Feel free to use Markdown!

Link Exchanges Configuration

Sometimes you wanted to share your blog with your friends, and your friends wanted to share his/her's too. With VueBlogger, you can easily add a link to your friends' blogs. Add the file /posts/data/links.json. The link format is as follows:

{
    "links": [
        {
            "title": "Website title",
            "desc": "Website description",
            "link": "Website link"
        }
    ]
}

Note: if you don't want to add any links, simpily create the file and set links as an empty list.

Writing Posts

You can write your posts in Markdown format, under /posts/. The file structure of /posts/ is as follows:

posts                             
├─ data                           
│  ├─ about.md                    
│  ├─ config.json                 
│  ├─ posts.json                  
│  ├─ project-des.md              
│  └─ projects.json               
├─ post-1-title.md                      
├─ post-2-title.md              
├─ post-3-title.md                  
└─ post-4-title.md

Yes, you need to do one more configuration on the posts: /posts/data/posts.json. Write down the posts you wanted to display on the blog and provide some information about it. Such as:

{
    "posts": [
        {
            "title": "VueBlogger: A simple blogging site for Vue",  // Post title
            "tags": ["Vue.js", "project", "frontend"],  // Post tags
            "cover": "https://dev-to-uploads.s3.amazonaws.com/i/95lvt23xz4ozer5byomi.png",  // Post cover image url
            "des": "There isn't really a simple blogging tool for Vue. VuePress works, but it's to complicated. So for that purpose, I developed this light-weight blogging site for Vue: VueBlogger.",  // Post description
            "date": [  // Post publish date
                2021,  // Year
                10,  // Month
                6  // Day
            ],
            "id": "blogue"  // Post file name stored under `/posts/`
        }
    ]
}

Note that the id of a post is its filename, NOT its title. Wrong configuration can led to a 404 response for a mis-configured post. Be sure to check the configuration!

So, every time you wanted to publish a post, you need to add it to the configuration file. Otherwise, it won't show up at all in your blog.

Publishing your blog

If this is the first time you publish your blog, you need to install VueBlogger dependencies first.

Run yarn (or npm install for npm) to install VueBlogger dependencies.

Currently, VueBlogger supports only GitHub pages to publish automatically. You need to build and publish your site manually if you wanted to host on other platforms.

First, create a repository named <your-github-username>.github.io, replacing <your-github-username> with your real GitHub username.

Then, run source publish.sh to build and publish your blog to GitHub pages. After running this script, your blog should be live on <your-github-username>.github.io in a few minutes.

Running your blog locally

Use yarn serve (or npm run serve if you're using npm) to start a hot-reloading development server on localhost:8080.

Building your blog manually

If you're not using GitHub Pages as your hosting provider, you'll need to build your site manually.

Run yarn build (or npm run build for npm) to build VueBlogger.

After building, the /dist directory contains the build assets for your blog (HTML, CSS, JavaScript, etc).

Last words

Please star or fork VueBlogger on GitHub to support me!

As always, feel free to report issues and open a PR to help the development!

Creating a Personal Blog with Vue.js - Part 1: Getting Started

Sam Zhang — Sun, 03 Jan 2021 09:14:17 +0000

Hello and welcome to my brand-new tutorial on building a personal blog using Vue & Markdown! In this tutorial, you are going to create a full-featured personal blog using Vue.js and deploy it to GitHub Pages! So before starting this tutorial, you might want to check out the Vue.js Docs. After you learned how to use the basic features of Vue.js, let's continue!

If you wanted to see what the project looks when it's finished, head over to my blog! It's using the very same code just like in this tutorial.

Starting Our Project

I assume you have Vue CLI installed. If not, follow the docs to install Vue CLI. We will use it through out our tutorial later on.

OK, so you've got all set, let's create our project. Type in vue create <project-name> in your terminal, replacing the placeholder name to your favorite name. I'll just use personal-blog:

Press enter, and you'll probably see an interface similar to this:

Select Manually select features using arrow keys, and press enter, you will see a form like this:

Toggle CSS Pre-processors and Router. Make sure you have the other two defaults toggled on. Then press enter. You can choose your features of selected extensions then, and below are mine.

Then press enter. The CLI tool will automatically create an empty project for you. After the installation is over, run cd <project-name> to get into your project folder.

First, run our project to make sure we've got everything all right with command npm run serve. Wait a few seconds, and then go to http://localhost:8080/ in your browser. You will see a page like this:

OK, so now we've got the website working, we need to work on styles next.

Choosing an UI Library

Choosing an UI library is very important. It defines the main design and look of your website, so we must be careful. Of course, you can write your own CSS and stuff, but that's too complected. The common Vue UI frameworks are Vuetify, a Vue UI framework for Material Design, BootstrapVue, a mix of Vue and Bootstrap, Quesar Framework, a Vue.js framework following the design guidelines. But, these are too normal. Since we are building a personal blog, we must have an unique design. So I chose Vuesax, a framework
components for Vue.js. It has a very unique design, not following any design guidelines, but very beautiful.

The version of Vuesax we are going to use is 4.x, which is still in Alpha state. But the components already made are enough to create a simple interface for us. You can see the current major version (3.x)'s docs here, or track the development of 4.x here on GitHub. To install it, open a new tab in your terminal and cd into your project, run npm install vuesax@next to install Vuesax 4. After installing, we are going to create a basic bone for our blog.

Creating the Basic Website

Emptying the Project

OK, so now we got to the code. Open HelloWorld.vue in the src/components folder, and delete everything in there. Then go to App.vue in the src folder, remove everything again. Then, we will delete About.vue in src/views at this moment, because we don't need it. After this process, you will probably see an error message in your console. Don't worry, it's because Vue can't find the file About.vue that we've just deleted. In src/router/index.js, remove the /about route:



import Vue from 'vue'
import VueRouter from 'vue-router'
import Home from '../views/Home.vue'

Vue.use(VueRouter)

const routes = [
  {
    path: '/',
    name: 'Home',
    component: Home
  }
]

const router = new VueRouter({
  mode: 'history',
  base: process.env.BASE_URL,
  routes
})

export default router

Now check the website again. It turned into a blank website. Great! That's what we wanted. So now we can work on our own site.

Working On the Blog

Let's rename our src/components/HelloWorld.vue to Index.vue now. Then you will get an error again because we didn't rename it inside src/views/Home.vue. Let's rename it:



<template>
  <div>
    <Index/>
  </div>
</template>

<script>
import Index from '@/components/Index.vue'

export default {
  name: 'Home',
  components: {
    Index
  }
}
</script>

Note that I made a few more changes like removing the class home on the div element, removing the Vue logo image, and removing the msg property on component Index. So now the website should work again. Let's go on.

Making a Navbar

Every website needs a navbar on the top for navigation, and so do our blog. Lucky for us, Vuesax had already crafted a Navbar component for us. So with that, we can easily create our navbar component.

First, create a new file Navbar.vue in src/components. This will be the file we'll be working on to get our navbar working.

Then, import Navbar.vue inside Index.vue:



<template>
  <div>
    <Navbar/>
  </div>
</template>

<script>
import Navbar from '@/components/Navbar.vue'

export default {
  name: 'Index',
  components: {
    Navbar
  }
}
</script>

Okay, so now we can finally work on our navbar. Let's get our basic template set:



<template>
  <div></div>
</template>

<script>
export default {
  name: 'Navbar'
}
</script>

Hang on, we'll get to the navbar part soon. But now, we need to import Vuesax to our project. Edit src/main.js to the code below:



import Vue from 'vue'
import App from './App.vue'
import router from './router'
import Vuesax from 'vuesax'
import 'vuesax/dist/vuesax.css'

Vue.config.productionTip = false

Vue.use(Vuesax, {})

new Vue({
  router,
  render: h => h(App)
}).$mount('#app')

Now we've added Vuesax to our project, let's make the navbar. I just copy and edited a few based on the navbar example given in the docs.



<template>
  <div>
    <vs-navbar shadow-scroll fixed center-collapsed v-model="active">
      <template #left>
        <h3>Hello World</h3>
      </template>
      <vs-navbar-item :active="active == 'home'">
        Home
      </vs-navbar-item>
      <template #right>
        <vs-button flat>Posts</vs-button>
      </template>
    </vs-navbar>
  </div>
</template>

<script>
export default {
  name: 'Navbar',
  data: () => {
    return {
      active: 'home'
    }
  }
}
</script>

The website at this moment looks like this:

The font looks awful, uh? Will, the poppins font came to rescue! I've uploaded it into the GitHub link at the bottom of this tutorial, so you can download it there if you want. Paste the whole fonts folder into public folder, and create a new folder called styles. Then, create a new CSS stylesheet called main.css inside it. The code below just gives the basic font support and adds a container and a divider, nothing special. Move it to main.css:



@font-face
{
  font-family: Poppins;
  src: url("../fonts/Poppins/Poppins-Regular.ttf");
  font-weight: 400;
  font-style: normal;
  -webkit-font-smoothing: antialiased;
  -moz-osx-font-smoothing: grayscale;
}

* {
  font-family: Poppins;
}

.container {
  margin-left: 50px;
  margin-right: 50px;
}

hr {
  border-width: 1px;
  border-style: solid;
  border-color: #EEF2F5;
  margin-bottom: 10px;
}

Then, add a new line to index.html in the public folder:



<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width,initial-scale=1.0">
    <link rel="icon" href="<%= BASE_URL %>favicon.ico">
    <!-- This line below -->
    <link rel="stylesheet" type="text/css" href="<%= BASE_URL %>styles/main.css"/>
    <title><%= htmlWebpackPlugin.options.title %></title>
  </head>
  <body>
    <noscript>
      <strong>We're sorry but <%= htmlWebpackPlugin.options.title %> doesn't work properly without JavaScript enabled. Please enable it to continue.</strong>
    </noscript>
    <div id="app"></div>
    <!-- built files will be auto injected -->
  </body>
</html>

All right, now reload the webpage and it will be fresh:

Neat, right? Well, that's the importance of looking for the correct font. I chose the Poppins because it's very similar to the font they used on the Vuesax Website.

Summing Up

So that's all for this part of this tutorial. We've created a basic navbar using the Vuesax framework and built a dead-simple tiny-little website that only has a navbar. And as always, if you have any questions on this tutorial, feel free to leave a comment below. Bye! I'll see ya next time.

Found a New UI Library for Vue: PrimeVue

Sam Zhang — Mon, 07 Sep 2020 13:01:32 +0000

Okay, so first off, this is NOT an ad. I really found this UI library great and wanted to share it. This post contains details about this lib and my thoughts, so let's take a look.

How I Found It

So one day I was bored and wanted to find some UI libraries for Vue to play with, and I actually found A LOT. I didn't even look at these libraries at that time, and they're just gathering dust in my bookmarks bar. And these days I'm looking for some UI libs to use, and I found the bookmarks I've added earlier. I opened almost every one of them and found only this very useful AND I haven't seen this one before. So I took a closer look at it, and found it is great.

What's Packed

There's 60+ components that you can use out-of-the-box, including autocomplete, markdown editor, rating, tables, upload, menus, charts, inplace editing and even terminals! Best of all, it has shipped 33 built-in themes, including 4 Bootstrap-based themes, 8 material-based themes, 12 PrimeOne Design themes, and 9 legacy themes. All themes includes dark themes. And finally, it's responsive!

My thoughts

I think this library is very useful is that it has packed the two major design systems in one library, so you don't need to switch between libraries. What I really like is the rich components, and some of them are very useful. I have not tried this theme yet, but I guess it will be easy to setup and use.

PrimeVue's GitHub: https://github.com/primefaces/primevue

PrimeVue's website: https://primefaces.org/primevue/showcase/#/

So, what do you think about this library? Let's discuss about it!

Crawling Google Search Results - Part 3: Fetching Wiki

Sam Zhang — Wed, 26 Aug 2020 10:13:55 +0000

So we've created a pretty awesome Google results spider that can crawl normal results plus related video results in the last two parts. Now, we're going to extend it: let's go looking for the wiki!

WARNING: Don't ever use this spider to scrape lots of data. As of Google provides a public API that allows you to call 100 times for free, your IP will be banned if Google noticed the unusual traffic from your computer. This spider is built only for learning purposes, and it shouldn't be used in real projects. So keep that in mind, and we'll get started.

Our Goal

Okay, let me explain it. If you search Google for Python, you will see a card containing wikis of Python in the upper-right corner.

Understood? All right, then let's analyze our page as usual.

Let's Analyze!

Container

Take a close look at the source code. What's the container? Right, it's a div element with class kp-wholepage as the picture shown below.

Wiki Title

Next, we'll look for the wiki's title. In our example, it's Python. It's inside a h2 element whose data-attrid is title.

Wiki Subtitle

Some wikis have subtitles, such as our example. To get it, we need to find a div with attribute data-attrid="subtitle" and get its text.

Wiki Description

Wikis also have descriptions. Its content is inside a span tag with a parent div. The div has a class called kno-rdesc.

Wiki Link

Did you see that all wikis have a wikipedia link after each of their descriptions? Well, that's the link that we're going to crawl. It's right below wiki's description, and in the same parent span.

Wiki Details

Finally, we've reached the hardest part of this post. We are going to fetch for the details as shown below.

No, it's not a table, sorry. In fact, it's generated with several divs. Take a close look. Each detail is actually surrounded by a div styled with class mod. But, if you search for class="mod" in the HTML, you'll get an extra one: the description (There's one more out there, but not in the same parent). We'll just skip that using [1:]. So, to be clear, we are going to look for several divs with class mod and has a parent div with class wp-ms.

The parent:

The details that we're looking for:

Name

We also look into the details. We need to fetch for two things in order to get this thing complete: the name and the detail description. First, let's take a look at the names. It's right inside the first span element of the div. We'll strip the : part out when we code.

Description

Then comes the description. It's in the rest of spans of our div, except for the first one.

The reason I say it this way, not saying the last one, is because there might be two spans being the description together. For example, the File extension tab of our HTML:

There are two spans in our sight. We need to add a for loop to keep adding spans to our target string variable.

Coding Time

All right, I guess you all are enough with analyzing stuff. Let have some fun now! Open up your favorite IDE, and let's start coding!

First, create a new method in class GoogleSpider as usual. I'll name it __search_wiki:

class GoogleSpider(object):
    # ...
    def __search_wiki(self, response: requests.Response) -> list:
        """Search for wiki results based on the given response

        Args:
            response (requests.Response): The response requested to Google

        Returns:
            list: A list of wiki results, usually only one if found.
        """
        pass
    # ...

And add it to our search function:

class GoogleSpider(object):
    # ...
    def search(self, query: str, page: int = 1) -> dict:
        # ...
        video = self.__search_video(response)
        result = self.__search_result(response)
        wiki = self.__search_wiki(response)  # <---
        pages = self.__get_total_page(response)
        results.extend(result)
        results.extend(video)
        results.extend(wiki)  # <---
        return {
            'results': results,
            'pages': pages
        }

OK. We're ready to develop our __search_wiki method now! First of all, we need to create a BeautifulSoup object and get the wiki container:

class GoogleSpider(object):
    # ...
    def __search__wiki(self, response: requests.Response) -> list:
        # ...
        soup = BeautifulSoup(response.text, 'html.parser')
        # Get the info container card
        container = soup.find('div', class_='kp-wholepage')

    # ...

And we will detect wether the container is empty or not:

class GoogleSpider(object):
    # ...
    def __search_wiki(self, response: requests.Response) -> list:
        # ...
        # If the container is empty (None), then there isn't a wiki tab for the
        # given response.
        if container is None:
            return []

Then, we will collect the details (I'm going to just paste my code here, leave a comment if you don't understand something):

class GoogleSpider(object):
    # ...
    def __search_wiki(self, response: requests.Response) -> list:
        # ...
        # Title
        title = container.find('h2', attrs={'data-attrid': 'title'}).text
        # Subtitle
        try:
            subtitle = container.find(
                'div', attrs={'data-attrid': 'subtitle'}).text
        except AttributeError:
            subtitle = None
        # Description
        des = container.find('div', class_='kno-rdesc').find('span').text
        # The link to Wikipedia
        url = container.find('div', class_='kno-rdesc').find('a')['href']

Finally, we will create the details part together. First, let's find the container (the table):

class GoogleSpider(object):
    # ...
    def __search_wiki(self, response: requests.Response) -> list:
        # ...
        # Details table
        try:
            table = container.findAll(
                'div', class_='wp-ms')[1].findAll('div', class_='mod')[1:]
        except IndexError:
            table = []

I've added a try-except here is that because not every result has a in-detail description table, so it might throw a IndexError while looking for the first div using [1]. If that's the case, then we will set table to [] (empty).

Then, we will loop through the table:

class GoogleSpider(object):
    # ...
    def __search_wiki(self, response: requests.Response) -> list:
        # ...
        details = []
        # Loop through all details
        for row in table:
            pass

And fetch the name:

class GoogleSpider(object):
    # ...
    def __search_wiki(self, response: requests.Response) -> list:
        # ...
        details = []
        # Loop through all details
        for row in table:
            name = row.find('span').text.strip(': ')

Here I removed : using strip. Note the whitespace! It won't work without it.

Then, we need to get the description:

class GoogleSpider(object):
    # ...
    def __search_wiki(self, response: requests.Response) -> list:
        # ...
        details = []
        # Loop through all details
        for row in table:
            # ...
            # Find all spans and get their text one-by-one
            detail_ = row.findAll('span')[1:]

...And the most interesting part comes. In order to get the formatted, nice-looking description even including several spans, we need to add another small loop:

class GoogleSpider(object):
    # ...
    def __search_wiki(self, response: requests.Response) -> list:
        # ...
        details = []
        # Loop through all details
        for row in table:
            # ...
            # Find all spans and get their text one-by-one
            detail_ = row.findAll('span')[1:]
            detail = ''
            # Get their text
            for _ in detail_:
                pass

Then, we will connect these spans together into one single string:

class GoogleSpider(object):
    # ...
    def __search_wiki(self, response: requests.Response) -> list:
        # ...
        details = []
        # Loop through all details
        for row in table:
            # ...
            # Get their text
            for _ in detail_:
                detail += _.text + ' '  # Add a whitespace to prevent descriptions connected to each other

As it saids in the comment, you must add into another whitespace to make it working. Otherwise, you'll get the response like this:

'PY filesfileinfo.com'

If you add it, it would look like this:

'PY files fileinfo.com '

So we need to strip it again when we add it into the result dict:

class GoogleSpider(object):
    # ...
    def __search_wiki(self, response: requests.Response) -> list:
        # ...
        details = []
        # Loop through all details
        for row in table:
            # ...
            details.append({
                'name': name,
                'detail': detail.strip()
            })

And append everything into the final result and return it:

class GoogleSpider(object):
    # ...
    def __search_wiki(self, response: requests.Response) -> list:
        # ...
        result = {
            'title': title,
            'subtitle': subtitle,
            'des': des,
            'url': url,
            'details': details,
            'type': 'wiki'
        }
        return [result]

The main reason I return it as a list is because that way, you will get the same response type for all private search methods. So that way, we can use extend method to any search type to append it into the final result list.

Full Code

The full code of this tutorial until Part 3 is below.

# Import dependencies
from pprint import pprint
from urllib.parse import quote

import requests
from bs4 import BeautifulSoup


class GoogleSpider(object):
    def __init__(self):
        """Crawl Google search results

        This class is used to crawl Google's search results using requests and BeautifulSoup.
        """
        super().__init__()
        self.headers = {
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:79.0) Gecko/20100101 Firefox/79.0',
            'Host': 'www.google.com',
            'Referer': 'https://www.google.com/'
        }

    def __get_source(self, url: str) -> requests.Response:
        """Get the web page's source code

        Args:
            url (str): The URL to crawl

        Returns:
            requests.Response: The response from URL
        """
        return requests.get(url, headers=self.headers)

    def __get_total_page(self, response: requests.Response) -> int:
        """Get the current total pages

        Args:
            response (requests.Response): the response requested to Google using requests

        Returns:
            int: the total page number (might be changing when increasing / decreasing the current page number)
        """
        soup = BeautifulSoup(response.text, 'html.parser')
        pages_ = soup.find('div', id='foot', role='navigation').findAll('td')
        maxn = 0
        for p in pages_:
            try:
                if int(p.text) > maxn:
                    maxn = int(p.text)
            except:
                pass
        return maxn

    def __search_video(self, response: requests.Response) -> list:
        """Search for video results based on the given response

        Args:
            response (requests.Response): the response requested to Google search

        Returns:
            list: A list of found video results, usually three if found
        """
        soup = BeautifulSoup(response.text, 'html.parser')
        try:
            cards = soup.find('g-scrolling-carousel').findAll('g-inner-card')
        except AttributeError:
            return []
        # Pre-process the video covers
        covers_ = soup.find('span', id='fld').findNextSiblings('script')[1:]
        # Get the cover images
        covers = []
        for c in covers_:
            # Fetch cover image
            try:
                covers.append(str(c).split('s=\'')[-1].split(
                    '\';var ii')[0].rsplit('\\', 1)[0])
            except IndexError:
                pass
        results = []
        # Generate video information
        for card in cards:
            try:
                # Title
                # If the container is not about videos, there won't be a div with
                # attrs `role="heading"`. So to catch that, I've added a try-except
                # to catch the error and return.
                try:
                    title = card.find('div', role='heading').text
                except AttributeError:
                    return []
                # Video length
                length = card.findAll('div', style='height:118px;width:212px')[
                    1].findAll('div')[1].text
                # Video upload author
                author = card.find(
                    'div', style='max-height:1.5800000429153442em;min-height:1.5800000429153442em;font-size:14px;padding:2px 0 0;line-height:1.5800000429153442em').text
                # Video source (Youtube, for example)
                source = card.find(
                    'span', style='font-size:14px;padding:1px 0 0 0;line-height:1.5800000429153442em').text
                # Video publish date
                date = card.find(
                    'div', style='font-size:14px;padding:1px 0 0 0;line-height:1.5800000429153442em').text.lstrip(source).lstrip('- ')  # Strip the source out because they're in the same container
                # Video link
                url = card.find('a')['href']
                # Video cover image
                try:  # Just in case that the cover wasn't found in page's JavaScript
                    cover = covers[cards.index(card)]
                except IndexError:
                    cover = None
            except IndexError:
                continue
            else:
                # Append result
                results.append({
                    'title': title,
                    'length': length,
                    'author': author,
                    'source': source,
                    'date': date,
                    'cover': cover,
                    'url': url,
                    'type': 'video'
                })
        return results

    def __search_result(self, response: requests.Response) -> list:
        """Search for normal search results based on the given response

        Args:
            response (requests.Response): The response requested to Google

        Returns:
            list: A list of results
        """
        # Initialize BeautifulSoup
        soup = BeautifulSoup(response.text, 'html.parser')
        # Get the result containers
        result_containers = soup.findAll('div', class_='rc')
        # Final results list
        results = []
        # Loop through every container
        for container in result_containers:
            # Result title
            title = container.find('h3').text
            # Result URL
            url = container.find('a')['href']
            # Result description
            des = container.find('span', class_='st').text
            results.append({
                'title': title,
                'url': url,
                'des': des,
                'type': 'result'
            })
        return results

    def __search_wiki(self, response: requests.Response) -> list:
        """Search for wiki results based on the given response

        Args:
            response (requests.Response): The response requested to Google

        Returns:
            list: A list of wiki results, usually only one if found.
        """
        soup = BeautifulSoup(response.text, 'html.parser')
        # Get the info container card
        container = soup.find('div', class_='kp-wholepage')
        # If the container is empty (None), then there isn't a wiki tab for the
        # given response.
        if container is None:
            return []
        # Title
        title = container.find('h2', attrs={'data-attrid': 'title'}).text
        # Subtitle
        try:
            subtitle = container.find(
                'div', attrs={'data-attrid': 'subtitle'}).text
        except AttributeError:
            subtitle = None
        # Description
        des = container.find('div', class_='kno-rdesc').find('span').text
        # The link to Wikipedia
        url = container.find('div', class_='kno-rdesc').find('a')['href']
        # Details table
        try:
            table = container.findAll(
                'div', class_='wp-ms')[1].findAll('div', class_='mod')[1:]
        except IndexError:
            table = []
        details = []
        # Loop through all details
        for row in table:
            name = row.find('span').text.strip(': ')
            # Find all spans and get their text one-by-one
            detail_ = row.findAll('span')[1:]
            detail = ''
            # Get their text
            for _ in detail_:
                detail += _.text + ' '  # Add a whitespace to prevent descriptions connected to each other
            details.append({
                'name': name,
                'detail': detail.strip()
            })
        result = {
            'title': title,
            'subtitle': subtitle,
            'des': des,
            'url': url,
            'details': details,
            'type': 'wiki'
        }
        return [result]

    def search(self, query: str, page: int = 1) -> dict:
        """Search Google

        Args:
            query (str): The query to search for
            page (int): The page number of search result

        Returns:
            dict: The search results and the total page number
        """
        # Get response
        response = self.__get_source(
            'https://www.google.com/search?q=%s&start=%d' % (quote(query), (page - 1) * 10))
        results = []
        video = self.__search_video(response)
        result = self.__search_result(response)
        wiki = self.__search_wiki(response)
        pages = self.__get_total_page(response)
        results.extend(result)
        results.extend(video)
        results.extend(wiki)
        return {
            'results': results,
            'pages': pages
        }


if __name__ == '__main__':
    pprint(GoogleSpider().search(input('Search for what? ')))

I've also uploaded all the tutorial code to GitHub, so check that out if you want. Click on this link to see the code for this tutorial until Part 3.

Conclusion

So that's pretty much all for today, and I hope it will help you. Leave a comment below for either suggestions or questions for this tutorial, and see you next time!

Crawling Google Search Results - Part 2: Crawling Video

Sam Zhang — Sun, 23 Aug 2020 03:20:04 +0000

In the previous part of this tutorial, we've created a very simple spider that allows us to crawl Google's normal search results. In this part of the tutorial, we're going further than before.

What We Are Crawling

You might not fully understand what I'm talking about this crazy video thing. All right, let me explain it. If you search Google for Python, for example, there will be a card containing videos related to the search as the picture shown below.

The part with a Videos heading is what we're going to crawl. Seems easy, right? Will, it's not gonna be as easy as Part 1's work.

Analyze Time

Okay, so now we know where we're building, let's take a look at the webpage.

If you look for the results, you can see that they're surrounded by a g-scrolling-carousel element.

Inside it, there's another g-inner-card element containing video details for every videos.

All right, so now we've got all the containers, let's take a look at the details. First, we need to have the video title. It's inside a div element with attribute role="heading".

...And the link inside an a element:

Then, we'll look for the video author, which has a style with max-height:1.5800000429153442em;min-height:1.5800000429153442em;font-size:14px;padding:2px 0 0;line-height:1.5800000429153442em:

We've also need to fetch for the video's source, or platform. For example, Youtube. It's inside a span whose parent is a div with style font-size:14px;padding:1px 0 0 0;line-height:1.5800000429153442em.

And we'll also get the video's upload date. It's right below video author's span, inside the same parent element. We'll just strip the video author's text to get its text when we reach the code.

Finally, we are going to look for the video length. It's in the second child element of a div styled with height:118px;width:212px.

Where's the Cover?

You might wonder about the video cover image. Where's that? Well, it's inside the JavaScript. If you take a closer look under the video details, you will find three <script> tags containing Base64 images. Copy one of them and you will probably get the video cover. So now we've got the information, let's see how can we locate them. The easiest way is to locate their parent <div> and find all the script tags. But, there's lots of them! The way I'm using is to locate their sibling element <span id="fld"></span>. With that, we can locate its sibling elements - script tags. The tags we're looking for are the last three script element, excluding the first one. We can just use [1:] in Python to get rid of it.

Alright, Hands On!

Create a function called __search_video and we'll put all of our code inside it.

class GoogleSpider(object):
    # ...
    def __search_video(self, response: requests.Response) -> list:
        """Search for video results based on the given response

        Args:
            response (requests.Response): the response requested to Google search

        Returns:
            list: A list of found video results, usually three if found
        """
        pass

And we will create our BeautifulSoup object from the response:

class GoogleSpider(object):
    # ...
    def __search_video(self, response: requests.Response) -> list:
        """Search for video results based on the given response

        Args:
            response (requests.Response): the response requested to Google search

        Returns:
            list: A list of found video results, usually three if found
        """
        soup = BeautifulSoup(response.text, 'html.parser')

Then, let's locate our g-inner-card:

class GoogleSpider(object):
    # ...
    def __search_video(self, response: requests.Response) -> list:
        # ...
        cards = soup.find('g-scrolling-carousel').findAll('g-inner-card')

And loop through them to generate our search result:

class GoogleSpider(object):
    # ...
    def __search_video(self, response: requests.Response) -> list:
        # ...
        results = []
        # Generate video information
        for card in cards:
            try:  # Just in case
                # Title
                title = card.find('div', role='heading').text
                # Video length
                length = card.findAll('div', style='height:118px;width:212px')[
                    1].findAll('div')[1].text
                # Video upload author
                author = card.find(
                    'div', style='max-height:1.5800000429153442em;min-height:1.5800000429153442em;font-size:14px;padding:2px 0 0;line-height:1.5800000429153442em').text
                # Video source (Youtube, for example)
                source = card.find(
                    'span', style='font-size:14px;padding:1px 0 0 0;line-height:1.5800000429153442em').text
                # Video publish date
                date = card.find(
                    'div', style='font-size:14px;padding:1px 0 0 0;line-height:1.5800000429153442em').text.lstrip(source).lstrip('- ')  # Strip the source out because they're in the same container
                # Video link
                url = card.find('a')['href']
            except IndexError:
                continue
            else:
                # Append result
                results.append({
                    'title': title,
                    'length': length,
                    'author': author,
                    'source': source,
                    'date': date,
                    'url': url,
                    'type': 'video'
                })
        return results

And finally, we'll do the cover part together. First, let's create a variable called covers_ to store the scripts we've found. Note that I used [1:] to create a slice of the list to remove the first tag.

class GoogleSpider(object):
    # ...
    def __search_video(self, response: requests.Response) -> list:
        # ...
        # Pre-process the video covers
        covers_ = soup.find('span', id='fld').findNextSiblings('script')[1:]
        # ...

Then, we will loop through them:

class GoogleSpider(object):
    # ...
    def __search_video(self, response: requests.Response) -> list:
        # ...
        # Pre-process the video covers
        covers_ = soup.find('span', id='fld').findNextSiblings('script')[1:]
        # Get the cover images
        covers = []
        for c in covers_:
            # TODO
        # ...

And we will append each base64-coded images to the list covers. Here we need to remove the unnecessary JavaScript code and only keep the image. If you don't know about rsplit which I've used in my code, well, it's a special version of split. It will start from the end to scan for the result and split them. For example, if I have a variable called text:

>>> text = 'Hi everyone! Would you like to say Hi to me?'

If you split it the normal way:

>>> text.split('Hi', 1)
['', ' everyone! Would you like to say Hi to me?']

But with rsplit:

>>> text.rsplit('Hi', 1)
['Hi everyone! Would you like to say ', ' to me?']

The reason I used it here is because there might be another ;var ii inside the base64 image, and if I used split, it would create a broken image.

class GoogleSpider(object):
    # ...
    def __search_video(self, response: requests.Response) -> list:
        # ...
        # Pre-process the video covers
        covers_ = soup.find('span', id='fld').findNextSiblings('script')[1:]
        # Get the cover images
        covers = []
        for c in covers_:
            # Fetch cover image
            try:
                covers.append(str(c).split('s=\'')[-1].split(
                    '\';var ii')[0].rsplit('\\', 1)[0])
            except IndexError:
                pass
        # ...

...and add them to our list:

class GoogleSpider(object):
    # ...
    def __search_video(self, response: requests.Response) -> list:
        # ...
        for card in cards:
            # ...
            try:  # Just in case
                # Video cover image
                try:  # Just in case that the cover wasn't found in page's JavaScript
                    cover = covers[cards.index(card)]
                except IndexError:
                    cover = None
            except IndexError:
                continue
            else:
                # Append result
                results.append({
                    # ...
                    'cover': cover,
                    # ...
                })
        # ...

OK, so we are almost there. There's one more thing that we need to work on, is that when we search something that doesn't contain a video result, our program will throw an AttributeError. To prevent that, we need to add a try-except:

class GoogleSpider(object):
    # ...

    def __search_video(self, response: requests.Response) -> list:
        # ...
        try:
            cards = soup.find('g-scrolling-carousel').findAll('g-inner-card')
        except AttributeError:
            return []
        # ...
        for card in cards:
            try:
                # Title
                # If the container is not about videos, there won't be a div with
                # attrs `role="heading"`. So to catch that, I've added a try-except
                # to catch the error and return.
                try:
                    title = card.find('div', role='heading').text
                except AttributeError:
                    return []
                # ...
            except IndexError:
                continue
            else:
                # ...
        return results

I re-structured the structure of the GoogleSpider class, so you might want to do the same thing as I did. I put all of the Part 1's code inside the __search_result method, and re-created the search function. All its doing is to call the private functions and put the results together:

class GoogleSpider(object):
    # ...

    def search(self, query: str, page: int = 1) -> dict:
        """Search Google

        Args:
            query (str): The query to search for
            page (int): The page number of search result

        Returns:
            dict: The search results and the total page number
        """
        # Get response
        response = self.__get_source(
            'https://www.google.com/search?q=%s&start=%d' % (quote(query), (page - 1) * 10))
        results = []
        video = self.__search_video(response)
        result = self.__search_result(response)
        pages = self.__get_total_page(response)
        results.extend(result)
        results.extend(video)
        return {
            'results': results,
            'pages': pages
        }

    # ...

Full Code

The full code of this tutorial until Part 2 is below.

# Import dependencies
from pprint import pprint
from urllib.parse import quote

import requests
from bs4 import BeautifulSoup


class GoogleSpider(object):
    def __init__(self):
        """Crawl Google search results

        This class is used to crawl Google's search results using requests and BeautifulSoup.
        """
        super().__init__()
        self.headers = {
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:79.0) Gecko/20100101 Firefox/79.0',
            'Host': 'www.google.com',
            'Referer': 'https://www.google.com/'
        }

    def __get_source(self, url: str) -> requests.Response:
        """Get the web page's source code

        Args:
            url (str): The URL to crawl

        Returns:
            requests.Response: The response from URL
        """
        return requests.get(url, headers=self.headers)

    def __search_video(self, response: requests.Response) -> list:
        """Search for video results based on the given response

        Args:
            response (requests.Response): the response requested to Google search

        Returns:
            list: A list of found video results, usually three if found
        """
        soup = BeautifulSoup(response.text, 'html.parser')
        try:
            cards = soup.find('g-scrolling-carousel').findAll('g-inner-card')
        except AttributeError:
            return []
        # Pre-process the video covers
        covers_ = soup.find('span', id='fld').findNextSiblings('script')[1:]
        # Get the cover images
        covers = []
        for c in covers_:
            # Fetch cover image
            try:
                covers.append(str(c).split('s=\'')[-1].split(
                    '\';var ii')[0].rsplit('\\', 1)[0])
            except IndexError:
                pass
        results = []
        # Generate video information
        for card in cards:
            try:
                # Title
                # If the container is not about videos, there won't be a div with
                # attrs `role="heading"`. So to catch that, I've added a try-except
                # to catch the error and return.
                try:
                    title = card.find('div', role='heading').text
                except AttributeError:
                    return []
                # Video length
                length = card.findAll('div', style='height:118px;width:212px')[
                    1].findAll('div')[1].text
                # Video upload author
                author = card.find(
                    'div', style='max-height:1.5800000429153442em;min-height:1.5800000429153442em;font-size:14px;padding:2px 0 0;line-height:1.5800000429153442em').text
                # Video source (Youtube, for example)
                source = card.find(
                    'span', style='font-size:14px;padding:1px 0 0 0;line-height:1.5800000429153442em').text
                # Video publish date
                date = card.find(
                    'div', style='font-size:14px;padding:1px 0 0 0;line-height:1.5800000429153442em').text.lstrip(source).lstrip('- ')  # Strip the source out because they're in the same container
                # Video link
                url = card.find('a')['href']
                # Video cover image
                try:  # Just in case that the cover wasn't found in page's JavaScript
                    cover = covers[cards.index(card)]
                except IndexError:
                    cover = None
            except IndexError:
                continue
            else:
                # Append result
                results.append({
                    'title': title,
                    'length': length,
                    'author': author,
                    'source': source,
                    'date': date,
                    'cover': cover,
                    'url': url,
                    'type': 'video'
                })
        return results

    def __get_total_page(self, response: requests.Response) -> int:
        """Get the current total pages

        Args:
            response (requests.Response): the response requested to Google using requests

        Returns:
            int: the total page number (might be changing when increasing / decreasing the current page number)
        """
        soup = BeautifulSoup(response.text, 'html.parser')
        pages_ = soup.find('div', id='foot', role='navigation').findAll('td')
        maxn = 0
        for p in pages_:
            try:
                if int(p.text) > maxn:
                    maxn = int(p.text)
            except:
                pass
        return maxn

    def __search_result(self, response: requests.Response) -> list:
        """Search for normal search results based on the given response

        Args:
            response (requests.Response): The response requested to Google

        Returns:
            list: A list of results
        """
        # Initialize BeautifulSoup
        soup = BeautifulSoup(response.text, 'html.parser')
        # Get the result containers
        result_containers = soup.findAll('div', class_='rc')
        # Final results list
        results = []
        # Loop through every container
        for container in result_containers:
            # Result title
            title = container.find('h3').text
            # Result URL
            url = container.find('a')['href']
            # Result description
            des = container.find('span', class_='st').text
            results.append({
                'title': title,
                'url': url,
                'des': des,
                'type': 'result'
            })
        return results

    def search(self, query: str, page: int = 1) -> dict:
        """Search Google

        Args:
            query (str): The query to search for
            page (int): The page number of search result

        Returns:
            dict: The search results and the total page number
        """
        # Get response
        response = self.__get_source(
            'https://www.google.com/search?q=%s&start=%d' % (quote(query), (page - 1) * 10))
        results = []
        video = self.__search_video(response)
        result = self.__search_result(response)
        pages = self.__get_total_page(response)
        results.extend(result)
        results.extend(video)
        return {
            'results': results,
            'pages': pages
        }


if __name__ == '__main__':
    pprint(GoogleSpider().search(input('Search for what? ')))

Summing Up

So now we can crawl Google's related video results, but you might be asking: Why can it only crawl 3 video results? Well, that's because in Google's source code, there's only three. If you found a way of scraping more, please leave a comment and I'll add it to the post as soon as I can. Of course, if you have any questions or having an error when coding, leave a comment below and I'll be happy to help.

Crawling Google Search Results - Part 1: Getting Started

Sam Zhang — Tue, 21 Jul 2020 07:24:34 +0000

Crawling websites are always fun. Have you ever thought about crawling Google search results? Well, in this tutorial, you'll create a dead-simple web spider to crawl Google's search results using Python. I assume that you know the basics about Python, BeautifulSoup and requests.

Getting Ready

First of all, install BeautifulSoup and requests using pip. I'll just skip that here 'cause I guess you guys know how to install them. Leave a comment if you are having trouble with installation.

Analyzing the Web Page

Search Google for something, for example, Python, and right-click to view the source code:

It's really a mess. Let's format it and remove the script tags and the style tags:

Wow, still 2000 lines of code! Let's see what's really important.

Important Things

Search for Welcome to Python.org (the first result when searching Python):

Then, search for Python (programming language) (the second result):

We can see that the two results are styled by the class rc. Great, now we've found the important class name - let's move on.

Notice that most of the class names in Google are generated randomly by a script. Never analyze these class names: your work might be wasted after you refresh the page!

Going Further

So now we've got the result container, let's work on the details. Take a look at the result container, and you can see that the link is right in an a element inside a div styled with the class r:

And the title in h3:

...and finally the description in a span with class st and surrounded by two divs. The outer one is styled by the class s:

Great! Now we've got the structure of the web page, and let's get to code!

Let's Code

Open your favorite code editor (I'm using VSCode), and open the folder you just created with the dependencies you just installed at the beginning of this tutorial. Create main.py and import the dependencies we needed:

# Import dependencies
from bs4 import BeautifulSoup
import requests

And let's create a class to put all of our code in it:

class GoogleSpider(object):
    def __init__(self):
        """Crawl Google search results

        This class is used to crawl Google's search results using requests and BeautifulSoup.
        """
        super().__init__()

    def __get_source(self, url: str) -> requests.Response:
        """Get the web page's source code

        Args:
            url (str): The URL to crawl

        Returns:
            requests.Response: The response from URL
        """
        pass

    def search(self, query: str) -> list:
        """Search Google

        Args:
            query (str): The query to search for

        Returns:
            list: The search results
        """
        pass

First, let's get the web page source. Take a look at the URL at your browser's address bar: there's lots of strange code that humans can't understand, right? Let's take them out and only leave the necessary ones:

https://www.google.com/search?q=Python

If you visit it, you can see that the page is the same than you just visited! Alright, let's see what's in the URL.

Analyzing the URL

We can see that the URL above only has one query param: q. You can see it like the query you are going to search. But, there's a problem: when you enter special chars, like +, %, (, ), and so on, Google won't understand them. We need to quote them to the form that websites can understand. To quote them, we need to use the function quote from urllib.parse:

# Import dependencies
from bs4 import BeautifulSoup
import requests
from urllib.parse import quote
# ...

Great, now we are ready to crawl the webpage!

Crawling the Web Page

It's pretty simple, you just need to call get() method from requests:

class GoogleSpider(object):
    # ...

    def __get_source(self, url: str) -> requests.Response:
        """Get the web page's source code

        Args:
            url (str): The URL to crawl

        Returns:
            requests.Response: The response from URL
        """
        return requests.get(url)

    # ...

Let's test it:

# ...
class GoogleSpider(object):
    # ...

    def search(self, query: str) -> list:
        """Search Google

        Args:
            query (str): The query to search for

        Returns:
            list: The search results
        """
        return self.__get_source('https://www.google.com/search?q=%s' % quote(query))


if __name__ == '__main__':
    print(GoogleSpider().search('Python'))

After you run the program, you'll likely be seeing the message below:

<Response [200]>

It looks normal, right? But, when you look at its source code:

print(GoogleSpider().search('Python').text)

You'll see that all of the class names are randomly generated! Don't worry, it's all because of one thing: the headers.

Configuring Headers

We need to define headers first. But, let's talk about headers now rather than coding. User-Agent headers are a special type of header that tells the website who requested this request. You can think like it's a name of the request sender. By default, requests will use python-requests/x.x.x, where as x.x.x is your requests version. A browser will send the User-Agent like this:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36

The UA above was sent by a Google Chrome browser on macOS 10.14. So, if we pretend that we are the browser, using the UA, we can easily go pass this "gate".

Alright, then let's get coding!

We need to define the headers as I said above. Let's define it in the __init__ method:

class GoogleSpider(object):
    def __init__(self):
        """Crawl Google search results

        This class is used to crawl Google's search results using requests and BeautifulSoup.
        """
        super().__init__()
        self.headers = {
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:79.0) Gecko/20100101 Firefox/79.0',
            'Host': 'www.google.com',
            'Referer': 'https://www.google.com/'
        }

    # ...

PS: I've added a few other headers to make us more "browser-like".

Then, we'll use the header to crawl Google:

class GoogleSpider(object):
    # ...

    def __get_source(self, url: str) -> requests.Response:
        """Get the web page's source code

        Args:
            url (str): The URL to crawl

        Returns:
            requests.Response: The response from URL
        """
        return requests.get(url, headers=self.headers)

    # ...

Now you run the program again, you'll see the normal source code. Next, we'll start the most important part of this spider: analyze the source code.

Using BeautifulSoup To Analyze the Source

Since I assume that you know BeautifulSoup basics, let's dive in and start coding.

First of all, we need to define a variable to store the response, instead of returning them:

class GoogleSpider(object):
    # ...

    def search(self, query: str) -> list:
        """Search Google

        Args:
            query (str): The query to search for

        Returns:
            list: The search results
        """
        # Get response
        response = self.__get_source('https://www.google.com/search?q=%s' % quote(query))

OK, then we'll initialize BS 4:

# Initialize BeautifulSoup
soup = BeautifulSoup(response.text, 'html.parser')

Then, we'll have to locate the result containers using soup.findAll():

# Get the result containers
result_containers = soup.findAll('div', class_='rc')

Finally, loop through every result and get the details:

# Final results list
results = []
# Loop through every container
for container in result_containers:
    # Result title
    title = container.find('h3').text
    # Result URL
    url = container.find('a')['href']
    # Result description
    des = container.find('span', class_='st').text
    results.append({
        'title': title,
        'url': url,
        'des': des
    })

At last, return the results:

return results

You can now run the program to see Google's search results. To search Google for different queries, you need to change the query param to the query you wanted to search for.

Full Code

The full code for this tutorial is below.

# Import dependencies
from bs4 import BeautifulSoup
import requests
from urllib.parse import quote
from pprint import pprint


class GoogleSpider(object):
    def __init__(self):
        """Crawl Google search results

        This class is used to crawl Google's search results using requests and BeautifulSoup.
        """
        super().__init__()
        self.headers = {
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:79.0) Gecko/20100101 Firefox/79.0',
            'Host': 'www.google.com',
            'Referer': 'https://www.google.com/'
        }

    def __get_source(self, url: str) -> requests.Response:
        """Get the web page's source code

        Args:
            url (str): The URL to crawl

        Returns:
            requests.Response: The response from URL
        """
        return requests.get(url, headers=self.headers)

    def search(self, query: str) -> list:
        """Search Google

        Args:
            query (str): The query to search for

        Returns:
            list: The search results
        """
        # Get response
        response = self.__get_source('https://www.google.com/search?q=%s' % quote(query))
        # Initialize BeautifulSoup
        soup = BeautifulSoup(response.text, 'html.parser')
        # Get the result containers
        result_containers = soup.findAll('div', class_='rc')
        # Final results list
        results = []
        # Loop through every container
        for container in result_containers:
            # Result title
            title = container.find('h3').text
            # Result URL
            url = container.find('a')['href']
            # Result description
            des = container.find('span', class_='st').text
            results.append({
                'title': title,
                'url': url,
                'des': des
            })
        return results


if __name__ == '__main__':
    pprint(GoogleSpider().search(input('Search for what? ')))

Conclusion

Of course, I know that this is really simple: it can't crawl video results, news results, picture results, and so on. But, it is a great template for learning web scraping in Python. Hope you enjoy this tutorial, and leave a comment if you are having trouble with anything in this tutorial.

DEV Community: Sam Zhang

I made a new web-components framework

What is Leafjs?

Getting Started

I built a terminal-like portfolio with Vue 3

Restful CRUD with Golang for beginners

Table of Contents

Getting started

Installing GORM

Defining database models

Connecting to database

RESTful APIs

Create

Read

Update

Delete

Conclusion

Learning Go Web Development From Zero

Installing Go

Installing Gin

Troubleshooting

Writing the first Gin request

Understanding the Hello World code

Live reloading

Conclusion

VueBlogger: A blogging site generator for Vue

What is VueBlogger?

Behind VueBlogger

How to use VueBlogger?

Configuration

General Configuration

Projects Configuration

About Yourself Configuration

Link Exchanges Configuration

Writing Posts

Publishing your blog

Running your blog locally

Building your blog manually

Last words

Creating a Personal Blog with Vue.js - Part 1: Getting Started

Starting Our Project

Choosing an UI Library

Creating the Basic Website

Emptying the Project

Working On the Blog

Making a Navbar

Summing Up

Found a New UI Library for Vue: PrimeVue

How I Found It

What's Packed

My thoughts

Crawling Google Search Results - Part 3: Fetching Wiki

Our Goal

Let's Analyze!

Container

Wiki Title

Wiki Subtitle

Wiki Description

Wiki Link

Wiki Details

Name

Description

Coding Time

Full Code

Conclusion

Crawling Google Search Results - Part 2: Crawling Video

What We Are Crawling

Analyze Time

Where's the Cover?

Alright, Hands On!

Full Code

Summing Up

Crawling Google Search Results - Part 1: Getting Started

Getting Ready

Analyzing the Web Page

Important Things

Going Further

Let's Code

Analyzing the URL

Crawling the Web Page