DEV Community: Mark Karamyar

Generic in Golang

Mark Karamyar — Wed, 17 Mar 2021 15:57:57 +0000

This year started with new news from the golang team, "Adding Generic!".

If you have familiar with languages like Java or C#, you already familiar with the concept of Generics. For example, let's talk about sorting an array.

package main

import (
    "fmt"
    "sort"
)

func main() {
    f := func(msg string, v []int) {
        fmt.Println(msg)
        for _, x := range v {
            fmt.Printf("%d ", x)
        }
        fmt.Println()
    }
    v := []int{10, 20, 100, 2, -1, 80}
    f("before sort", v)
    sort.Ints(v)
    f("after sort", v)
}

In the above code, we just used the sort package to sorting a slice of integers and print them out.

Everything looks good, right? So what if we wanted to sort a slice of other numeric types? The sort package has another function called Float64s (instead of Ints) but it's definitely not enough because we might need to do some sorting operation on other numeric types like int32.

So one approach is to define one function for each numeric type but we all know it's not the best idea! it's the place that Generic comes to picture.

Here's the definition of Generic in Wikipedia :

Generic programming is a style of computer programming in which algorithms are written in terms of types to-be-specified-later that are then instantiated when needed for specific types provided as parameters.

So the point of generic is, specify the type of the variable later and provide it as a parameter. That's exactly what we need for example in our sort function. The only thing that we need is to make sure our input comply our conditions. for example in the sorting scenario, it must be numeric.

Let's see how we can use generic in golang:

func Bored[T any](s ...T) T {
    return s[0]
}

Bored function gives a slice of T as input and returns the first item in the slice as output. So WTH is T? first thing first, let's see how to use our Bored function:

func main() {
    output := Bored("Hello", "Generics")
    fmt.Println(output)
}

That's great, a generic function is used exactly like a normal function! so let's talk about T: In this case, T is string. so what's the difference between generic and the old-fashioned go interface{}? I'll explain it with an example.

func Bored(s ...interface{}) interface{} {
    return "I am string"
}
func main() {
    output := Bored(1,1.1,"hello", struct{}{})
    fmt.Println(output)
}

The difference is, interface{} could be anything but in the first function the type of T determined while passing the first element to it. if it's a string, the rest of them also must be a string. otherwise, you'll get an error while compiling the application.


func Bored[T any](s ...T) T {
    return s[0]
}

func main() {
    output := Bored("Hello", "Generics", 1)
    fmt.Println(output)
}

output:

type checking failed for main
prog.go2:11:39: default type int of 1 does not match inferred type string for T

It's nice, isn't it? for Java or C# developers something like Repository<T> is pretty familiar which T is almost always one of the application's entities.

Back to Golang, the other type (instead of any) that introduced now, is comparable. It means the item must have the capability of being in equality operation.

package main

import (
    "fmt"
)

func main() {
    fmt.Println(Equal(1, 1)) // true
    fmt.Println(Equal(2, 1)) // false
}

func Equal[T comparable] (input1, input2 T) bool {
    return input1 == input2
}

and same previous example we got a compile-time if pass multiple data type to this function:

func main() {
    fmt.Println(Equal(1.1, 1))
}
/*
output: type checking failed for main
prog.go2:8:25: default type int of 1 does not match inferred type float64 for T
*/

I think the golang team provides some ways for developers to define the restriction of the type T themselves. for now, I just have seen any and comparable, if you know more typed parameter please share them in the comment section;

Working with big files in golang

Mark Karamyar — Sat, 13 Mar 2021 23:00:00 +0000

Today, I am going to show you how to read files in golang line-by-line. Let's imagine that have a jsonl file. What's jsonl? it's json lines in a simple term, it's a file that each line of it represents a valid json object. So if we read the file line by line, we can Marshal/Unmarshal each line of it separately. Here's an example of a jsonl file.

Each line of this file represents the data of a world cup.

{"year":"2018","host":"Russia","winner":"France"}
{"year":"2014","host":"Brazil","winner":"Germany"}
{"year":"2010","host":"South Africa","winner":"Spain"}
{"year":"2006","host":"Germany","winner":"Italy"}

Working with bufio.Scanner

So let’s read this file line by line most easily and conveniently. the easiest way to read a file (at least for me) is using the scanner from the bufio package in the standard library. First, we need to create an instance with the NewScanner function which is a very familiar way for constructing the structs in golang. this function accepts a Reader interface as input, and the good news is os.File implemented this interface. It means, we can open a file and pass the pointer of the file to bufio.NewScanner. Let’s see it in action.

// first open the file
file, err := os.Open("/Users/Mark/fifa-winners.jsonl")
if err != nil {
    log.Fatalf("could not open the file: %v", err)
}
// don't forget to close the file.
defer file.Close()
// finally, we can have our scanner
scanner := bufio.NewScanner(file)

So, we have the scanner, we are ready to go... scanner has a function named Scan. this function moves the scanner to the next token. I'll tell you what it means but for now, let's say each time Scan called, we read one line of our file. So if we want to move the scanner all through the file, we'd call the scan function in an infinite loop! The question is How do we know, we can break the loop? It's easy! Scan returns true as the return unless it meets the end of the file.

for {
    if scanner.Scan() {
        // we have a new line in each iteration
        continue
    }
    // we are done let's break the loop
    break
}
// the rest of our spaghetti

This code works, but have you know we can say golang to keep a loop running until met a specific condition. All the code above can be as simple as the code below:

for scanner.Scan() {
    // we have a new line in each iteration
}
// the rest of our spaghetti

Well, Let's get back to the point! we can have our bytes or string in each line easily by calling Bytes() and Text() functions.

for scanner.Scan() {
  // b is an array of bytes ([]byte)
  b := scanner.Bytes()
  // s is string
  s := scanner.Text()
}

Frankly, These functions are the same! for example string(scanner.Bytes()) will give you the same result and that's what exactly happens in the Text() function.

We read our file, so is the mission completed? Not exactly , because we didn't handle any error yet.

the scanner has another function called Err(). This function gives you the first error that happened during the scan process. It means, when the scanner trying to move through the file, the Scan function returns false, if something bad happened. So our loop instantly breaks and we will out of the loop. Now we can get that error and deal with it.

If we want to know in each line of the file that error happened, we should use a traditional way, we know the scanner starts from the beginning of the file (line number 1) so we can define a variable outside of the loop that represents the line number and increase it in each iteration.

lineNumber := 0
for scanner.Scan() {
    lineNumber++
        fmt.Println(scanner.Text())
}
// the rest of our spaghetti
if err := scanner.Err(); err != nil {
    log.Fatalf("something bad happened in the line %v: %v", lineNumber, err)
}

another thing that we should consider about the Err() function is it ignores the io.EOF so if we will give an error, it's a REAL one!

Let's have a run:

➜ big-files (main) ✗ go run main.go
{"year":"2018","host":"Russia","winner":"France"}
{"year":"2014","host":"Brazil","winner":"Germany"}
{"year":"2010","host":"South Africa","winner":"Spain"}
{"year":"2006","host":"Germany","winner":"Italy"}

It worked, So what's next?

Fix bufio.Scanner: token too long error

We said, each line of a jsonl file represents a valid json, so it could be too long. We also said the Scan function moves the scanner to the next token but the question is where is the next token!?

The scanner function has another method that is not as famous as its siblings, Buffer and it needs a buffer and an integer as input and you can set the maximum size of the buffer with this function.

bufio package has a maximum token size which equals 64 * 1024 (~65.6kb). So if one line of our lines is bigger than this size, we got this error token too long error.

We found the answer to our question: The next token is where the scanner reaches max size (default 65kb) OR the end of the line.

Approach 1: Bigger buffer size

The first approach to tackling this problem is to increase the buffer size. Actually, the name bufio.MaxScanTokenSize is a little misleading because it's not the actual maximum it's THE DEFAULT MAXIMUM size. so we can increase it.

buf := []byte{}
scanner := bufio.NewScanner(file)
// increase the buffer size to 2Mb
scanner.Buffer(buf, 2048*1024)

Now we can process jsonl files with lines up to 2Mb. It's good but what if we need more? We can increase this number as much as we want (probably) but if our file has 5.000.000 rows and just one of them is 100Mb we need to increase our scanner to this size just for one line, or use another approach!

Approach2

the next way to read such a tough file! is using another function! Bufio (buffer-io) gave us way more than a simple scanner to work with files and we have to choose one of them based on our needs and requirements. in this case, Scanner cannot satisfy what we need so let's take a look at ReadLine function of bufio.Reader. It's a little bit lower level than the scanner. generally speaking, when you hear the word lower-level you should do more for simple things but you have more access and power!

So let's get started. First we need a reader:

reader := bufio.NewReader(file)

reader, has ReadLine function which tries to read the entire line. Just like the scanner, we need to call this function in a for loop but since we are at the lower level! we don't have a nice-easy boolean in return anymore to know that we can break the loop.

the other difference is the error that we will give from the ReadLine function, which can also be io.EOF. It's not going to be a real error for us, so we have to handle it too.

reader := bufio.NewReader(file)
for {
    line, _, err := reader.ReadLine()
    if err != nil {
        if err == io.EOF {
            break
        }
        log.Fatalf("a real error happened here: %v\n", err)
    }
    fmt.Println(string(line))
}

As you probably already know, We just read the file so far, we did actually solve the problem that we had with the gigantic lines.

we ignore the second parameter that we gave from the ReadLine function and that one is what we exactly need to solve our problem. It's a boolean named isPrefix. If the line is too long and ReadLine cannot put all of its content in the buffer, It returns the filled buffer and set isPrefix to true which means we will give the next part of the line in the next call of the ReadLine function.

So we just need to call the ReadLine function until isPrefix becomes false then we can go for the next line of our file. You probably already noticed that we are talking about a recursive function. First I define the function that we want to call recursively.

func read(r *bufio.Reader) ([]byte, error) {
    var (
        isPrefix = true
        err error
        line, ln []byte
    )

    for isPrefix && err == nil {
        line, isPrefix, err = r.ReadLine()
        ln = append(ln, line...)
    }

    return ln, err
}

isPrefix is true at the first place and error is also nil so we make sure the for loop will run at least one time. It behaves like the do-while loop. We re-assign variables inside the loop so we call r.ReadLine unless we got an error OR isPrefix is false. in each iteration, we append the bytes that we get from r.ReadLine() to another variable. Now it's time to call this function inside the main function.

reader := bufio.NewReader(file)
for {
    line, err := read(reader)
    if err != nil {
        if err == io.EOF {
            break
        }
        log.Fatalf("a real error happened here: %v\n", err)
    }
    fmt.Println(string(line))
}

That's it! We solve the problem. here's the complete code:

package main

import (
    "bufio"
    "fmt"
    "io"
    "log"
    "os"
)

func main() {
    // first open the file
    file, err := os.Open("./fifa-winners.jsonl")
    if err != nil {
        log.Fatalf("could not open the file: %v", err)
    }
    defer file.Close()
    log.Println(" *******************READ WITH SCANNER*******************")
    readWithScanner(file)
    log.Println("*******************READ WITH READLINE()*******************")

    // we just reset the offset. because we read this file once
    // imagine the cursor is in the end of the file so we have to get back to the first line and read it again 
    file.Seek(0, 0)
    readWithReadLine(file)

    log.Println("we read a file twice!")
}

// Read with simple scanner

func readWithScanner(file *os.File) {
    // first open the file
    file, err := os.Open("./fifa-winners.jsonl")
    if err != nil {
        log.Fatalf("could not open the file: %v", err)
    }
    // finally, we can have our scanner
    buf := []byte{}
    scanner := bufio.NewScanner(file)
    scanner.Buffer(buf, 2048*1024)
    lineNumber := 1
    for scanner.Scan() {
        fmt.Println(scanner.Text())
        lineNumber++
    }
    // the rest of our spaghetti
    if err := scanner.Err(); err != nil {
        log.Fatalf("something bad happened in the line %v: %v", lineNumber, err)
    }
}

// Read with Readline function

func read(r *bufio.Reader) ([]byte, error) {
    var (
        isPrefix = true
        err error
        line, ln []byte
    )

    for isPrefix && err == nil {
        line, isPrefix, err = r.ReadLine()
        ln = append(ln, line...)
    }

    return ln, err
}

func readWithReadLine(file *os.File) {
    reader := bufio.NewReader(file)
    for {
        line, err := read(reader)
        if err != nil {
            if err == io.EOF {
                break
            }
            log.Fatalf("a real error happened here: %v\n", err)
        }
        fmt.Println(string(line))
    }
}

How to embed the content of a file to a variable in Golang

Mark Karamyar — Thu, 11 Mar 2021 23:00:00 +0000

In this article, we were talking about how to deal with a large file in Golang. But have you ever seen yourself in a situation that just wanted to read a file and load its entire content to a variable and work with it? I have been in the same situation a lot. For example for reading a specific configuration from a file and somehow, this file is always there! I mean you know that you need that file to build/run your application. Let's talk about it in code, Imagine we have a file named config.txt with the content below:

I am a file that you need me ;)

And wanted to read this file.

package main

import (
    "fmt"
    "io/ioutil"
    "log"
    "os"
)

func main() {
    file, err := os.Open("config.txt")
    if err != nil {
        log.Fatalf("missing config file: %v", err)
    }
    content, err := ioutil.ReadAll(file)
    if err != nil {
        log.Fatalf("could not read config file: %v", err)
    }
    fmt.Println(string(content))
}
// output: I am a file that you need me ;)

if the code above sounds familiar to you, I have great news for you, Go offers a better, nice, and shorter way to do that. There is a package called embed do all of it for you in a single line. All you need is just pass the file path in a directive above your variable

package main

import (
    _ "embed"
    "fmt"
)

//go:embed config.txt
var content string

func main() {
    fmt.Print(content)
}
// output: I am a file that you need me ;)

That's it! you have the content of the file in the content variable! do you need to have it in []byte? you just need to change your variable type.

package main

import (
    _ "embed"
    "fmt"
)

//go:embed config.txt
var content []byte

func main() {
    fmt.Print(content)
}
// output: [73 32 97 109 32 97 32 102 105 108 101 32 116 104 97 116 32 121 111 117 32 110 101 101 100 32 109 101 32 59 41]

Error handling in goroutines with errgroup

Mark Karamyar — Mon, 11 Jan 2021 16:53:52 +0000

Golang made life easier with goroutine , however, sometimes it's difficult to handle errors that happened inside a goroutine effectively. For example, imagine you have an array of some kind of actions and wanted to run a specific function on each one of them. On the other hand, in case of an error, you want to propagate that error to the higher function.

Let's explain it in the code, Imagine that we have a set of runners and each runner has a Handle function.

type HandlerFunc func(input string) error
type Runner struct {
    Name string
    Handle HandlerFunc
}

I also like to define another type named Runners. It's just a simple wrapper around arrays of runners

type Runners []Runner

hence I can define a function that runs through all runners like this:

func (r Runners) Execute() error {
    for _, runner := range r {
        if err := runner.Handle(runner.Name); err != nil {
            return err
        }
    }
    return nil
}

and finally, define some runners and execute them:

func main() {
    runners := Runners{
        Runner{
            Name: "1",
            Handle: func(input string) error {
                fmt.Printf("runner %s is running\n", input)
                return nil
            },
        },
        Runner{
            Name: "2",
            Handle: func(input string) error {
                return fmt.Errorf("something bad happened in runner [%s]", input)
            },
        },
        Runner{
            Name: "3",
            Handle: func(input string) error {
                fmt.Printf("runner %s is running\n", input)
                return nil
            },
        },
    }

    err := runners.Execute()
    if err != nil {
        fmt.Printf("execution failed: %v", err)
    }  
}

By running this piece of code I get this output: runner 1 is running. The problem is we didn't run the third runner. So in this case we print errors and continue running the Execution but we cannot propagate the error to the higher function!

func (r Runners) Execute() error {
    for _, runner := range r {
        if err := runner.Handle(runner.Name); err != nil {
      fmt.Printf("error happened in runner [%s]: %v", runner.Name, err)
        }
    }
    return nil
}

Now, What if we want to run each runner in a different goroutine with the exact same scenario? Let's change a code a little bit to see if it works.

First thing first we add a go behind the function call. So our Execute function should be something like it:

func (r Runners) Execute() error {
    for _, runner := range r {
        go func(runner Runner) error {
            if err := runner.Handle(runner.Name); err != nil {
                return err
            }
            return nil
        }(runner)
    }
    return nil
}

However, As you probably know if we run the program again, there will be nothing in output, because we never said the application to wait until the goroutines finish their work. for sake of simplicity, let's just add a Sleep to the main function:

err := runners.Execute()
time.Sleep(3 * time.Second)
if err != nil {
    fmt.Printf("execution failed: %v", err)
}

Now, let's run it again. this time the output should be something like this:

runner 1 is running
runner 3 is running

OK, it's not OK! The execution worked well because we could execute runners 1 and 3, however, we still didn't do anything about the error.

Welcome to errgroup

Now, it's time to solve the problem with errgroup. It's REALLY simple and easy. It works like waitgroups under the sync package. Honestly, it's using wait groups behind the scene but since the mentioned scenario is quite common, errgroup make life easier for us!

If you're familiar with wait group, You should know what wg.Done() and wg.Wait() mean. errgroup offeres same thing. Let's make things clear in code. First, we declare an g variable

g := new(errgroup.Group)

and run our execute function inside a goroutine with Go func. so our Execute function turns to this:

func (r Runners) Execute() {
    for _, runner := range r {
        rx := runner
        g.Go(func() error {
            return rx.Handle(rx.Name)
        })
    }
}

the Go function gives and func which returns an error, if this error is not nil you will have that in the returns of the Wait function.

Here's the Wait func:

runners.Execute()
err := g.Wait()
if err != nil {
    fmt.Printf("execution failed: %v", err)
}

As you see, we don't need time.Sleep() anymore, because the Wait func, waits until the last goroutine gets the result and returns the first error that happened. If all functions run without error, it returns nil

the output is something like it:

runner 3 is running
runner 1 is running
execution failed: something bad happened in runner [2]

and here's the complete code:

package main

import (
    "fmt"

    "golang.org/x/sync/errgroup"
)

var g errgroup.Group

type HandlerFunc func(input string) error

type Runner struct {
    Name string
    Handle HandlerFunc
}

type Runners []Runner

func (r Runners) Execute() {

    for _, runner := range r {
        rx := runner
        g.Go(func() error {
            return rx.Handle(rx.Name)
        })
    }
}

func main() {
    runners := Runners{
        Runner{
            Name: "1",
            Handle: func(input string) error {
                fmt.Printf("runner %s is running\n", input)
                return nil
            },
        },
        Runner{
            Name: "2",
            Handle: func(input string) error {
                return fmt.Errorf("something bad happened in runner [%s]", input)
            },
        },
        Runner{
            Name: "3",
            Handle: func(input string) error {
                fmt.Printf("runner %s is running\n", input)
                return nil
            },
        },
    }
    runners.Execute()
    err := g.Wait()
    if err != nil {
        fmt.Printf("execution failed: %v", err)
    }
}

chromedp: How to get the URL of the current page

Mark Karamyar — Mon, 04 Jan 2021 19:05:16 +0000

In the previous article, we developed a simple application and open google.com with chromedp. in this article we are going to take a look and one of the chromedp functionalities to deal with the page and URL.

The way that the developers decided to use to do that is a very popular way along with the golang developers, Passing the reference of a variable to the function. I see the same pattern a lot in go libraries, for example, GORM also using the same pattern.

In today’s scenario, we want to open the chromedp Github page and go to one of the files inside the repository and print the URL of the page. Simple and easy! so let’s get started!!!

For this purpose, we need to select the element that we want to click on it. Remember we just want to make things that we do manually, automatically. it means if we want to open the file named util_test.go, first, we need to go to the URL https://github.com/chromedp/chromedp then find the file inside the list and then click on it. For finding the elements inside the page we can simply use CSS Selectors.

// our selector to find the file inside the page.
selector := "a[title='util_test.go']"
tasks := chromedp.Tasks{
  // go to the page
    chromedp.Navigate("https://github.com/chromedp/chromedp"),
  // wait until the elemnt is visible and available inside the page
    chromedp.WaitEnabled(selector),
  // do a click on it
    chromedp.Click(selector),
}

so we already go to the page that we wanted the only step that still remains is getting the page URL. For this purpose, we define a string variable and pass it to the Location function of chromedp.

var u string
...
chromedp.Location(&u),

and here's the complete code:

package main

import (
    "context"
    "fmt"

    "github.com/chromedp/chromedp"
)

func main() {
    var u string
    opts := append(
        // select all the elements after the third element
        chromedp.DefaultExecAllocatorOptions[3:],
        chromedp.NoFirstRun,
        chromedp.NoDefaultBrowserCheck,
    )
    // create chromedp's context
    parentCtx, cancel := chromedp.NewExecAllocator(context.Background(), opts...)
    defer cancel()

    selector := "a[title='util_test.go']"
    tasks := chromedp.Tasks{
        chromedp.Navigate("https://github.com/chromedp/chromedp"),
        chromedp.WaitEnabled(selector),
        chromedp.Click(selector),
        chromedp.Location(&u),
    }
    ctx, cancel := chromedp.NewContext(parentCtx)
    defer cancel()
    if err := chromedp.Run(ctx, tasks); err != nil {
        panic(err)
    }
    fmt.Printf("URL => %s\n", u)
}

the output is:

URL => https://github.com/chromedp/chromedp/blob/master/util_test.go

chromedp: Working with nodes and tabs

Mark Karamyar — Sun, 03 Jan 2021 22:27:01 +0000

Today we are going to have some fun with chromedp. Today's scenario is gathering all the links on a page and loop through them. It should be simple, easy, and fun. let's get started.

As we discussed before, for doing this kind of actions in chromedp, we need to pass the pointer of our variable to the corresponding function.

We wanted to go to the Notpad++ website, print all links and open them in different tab. So first we need to define our pointer to the array of nodes.

var nodes []*cdp.Node

and then fill it with the Nodes function.

selector := "#main ul li a"
pageURL := "https://notepad-plus-plus.org/downloads/"
chromedp.Run(ctx, chromedp.Tasks{
    chromedp.Navigate(pageURL),
    chromedp.WaitReady(selector),
    chromedp.Nodes(selector, &nodes),
})

so let's take a look at the nodes and see what we have on them.

for _, n := range nodes {
  u := n.AttributeValue("href")
    fmt.Printf("node: %s | href = %s\n", n.LocalName, u)
}

the output is:

node: a | href = https://notepad-plus-plus.org/downloads/v7.9.2/
node: a | href = https://notepad-plus-plus.org/downloads/v7.9.1/
node: a | href = https://notepad-plus-plus.org/downloads/v7.9/
node: a | href = https://notepad-plus-plus.org/downloads/v7.8.9/
node: a | href = https://notepad-plus-plus.org/downloads/v7.8.8/
node: a | href = https://notepad-plus-plus.org/downloads/v7.8.7/
node: a | href = https://notepad-plus-plus.org/downloads/v7.8.6/
node: a | href = https://notepad-plus-plus.org/downloads/v7.8.5/
node: a | href = https://notepad-plus-plus.org/downloads/v7.8.4/
node: a | href = https://notepad-plus-plus.org/downloads/v7.8.3/
node: a | href = https://notepad-plus-plus.org/downloads/v7.8.2/
...

so we grab all links, let's store them in an array, and then open them in a new tab. for opening a new tab in chromedp we need to create a new context from the context that we already have. We have a function name NewContext that does it for us.

clone, cancel := chromedp.NewContext(ctx)
defer cancel()

from now on, the clone context will do everything in a new tab. the rest of the process is exactly the same so we can easily run chromedp tasks in our new context.

for _, n := range nodes {
    u := n.AttributeValue("href")
    clone, cancel := chromedp.NewContext(ctx)
    defer cancel()
    chromedp.Run(clone, chromedp.Navigate(u))
}

if you disable headless mode and run the project, you see that URLs are opening one by one in a new tab. however, we can open all of them in a manner of second by using goroutine

f := func(ctx context.Context, url string) {
    clone, cancel := chromedp.NewContext(ctx)
    defer cancel()
    chromedp.Run(clone, chromedp.Navigate(url))
}
for _, n := range nodes {
    u := n.AttributeValue("href")
    go f(ctx, u)
}

that's it, and here's the complete code:

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/chromedp/cdproto/cdp"
    "github.com/chromedp/chromedp"
)

func main() {
    ctx, cancel := chromedp.NewContext(context.Background(), chromedp.WithErrorf(log.Printf))
    defer cancel()
    var nodes []*cdp.Node
    selector := "#main ul li a"
    pageURL := "https://notepad-plus-plus.org/downloads/"
    if err := chromedp.Run(ctx, chromedp.Tasks{
        chromedp.Navigate(pageURL),
        chromedp.WaitReady(selector),
        chromedp.Nodes(selector, &nodes),
    }); err != nil {
        panic(err)
    }
    f := func(ctx context.Context, url string) {
        clone, cancel := chromedp.NewContext(ctx)
        defer cancel()
        fmt.Printf("%s is opening in a new tab\n", url)

        if err := chromedp.Run(clone, chromedp.Navigate(url)); err != nil {
            // do something nice with you errors!
            panic(err)
        }
    }
    for _, n := range nodes {
        u := n.AttributeValue("href")
        go f(ctx, u)
    }
}

chromedp: Get started to surfing the web with Go

Mark Karamyar — Sat, 02 Jan 2021 18:30:12 +0000

chromedp helps you to flip through! the web fast , easy, and programmatically in Golang. If you have ever worked with Selenium or PhantomJS or other similar tools, this concept is familiar to you.

In this article we are going to get start working with chromedp and do some simple tasks with it. Let's get started.

First thing first you have to add chromedp to your project's dependency. It can done just like other golang packages that you used before:

go get -u github.com/chromedp/chromedp

for starting work with chromedp, you do not need to do a lot! the only thing that you need to do is creating a context and start working with it;

ctx, cancel := chromedp.NewContext(context.Background())
defer cancel()
chromedp.Run(ctx, chromedp.Navigate("https://www.google.com"))

That's it. Trust me, chromedp just open google even if you don't see anything yet! the reason that you didn't see the browser is by default it runs in headless mode and as I expected we can change the behavior. Actually, there are a list of default behaviors that passed to the library by default

// DefaultExecAllocatorOptions are the ExecAllocator options used by NewContext
// if the given parent context doesn't have an allocator set up. Do not modify
// this global; instead, use NewExecAllocator. See ExampleExecAllocator.
var DefaultExecAllocatorOptions = [...]ExecAllocatorOption{
    NoFirstRun,
    NoDefaultBrowserCheck,
    Headless,

    // After Puppeteer's default behavior.
    Flag("disable-background-networking", true),
    Flag("enable-features", "NetworkService,NetworkServiceInProcess"),
    Flag("disable-background-timer-throttling", true),
    Flag("disable-backgrounding-occluded-windows", true),
    Flag("disable-breakpad", true),
    Flag("disable-client-side-phishing-detection", true),
    Flag("disable-default-apps", true),
    Flag("disable-dev-shm-usage", true),
    Flag("disable-extensions", true),
    Flag("disable-features", "site-per-process,TranslateUI,BlinkGenPropertyTrees"),
    Flag("disable-hang-monitor", true),
    Flag("disable-ipc-flooding-protection", true),
    Flag("disable-popup-blocking", true),
    Flag("disable-prompt-on-repost", true),
    Flag("disable-renderer-backgrounding", true),
    Flag("disable-sync", true),
    Flag("force-color-profile", "srgb"),
    Flag("metrics-recording-only", true),
    Flag("safebrowsing-disable-auto-update", true),
    Flag("enable-automation", true),
    Flag("password-store", "basic"),
    Flag("use-mock-keychain", true),
}

As you can see, the third item in the array is Headless so we can modify it. Let's create our configs for creating a new context.

opts := append(
  // select all the elements after the third element
    chromedp.DefaultExecAllocatorOptions[3:],
    chromedp.NoFirstRun,
    chromedp.NoDefaultBrowserCheck,
)

the latest step is to create an ExecAllocator and pass it as our parent context.

parentCtx, _ := chromedp.NewExecAllocator(context.Background(), opts...)
ctx, _ := chromedp.NewContext(parentCtx)

Now if you run the project you will see the browser open and navigates to the google website. here's the complete code:

package main

import (
    "context"
    "fmt"

    "github.com/chromedp/chromedp"
)

func main() {
    opts := append(
        // select all the elements after the third element
        chromedp.DefaultExecAllocatorOptions[3:],
        chromedp.NoFirstRun,
        chromedp.NoDefaultBrowserCheck,
    )
    // create chromedp's context
    parentCtx, cancel := chromedp.NewExecAllocator(context.Background(), opts...)
    defer cancel()

    ctx, cancel := chromedp.NewContext(parentCtx)
    defer cancel()

    if err := chromedp.Run(ctx, chromedp.Navigate("https://www.google.com")); err != nil {
        panic(err)
    }

    fmt.Println("I've just saw Google!!!")
}

in MacOS

Mark Karamyar — Thu, 31 Dec 2020 23:00:00 +0000

The stdc++.h file is just a file with a bunch of include. the only thing that you have to do is to put this file in the default location of the c++ import files. Here's a quick way to add it in MacOS.

With XCode installed

create a file named stdc++.h in this path: (you should create bits folder first)

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/bits

Without XCode

create a file named stdc++.h in this path: (you should create bits folder first)

/usr/local/include/bits

Now, simply fill the created file with the content below:

// C++ includes used for precompiling -*- C++ -*-

// Copyright (C) 2003-2013 Free Software Foundation, Inc.
//
// This file is part of the GNU ISO C++ Library. This library is free
// software; you can redistribute it and/or modify it under the
// terms of the GNU General Public License as published by the
// Free Software Foundation; either version 3, or (at your option)
// any later version.

// This library is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.

// Under Section 7 of GPL version 3, you are granted additional
// permissions described in the GCC Runtime Library Exception, version
// 3.1, as published by the Free Software Foundation.

// You should have received a copy of the GNU General Public License and
// a copy of the GCC Runtime Library Exception along with this program;
// see the files COPYING3 and COPYING.RUNTIME respectively. If not, see
// <http://www.gnu.org/licenses/>.

/** @file stdc++.h
 * This is an implementation file for a precompiled header.
 */

// 17.4.1.2 Headers

// C
#ifndef _GLIBCXX_NO_ASSERT
#include <cassert>
#endif
#include <cctype>
#include <cerrno>
#include <cfloat>
#include <ciso646>
#include <climits>
#include <clocale>
#include <cmath>
#include <csetjmp>
#include <csignal>
#include <cstdarg>
#include <cstddef>
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <ctime>

#if __cplusplus >= 201103L
#include <ccomplex>
#include <cfenv>

or you can download the file and put it the right folder:

mkdir /usr/local/include/bits
curl https://gist.githubusercontent.com/devmarkpro/94d0b20a87bd0cc5f094a9b98336aa6d/raw/724c53e3210029f7bfd5dfa6254d124791547153/stdc++.h > /usr/local/include/bits/stdc++.h

NOTE: You have to repeat these steps each time that you update the XCode Command-line Tools so if you know a permanent solution for this problem, please share it in the comment section.