DEV Community

loading...

Discussion on: Comparing the same web scraper in Haskell, Python, Go

Collapse
ryantenorio profile image
Ryan

I would recommend Colly (github.com/gocolly/colly) to get a better comparison since you are using BeautifulSoup for Python. Both scraper libraries have superb APIs.

Collapse
yujiri8 profile image
Ryan Westlund Author

Wow! I didn't know about that library. That does much more for me here than even BeautifulSoup! New&Improved Go version:

package main

import (
    "fmt"

    "github.com/gocolly/colly"
)

func main() {
    var col = colly.NewCollector()
    col.OnHTML("#recentcomments > *:first-child", func(e *colly.HTMLElement) {
        fmt.Println(e.Text)
    })
    col.Visit("https://fakenous.net")
}

That gets it down to about same number of "meaningful" lines as Python. Technically can drop 2 more lines by putting the function inline, but I wouldn't do that IRL.