Hello Telescope

What is Telescope

Telescope is an example of planet: a tool for
aggregating blog feeds. It reminds me of google reader which I used for a long time to read the news.

Setup Environment

I am working on a fedora33 with KDE Plasma Desktop. Docker still not support CgroupsV2. It takes a little time to install docker and I have written a blog post for the installation https://dev.to/yzwdroid/fedora-33-docker-445k.

For some reason, redis has a permission error when docker-compose

Starting telescope_redis_1        ... done 
Starting telescope_elasticsearch_1 ... done 
Attaching to telescope_redis_1, telescope_elasticsearch_1 
redis_1         | find: '.': Permission denied 
redis_1         | chown: changing ownership of '.': Permission denied 
telescope_redis_1 exited with code 1 

So I installed Redis by sudo dnf install redis, and start it with
redis-server

All other part went well, I just follow the environment-setup.md

Modify goURL

To work with the REST API of telescope, I added a new flag -u --url, fetch
the json data, convert the data to a slice of post, get each post's contents by
the url. I just added one new function to make this work. Below is the diff I
made.

git diff HEAD main.go utils.go > lab6.diff

The diff

diff --git c/main.go w/main.go
index 4679a1a..fd3d249 100644
--- c/main.go
+++ w/main.go
@@ -25,6 +25,7 @@ var js = flag.BoolP("json", "j", false, "output json format to stdout")
 var fp = flag.StringP("file", "f", "", "file name to check")
 var ignore = flag.BoolP("ignore", "i", false, "ignore url patterns")
 var failOnly = flag.Bool("fails", false, "show only urls that failed")
+var urlFlag = flag.BoolP("url", "u", false, "read telescope restful API")

 func main() {
    flag.Parse()
@@ -43,14 +44,20 @@ go run main.go -v or --version check version.
        os.Exit(-1)
    }

-   dat, err := ioutil.ReadFile(*fp)
-   check(err)
-
+   var urls []string
+   var dat []byte
+   if *urlFlag {
+       dat = dataTelscope()
+   } else {
+       var err error
+       dat, err = ioutil.ReadFile(*fp)
+       check(err)
+   }
    // use xurls tool to exact links from file. Strict mod only match http://
    // and https:// schema
    rxStrict := xurls.Strict()
    // urls is a slice of strings
-   urls := rxStrict.FindAllString(string(dat), -1)
+   urls = rxStrict.FindAllString(string(dat), -1)
    urls = removeDuplicate(urls)

    if *ignore {
diff --git c/utils.go w/utils.go
index 5c27fc7..aee4859 100644
--- c/utils.go
+++ w/utils.go
@@ -2,7 +2,9 @@ package main

 import (
    "bufio"
+   "encoding/json"
    "fmt"
+   "io/ioutil"
    "log"
    "net/http"
    "os"
@@ -18,6 +20,43 @@ type urlStatus struct {
    Status int
 }

+type post struct {
+   ID  string
+   URL string
+}
+
+func dataTelscope() []byte {
+   var data []byte
+   var posts []post
+   resp, err := http.Get("http://localhost:3000/posts")
+   check(err)
+   defer resp.Body.Close()
+
+   if resp.StatusCode == http.StatusOK {
+       body, err := ioutil.ReadAll(resp.Body)
+       check(err)
+       if err := json.Unmarshal(body, &posts); err != nil {
+           panic(err)
+       }
+       for _, p := range posts {
+           resp, err := http.Get("http://localhost:3000" + p.URL)
+           if err != nil {
+               fmt.Println(err)
+           }
+           defer resp.Body.Close()
+           if resp.StatusCode == http.StatusOK {
+               bodyData, err := ioutil.ReadAll(resp.Body)
+               if err != nil {
+                   fmt.Println(err)
+               }
+               data = append(data, bodyData...)
+           }
+       }
+   }
+
+   return data
+}
+
 func removeDuplicate(urls []string) []string {
    result := make([]string, 0, len(urls))
    temp := map[string]struct{}{}

The gist version is here https://gist.github.com/yzwdroid/26b6cb71f75f3799dfccd5fb0ed9ec79