DEV Community: Zex

Back from Africa， AMA

Zex — Tue, 15 Oct 2019 00:48:48 +0000

Finally, I made my way to Africa. The major purpose for this trip was safaris and to meet my favorite, cheetah. It's aweeeeeeeesome.💪

Ask anything that comes to you mind 😆

SLI, SLO, and SLA

Zex — Sun, 25 Aug 2019 11:39:26 +0000

SLI - Service Level Indicator

Defines a measure of service level provided by service providers.
Examples

Uptime of the service
Number of Transactions

SLO - Service Level Objective

Defines a target value or a target range of service level, measured by SLI.
Examples

Availability of the service should be 99.999%

SLA - Service Level Agreement

Defines a commitment between service providers and customers.
Examples

Refund service fee
Provide free service for a period of time

The Nines

Percentages to measure the service.

Availability %	Downtime per year	Downtime per month	Downtime per week	Downtime per day
55.5555555% ("nine fives")	162.33 days	13.53 days	74.92 hours	10.67 hours
90% ("one nine")	36.53 days	73.05 hours	16.80 hours	2.40 hours
95% ("one nine five")	18.26 days	36.53 hours	8.40 hours	1.20 hours
97%	10.96 days	21.92 hours	5.04 hours	43.20 minutes
98%	7.31 days	14.61 hours	3.36 hours	28.80 minutes
99% ("two nines")	3.65 days	7.31 hours	1.68 hours	14.40 minutes
99.5% ("two nines five")	1.83 days	3.65 hours	50.40 minutes	7.20 minutes
99.8%	17.53 hours	87.66 minutes	20.16 minutes	2.88 minutes
99.9% ("three nines")	8.77 hours	43.83 minutes	10.08 minutes	1.44 minutes
99.95% ("three nines five")	4.38 hours	21.92 minutes	5.04 minutes	43.20 seconds
99.99% ("four nines")	52.60 minutes	4.38 minutes	1.01 minutes	8.64 seconds
99.995% ("four nines five")	26.30 minutes	2.19 minutes	30.24 seconds	4.32 seconds
99.999% ("five nines")	5.26 minutes	26.30 seconds	6.05 seconds	864.00 milliseconds
99.9999% ("six nines")	31.56 seconds	2.63 seconds	604.80 milliseconds	86.40 milliseconds
99.99999% ("seven nines")	3.16 seconds	262.98 milliseconds	60.48 milliseconds	8.64 milliseconds
99.999999% ("eight nines")	315.58 milliseconds	26.30 milliseconds	6.05 milliseconds	864.00 microseconds
99.9999999% ("nine nines")	31.56 milliseconds	2.63 milliseconds	604.80 microseconds	86.40 microseconds

Source: Wiki

Hold My Emoji

Zex — Mon, 12 Aug 2019 14:33:39 +0000

Let's create a random emoji generator in package emoji.

Prepare a list of funny emoji codes

🚲 🐸 🐡 🎉 😆 👏 🏄 🔞 🍫 🏖️ 🐳 💯 🤓 🌝 🤘 🤣 🍪 🌼 🔔 🎅 🍄 🚗 🦄 🎃 👈 :trollface: 🐒 🌞 :octocat: 👇 🐽 🛍️ 🍼 :feelsgood: 🍻 🙀 👀 ☃️ 🐣 🚀 🐮 💅 ‼️ 🚁

var (
  AvailEmojies = []string{
    ":bike:",
    ":frog:",
    ":blowfish:",
    ":tada:",
    ":laughing:",
    ":clap:",
    ":surfer:",
    ":underage:",
    ":chocolate_bar:",
    ":beach_umbrella:",
    ":whale:",
    ":100:",
    ":nerd_face:",
    ":full_moon_with_face:",
    ":metal:",
    ":rofl:",
    ":cookie:",
    ":blossom:",
    ":bell:",
    ":santa:",
    ":mushroom:",
    ":red_car:",
    ":unicorn:",
    ":jack_o_lantern:",
    ":point_left:",
    ":trollface:",
    ":monkey:",
    ":sun_with_face:",
    ":octocat:",
    ":point_down:",
    ":pig_nose:",
    ":shopping:",
    ":baby_bottle:",
    ":feelsgood:",
    ":beers:",
    ":scream_cat:",
    ":eyes:",
    ":snowman_with_snow:",
    ":hatching_chick:",
    ":rocket:",
    ":cow:",
    ":nail_care:",
    ":bangbang:",
    ":helicopter:",
  }
)

Prepare a helper function to generate a code every time being called. Using a random integer generation function in math/rand

func RndEmoji() string {
  return AvailEmojies[rand.New(rand.NewSource(time.Now().UnixNano())).Intn(len(AvailEmojies))]
}

Almost done, for now, you can use it somewhere else for fun. Here I'd like to use the test feature provided by go.
Now, create a test file with suffix _test in the same location as emoji.go, called emoji_test.go

import (
  "testing"
)

func TestEmoji(t *testing.T) {
  for i := 0; i < 10; i++ {
    t.Log(RndEmoji())
  }
}

Simple. Isn't it?
To run the test just go test it, use verbose mode to allow Log level info to print.
go test -v <GOPATH>/holdmyemoji

=== RUN   TestEmoji
--- PASS: TestEmoji (0.01s)
    emoji_test.go:9: :whale:
    emoji_test.go:9: :unicorn:
    emoji_test.go:9: :metal:
    emoji_test.go:9: :octocat:
    emoji_test.go:9: :monkey:
    emoji_test.go:9: :sun_with_face:
    emoji_test.go:9: :nail_care:
    emoji_test.go:9: :helicopter:
    emoji_test.go:9: :unicorn:
    emoji_test.go:9: :bell:
PASS

The test can also be built first by go test <GOPATH>/rndemoji -c -o emoji, then you can run ./emoji -test.v multiple times.

=== RUN   TestEmoji
--- PASS: TestEmoji (0.01s)
    emoji_test.go:9: :santa:
    emoji_test.go:9: :tada:
    emoji_test.go:9: :pig_nose:
    emoji_test.go:9: :rocket:
    emoji_test.go:9: :eyes:
    emoji_test.go:9: :full_moon_with_face:
    emoji_test.go:9: :100:
    emoji_test.go:9: :trollface:
    emoji_test.go:9: :pig_nose:
    emoji_test.go:9: :bangbang:
PASS

Also, to have more fun, shuffle the list every time before generating, the list changes from time to time. It, of course, will cost more computation.

  for i := range AvailEmojies {
    j := rand.New(rand.NewSource(time.Now().UnixNano())).Intn(i+1)
    AvailEmojies[j], AvailEmojies[i] = AvailEmojies[i], AvailEmojies[j]
  }

Use your funny list, play with it 🚀

Something About Grok

Zex — Sun, 11 Aug 2019 08:15:11 +0000

Here Grok, a filter plugin used in ELK stack. It converts unstructured logs to structured ones.

Grok is built upon regular expression. The syntax is %{SYNTAX:SEMANTIC}. For example:

A log line

10.0.1.13 GET /home HTTP/2.0 200 13969

A filter defined in logstash configure file as below

filter {
  grok {
    match => {
      "message" => '%{IPORHOST:ip} \"%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int})'
    }
  }
}

The log line passes through the filter, it produces the result

ip: 10.0.1.13
method: GET
request: /home
response: 200
bytes: 13969

With structured data, you can search by specific field while using Elasticsearch.

Grok Debugger

If you are not so sure the pattern would work on your log or not, here are some grok debugger to help you verify the patterns.

Make log processing easier. :D

What books are you reading recently?

Zex — Sun, 04 Aug 2019 14:41:08 +0000

Brainstorm: Some Thoughts About User Behavior Tracking

Zex — Mon, 22 Jul 2019 15:46:57 +0000

👉Remain on high level
👉No specific tools/frameworks included

Why it matters👈

Understand your users better
Narrow the gap between how you expect user to use and how user actually use it
Marketing data analytics

How to do it👇

On client side

Sync, send and wait
Async, send and go
Send in business components
Send in a separate service
Keep a local cache for unsent events for resend later
Local cache size limitation
API call timeout/failure handling

On server side

Log collection, centralize all log from all backend servers
Generate an event based on predefined event data structure on request
Direct to database
Publish over message queue

Employ a third-party service

Direct request
Integrate SDK
Shared database/separate database

Data definition☝️

Event

EventType, what happened
CreatedAt, when did it happen
Source, where did it happen
Payload, who/what was involved

Result

StatusCode
StatusMessage

API🌻

create

Create(Event) Result

Evil DFS: How To Successfully Distract Yourself

Zex — Thu, 18 Jul 2019 01:20:02 +0000

Reading link1
Found interesting link2
Go to link2
Read & found interesting link3
Go to link3
Read & found interesting link4
Go to link4
...
...
No more interesting link
Exit

Ok, this is DFS.

For some pages simply constructed with interesting links, it goes this path

For link1..n read

BFS is evil too. If it happened to include DFS cases 🙃 a day is over.

Pros:

Learn many by starting with one
Satisfy curiosity
Collect some topics for next time meet a nerd 🤓

Cons:

Time-consuming ｡ﾟ( ﾟஇ‸இﾟ)ﾟ｡
Possibly forget what was about to do

...Wait, that sounds like human web crawler 🤔

Here are some links for practice

-> redis sentinel
-> roshi on github
-> mixu's book
-> Finnish Defence Forces

OK, let's talk about another how-to.

How to break the loop before it is too late

Just open the links in another new tab without switching to them. This is easy to achieve by configure the browser or by right-click option. Supported by most browsers.
Keep a to-do list in whatever format that you like. I also use Trello. A list of interesting papers can also be found at my Github page. Check it out if interested.

Hopefully this help you be more productive😃

Play With Mattermost

Zex — Mon, 15 Jul 2019 14:06:06 +0000

Mattermost provides a series of API to interact with the server.

Let's create a service to post Mattermost message daily if we have new orders.

The new order message contains a plaintext message and an Excel file, which is generated by the service as well.

The message contains Emoji, just include them as plaintext.

We have new orders today!! 👏👏

msg := "We have new orders today!! :clap::clap:"

Upload the file first so that the post can include it by adding the file ID. data is the file content.

frsp, rsp := cli.UploadFile(data, chann_id, fd.Name())

Great! Now the file IDs are in frsp, which is a FileUploadResponse. Extract them and save in file_ids.

  post, rsp := cli.CreatePost(&mm.Post{
    FileIds: file_ids,
    ChannelId: chann_id,
    Message: msg,
  })

Then we have a new post on the channel with chann_id, members on that channel can see it. :D

Homebrew Performance Evaluator

Zex — Mon, 08 Jul 2019 14:52:19 +0000

Performance improvement is life-long work. First of all, you need to know how it does right now. The way to do it usually depends on the scale.

Developed a distributed performance test system before with Python, which took quite a long time to evolve and it's complicated. I wanted to try something different in this brand new project. Before introducing all kinds of performance test framework, I would like to make a simple and handy application.

There we Go.

Keep in mind: simple and handy.

Then it shall be able to

Send requests to API server
Evaluate the time cost
Generate some reports
Cookie support
Configure

The Evaluator

type Evaluator struct {
  cli *http.Client
  report []Item
  cookie *Cookie
}

Item to keep the request results

type Item struct {
  Url *url.URL `json:"url,omitempty"`
  Elapsed time.Duration `json:"elapsed,omitempty"`
  CreatedAt time.Time `json:"created_at,omitempty"`
}

Cookie to keep session-related data.

type Cookie struct {
  data map[string]interface{}
}

Get some data

func (self *Evaluator) Elapsed(req *http.Request) (*http.Response, error) {
  before := time.Now()
  rsp, err := self.cli.Do(req)
  self.report = append(self.report, Item{
    Url: req.URL,
    Elapsed: time.Now().Sub(before),
    CreatedAt: before,
  })
  return rsp, err
}

Time to print some results.

func (self *Evaluator) GenReport() error {
  fmt.Println()
  fmt.Println("-- report --")
  for _, v := range self.report {
    fmt.Println(v.String())
  }
  return nil
}

To get a pretty print, create a String function for Item as well.

func (self Item) String() string {
  return fmt.Sprintf("%v [%s:%v] %s", self.CreatedAt.Unix(), self.Url.Path, self.Url.RawQuery, self.Elapsed.String())
}

Maybe generate some CSV for further processing. I was using pretty print in Go, good to read but difficult to process later.

func (self *Evaluator) GenReportCSV(path string) error {
  // ...
  w := csv.NewWriter(fd)
  w.Write([]string{"Timestamp", "Api", "Query", "Elapsed"})
  for _, v := range self.report {
    if err = w.Write([]string{
        fmt.Sprintf("%d", v.CreatedAt.Unix()),
        v.Url.Path,
        fmt.Sprintf("%s", v.Url.RawQuery),
        fmt.Sprintf("%.4f", v.Elapsed.Seconds())});
      err != nil {
        return err
      }
  }
  w.Flush()
  return nil
}

Create a dummy client to do something crazy, wg is a WaitGroup
variable to wait for all crazy clients to finish their operations.

func dummy_terminal() {
  app := apps.NewApp()
  defer app.Close()

  // ... do something crazy to API ...

  if err := app.GenReport(); err != nil {
    fmt.Println("generate report failed: ", err)
  }

  out := fmt.Sprintf("%s/performance-%d.csv", *output_base, time.Now().UnixNano())
  if err := app.GenReportCSV(out); err != nil {
    fmt.Println("generate report csv failed: ", err)
  }
  wg.Done()
}

Configure in command line with flag

var(
  terminals   = flag.Int("terminals", 30, "total concurrent terminals")
  output_base = flag.String("output_base", "/tmp", "backend output_base")
)

Found a great logging module in Fabric :D

go get github.com/hyperledger/fabric/common/flogging

The CSV output looks like

Timestamp,Api,Query,Elapsed
1562142836,/api/home,,2.0599
1562142842,/api/list,,0.0311
1562142842,/api/somepage,id=99,0.0323
1562142842,/api/somepage,id=66,1.6131
1562142845,/api/somepage,id=33,0.0432

The console output is pretty much the same, except the time is formatted.

How Did I Recover From An Alcohol Party And Get Back To Work The Next Day

Zex — Sun, 30 Jun 2019 06:29:14 +0000

Even though I know I should not drink too much, sometimes I would just leave all the rules behind and enjoy the party.

So now what? I need to recover from that.

Soup

Usually, I have muffins and coffee for breakfast. In the morning after the party night, those are the last thing I want. The only thing came to my mind is soup.

Soup is easier to digest and absorb, provides nutrition, energy and water.

Sports & Energy Drink

Water is crucial to clean up the waste and residual alcohol in the body, but pure water is not enough. I need something to wake up my tongue.

Sports drink to supply the body inorganic salt, vitamins as well as water, also taste good.

Vitamin Supplements

I have some vitamin supplements with me, just took some as usual.

Sleep

Alcohol genuinely helps sleep, so no worry about it. Just get a comfortable place the body knows what to do.

Work hard and don't forget to have fun. Overdrinking alcohol is not encouraged still.🐒

Mini-Program, FTW?

Zex — Sun, 23 Jun 2019 03:16:30 +0000

If you haven't experienced WeChat you might not hear of mini-program. It's a sub-application in WeChat ecosystem.

In 90% of the time I work on backend development, mini program development is new to me, just got my hands on it this week.

Here is McDonald's order mini program.

A mini-program can be developed on Mac or Windows as the toolkits are available for only these two.

It usually contains

WXML, defines how the UI looks like.
WXSS, WeChat style sheet.
JS/TS, defines business logic.
Configs, define how it should run.

Similar to Web UI project except it uses some WeChat defined tags and API. Comparing to native apps, it's more lightweight. Check out the reference here

Unfortunately, the documents have poor globalization support. Most of the documents are originally written in Chinese. Well, they are improving it, English version can be found here.

Interested? Try it out🙃

Let's Talk About Garbage

Zex — Sun, 16 Jun 2019 05:36:11 +0000

Our city is seriously working on waste classification which is good for resource recycling and environmental protection.🐒 All waste must be dumped into the right place. Since July 1st, those
individuals don't obey the regulations could be fined up to 200RMB, for organizations could be up to 500K RMB.😢

Resource Management

Good developers borrow resource from the system and release them when work is done. But human being is lazy animal, so languages that "automatically" recycle the waste are created. Therefore, developers can focus on their business without bothering the resource stuff.

C++ doesn't have language level garbage collection due to implementation difficulties and performance consideration. A set of smart pointers are created as part of STL to serve the resource management purpose.

Languages like Java, Go, Python and JavaScript, performs GC automatically.

GC Techniques

Reference Counting

Maintains a counter for each object to denote references to this object. When an object is not referenced by anyone it would be freed. This leads to cyclic reference issue, meaning when two objects are referenced by each other, neither of them could be freed eventually.

Here come a workaround called weak reference, which doesn't protect the referenced object being recycled. It doesn't increase the reference count for the referent object.

Mark and Sweep

As a tracing garbage collector, it contains two stages: mark stage and sweep stage.

First of all the garbage collector walk through the objects graph from the root, find out those reachable objects and set the mark bit to 1, which is 0 by default.

Secondly, scans all the objects, recycle those unreachable (marked as 0), set the mark bit of those reachable to 1.

The downside of this method is it stops the program execution for quite a while.

Tri-Colour

It uses three colours, white, grey and black to differentiate states of objects.

The white set contains candidates to recycle.
The grey set contains objects that reachable from the root and have references to objects in the white set.
The black set contains objects that reachable from the root, have no reference to objects in the white set and aren't candidates to recycle.

Objects can only be moved from grey to black and from white to grey.

The algorithm is demonstrated as below


While grey set not empty:
  Pick an object from a grey set
  Move the object to black set
  Move white objects it references to grey set

It ensures that no black objects are referencing white objects, which are not reachable from the root and can be recycled once the grey set is empty. It can work on the fly without suspending the system for a period of time.

Implementations

Stop the World

The garbage collector comes with a
recycle threshold when closed or reached, a recycle happened.

It stops the program execution until the garbage collection is done. This brings high throughput for GC. It's suitable for non-interactive applications. By stopping the world for a noticeable period of time with a specific frequency it introduces performance issue as well.

Incremental GC

Performs garbage collection discretely by breaking the collection into pieces of processes and completes them one by one. Instead of freezing for a long time like STW does, it breaks it down to many tiny periods of time.

Concurrent GC

It runs garbage collection concurrently without stopping the world. Minimizing the pause time makes it a good choice for interactive programs. Low throughout makes it costs longer time to complete GC.

Garbage Collection in Languages

GC in Python

The gc module in Python allows you to interact with the GC system. It could be disabled by gc.disable() and manually GC by invoking gc.collect(). gc.garbage is a list of objects to be collected.

Interested in learning more about it? Check this out.😎

GC in Go

Go allows you to disable GC by running the application with GOGC=off.

Go comes with two knobs to control the GC. The first one is GCPercent. Basically this is a knob that adjusts how much CPU you want to use and how much memory you want to use. The default is 100 which means that half the heap is dedicated to live memory and half the heap is dedicated to allocation. You can modify this in either direction.

MaxHeap, which is not yet released but is being used and evaluated internally, lets the programmer set what the maximum heap size should be.

Learn more about the Go garbage collector from source code mgc.

GC in JavaScript

GC in Javascript is based on reachability and completely automatic. Found this article explains it very well, I recommend you read through it.

Helpful Resources

Fundamentals of Garbage Collection

Wiki:Garbage Collection