DEV Community: Ryo Nakao

Announcing PipeCD

Ryo Nakao — Tue, 06 Oct 2020 14:00:32 +0000

This article is originally posted here.

Today we are excited to announce the open-source availability of PipeCD: a continuous delivery system for declarative Kubernetes, Serverless, and Infrastructure applications.
PipeCD aims to provide a unified CD solution for multiple application kinds on multi-cloud that empowers engineers to deploy faster with more confidence.
It is also available as a GitOps tool that enables doing deployment operations by pull request on Git.

Background

As one of our Developer Productivity team's missions, we aim to empower engineers to deploy their services faster, more frequently with reliability.
Martin Fowler, in his book Continuous Delivery, points out that "The biggest risk to any software effort is that you end up building something that isn't useful. The earlier and more frequently you get working software in front of real users, the quicker you get feedback to find out how valuable it really is."

Recently, with the popularity of cloud services and the container technology, engineers have even more options in choosing the infrastructure model, the cloud services, which are most suitable for their team's requirements.
At CyberAgent, we have a large number of services from many teams where each team can have a different infrastructure model and a different cloud service. Some big projects also deploy their services on multi-cloud.
This diversification leads to a problem that we are facing, lacking a robust CD system for all teams.

So we decided to create a new CD system that provides a unified interface for many application kinds to improve the developer experience.

Key Features

While designing PipeCD, we focused on the following 4 key features with the aim of creating a CD system that provides a good experience for both developers and operators.

Visibility

Visibility is one of the most requested factors we received when surveying engineers in our company.
Visibility is the ability to see what's going on in the cluster, the ability to see how each component of the application had been deployed, the ability to know quickly why the deployment was failed.
Visibility for a team leader is the ability to know the delivery performance of the team and what metrics should be improved.
With PipeCD, we always strive to maximize the visibility for engineers, operators as well as team leaders. Currently, it includes:

Deployment pipeline UI shows clarify what is happening
Separate logs viewer for each individual deployment
Realtime visualization of application component and state
Deployment notifications to slack, webhook endpoints
Insights show the delivery performance

In addition, the entire state of the service is managed through Git, so you can view the whole state of the cluster and all audit logs provided by Git.

Automation

Automation reduces or removes repetitive overhead of frequent releases. So maximizing automation helps to minimize human error during the deployment process, as well as reduce the amount of work engineers need to do.
PipeCD has the following automated functionalities:

Automated deployment analysis based on metrics, logs, emitted requests
Automatically roll back to the previous state as soon as analysis or a pipeline stage fails
Automatically detect configuration drift to notify and render the changes
Automatically watch and detect the new container images to deploy

Secure

The CD system often carries a lot of credentials needed to access the cluster and the necessary services of the teams.
Ensuring the safety of the teams is always on our top priority.
So while designing the PipeCD, we decided not to store those credentials in a central place. Instead of that, all user's credentials always stay inside their clusters.

Support single sign-on and role-based access control
Credentials are not exposed outside the cluster and not saved in the control-plane
Piped makes only outbound requests and can run inside a restricted network

Multi-provider & Multi-Tenancy

Multi-provider means supporting multiple cloud services, multiple container registries, multiple monitoring services for doing deployment analysis.
You can use PipeCD to deploy your Kubernetes applications, CloudRun, AWS Lambda application and Terraform application.
It also supports doing progressive delivery with canary and bluegreen strategy.

Easy to operate multi-cluster, multi-tenancy by separating control-plane and piped
Support multiple analysis providers including Prometheus, Datadog, Stackdriver, and more
Support multiple application kinds on multi-cloud including Kubernetes, Terraform, Cloud Run, AWS Lambda

While designing PipeCD, we simplified its architecture by minimizing the number of components, so you do not have to install many things to enable all features.
In addition, PipeCD also supports storing data in several fully-managed services to minimize the operating cost.

Currently, we have completed the basic features and many of the features are in the alpha status. We are working hard to increase the stability and planning to release a stable version in the next months.

Community

PipeCD team hopes to receive a warm welcome and feedback from the open-source community.
We value every contribution and invite you to join us on GitHub, Slack and Twitter.

Visit our website and documentation at https://pipecd.dev
Check out the code at https://github.com/pipe-cd/pipe or explore the examples
Join us on Slack @cloud-native/pipecd to chat with other developers
Follow us on Twitter @pipecd_dev to get the latest news

PipeCD team is hiring engineers/interns to work on PipeCD. Please contact us if you are interested.

Thanks

Finally, we would like to thank the existing open-source CD projects like Spinnaker, FluxCD, ArgoCD... PipeCD has been built on many great ideas from those great projects.
PipeCD team would also like to thank CyberAgent's engineers and collaborators from other companies, who have sent us so many valuable feedback throughout the development process to this day.

Take a walk the Go AST

Ryo Nakao — Mon, 28 Sep 2020 08:54:25 +0000

This article is originally posted here.

What do you refer to when you're curious about the Go AST? A document? Source code?
While reading the documentation helps you understand it in the abstract, you can't see how APIs relate to each other, etc.
Not to mention, read the entire source code and you'll see it completely, but you should get tired from trying to read the whole stuff.

Therefore, this article aims to be in the middle of them. Let's take it easy and take a walk through AST to understand how the Go code we usually write is represented internally.

It doesn't delve into how to parse the source code, and starts with the description after the AST is built.
If you're curious about how the code is converted to the AST, navigate to Digging deeper into the analysis of Go-code.

Let’s get started!

Interfaces

First up, let me briefly describe the interface that represents each of the AST nodes.
All AST nodes implement the ast.Node interface, which just returns a position in the AST.

In addtion, there are 3 main interfaces that implement ast.Node:

ast.Expr — representing expressions and types node
ast.Stmt — representing statement node
ast.Decl — representing declaration node

You can see from the definition that every Node satisfies the ast.Node interface:

ast/ast.go

// All node types implement the Node interface.
type Node interface {
    Pos() token.Pos // position of first character belonging to the node
    End() token.Pos // position of first character immediately after the node
}

// All expression nodes implement the Expr interface.
type Expr interface {
    Node
    exprNode()
}

// All statement nodes implement the Stmt interface.
type Stmt interface {
    Node
    stmtNode()
}

// All declaration nodes implement the Decl interface.
type Decl interface {
    Node
    declNode()
}

Getting started with walking

Let's start walking! Take a look at the file we convert to the AST:

package hello

import "fmt"

func greet() {
    fmt.Println("Hello, World")
}

Nothing fancy — an overly simple Hello, World program. The AST built on this is:

ast.File

*ast.File {
.  Package: dummy.go:1:1
.  Name: *ast.Ident {
.  .  NamePos: dummy.go:1:9
.  .  Name: "hello"
.  }
.  Decls: []ast.Decl (len = 2) {
.  .  0: *ast.GenDecl {
.  .  .  TokPos: dummy.go:3:1
.  .  .  Tok: import
.  .  .  Lparen: -
.  .  .  Specs: []ast.Spec (len = 1) {
.  .  .  .  0: *ast.ImportSpec {
.  .  .  .  .  Path: *ast.BasicLit {
.  .  .  .  .  .  ValuePos: dummy.go:3:8
.  .  .  .  .  .  Kind: STRING
.  .  .  .  .  .  Value: "\"fmt\""
.  .  .  .  .  }
.  .  .  .  .  EndPos: -
.  .  .  .  }
.  .  .  }
.  .  .  Rparen: -
.  .  }
.  .  1: *ast.FuncDecl {
.  .  .  Name: *ast.Ident {
.  .  .  .  NamePos: dummy.go:5:6
.  .  .  .  Name: "greet"
.  .  .  .  Obj: *ast.Object {
.  .  .  .  .  Kind: func
.  .  .  .  .  Name: "greet"
.  .  .  .  .  Decl: *(obj @ 23)
.  .  .  .  }
.  .  .  }
.  .  .  Type: *ast.FuncType {
.  .  .  .  Func: dummy.go:5:1
.  .  .  .  Params: *ast.FieldList {
.  .  .  .  .  Opening: dummy.go:5:11
.  .  .  .  .  Closing: dummy.go:5:12
.  .  .  .  }
.  .  .  }
.  .  .  Body: *ast.BlockStmt {
.  .  .  .  Lbrace: dummy.go:5:14
.  .  .  .  List: []ast.Stmt (len = 1) {
.  .  .  .  .  0: *ast.ExprStmt {
.  .  .  .  .  .  X: *ast.CallExpr {
.  .  .  .  .  .  .  Fun: *ast.SelectorExpr {
.  .  .  .  .  .  .  .  X: *ast.Ident {
.  .  .  .  .  .  .  .  .  NamePos: dummy.go:6:2
.  .  .  .  .  .  .  .  .  Name: "fmt"
.  .  .  .  .  .  .  .  }
.  .  .  .  .  .  .  .  Sel: *ast.Ident {
.  .  .  .  .  .  .  .  .  NamePos: dummy.go:6:6
.  .  .  .  .  .  .  .  .  Name: "Println"
.  .  .  .  .  .  .  .  }
.  .  .  .  .  .  .  }
.  .  .  .  .  .  .  Lparen: dummy.go:6:13
.  .  .  .  .  .  .  Args: []ast.Expr (len = 1) {
.  .  .  .  .  .  .  .  0: *ast.BasicLit {
.  .  .  .  .  .  .  .  .  ValuePos: dummy.go:6:14
.  .  .  .  .  .  .  .  .  Kind: STRING
.  .  .  .  .  .  .  .  .  Value: "\"Hello, World\""
.  .  .  .  .  .  .  .  }
.  .  .  .  .  .  .  }
.  .  .  .  .  .  .  Ellipsis: -
.  .  .  .  .  .  .  Rparen: dummy.go:6:28
.  .  .  .  .  .  }
.  .  .  .  .  }
.  .  .  .  }
.  .  .  .  Rbrace: dummy.go:7:1
.  .  .  }
.  .  }
.  }
.  Scope: *ast.Scope {
.  .  Objects: map[string]*ast.Object (len = 1) {
.  .  .  "greet": *(obj @ 27)
.  .  }
.  }
.  Imports: []*ast.ImportSpec (len = 1) {
.  .  0: *(obj @ 12)
.  }
.  Unresolved: []*ast.Ident (len = 1) {
.  .  0: *(obj @ 46)
.  }
}

How to walk

All we have to do is traverse this AST node in depth-first order.
Let's print each Node one by one by calling ast.Inspect() recursively.

Also, printing AST directly then we will typically see stuff that is not human readable.
To prevent that from happening, we're going to use ast.Print, a powerful API for human reading of AST:

package main

import (
    "fmt"
    "go/ast"
    "go/parser"
    "go/token"
)

func main() {
    fset := token.NewFileSet()
    f, _ := parser.ParseFile(fset, "dummy.go", src, parser.ParseComments)

    ast.Inspect(f, func(n ast.Node) bool {
        // Called recursively.
        ast.Print(fset, n)
        return true
    })
}

var src = `package hello

import "fmt"

func greet() {
    fmt.Println("Hello, World")
}
`

ast.File

The first Node to visit is *ast.File, which is the root of all AST nodes.
It implements only the ast.Node interface.

ast.File has references to a package name, import declarations, and function declarations as child nodes.
To be precise, it has also Comments and so on, but let me omit them for simplicity.
Let's start with Package Name.

(Note that fields with a nil value are omitted. See the document for a complete list of fields for each node type.)

Package Name

ast.Ident

*ast.Ident {
.  NamePos: dummy.go:1:9
.  Name: "hello"
}

A package name can be represented by the AST node type *ast.Ident, which implements the ast.Expr interface.
All identifiers are represented by this structure. It mainly contains its name and a source position within a file set.

From the code shown above, we can see that the package name is hello and is declared in the first line of dummy.go.

We can't dive any deeper into this node, let's go back to the *ast.File.

Import Declarations

ast.GenDecl

*ast.GenDecl {
.  TokPos: dummy.go:3:1
.  Tok: import
.  Lparen: -
.  Specs: []ast.Spec (len = 1) {
.  .  0: *ast.ImportSpec {/* Omission */}
.  }
.  Rparen: -
}

A declaration of import is represented by the AST node type *ast.GenDecl, which implements the ast.Decl interface.
ast.GenDecl represents all declarations except for functions; That is, import, const, var, and type.

Tok represents a lexical token — which is specifies what the declaration is about (IMPORT or CONST or TYPE or VAR).

This AST Node tells us that the import declaration is on line 3 in dummy.go.

Let's visit ast.GenDecl in depth-first order. Take a look *ast.ImportSpec, the next Node.

ast.ImportSpec

*ast.ImportSpec {
.  Path: *ast.BasicLit {/* Omission */}
.  EndPos: -
}

An ast.ImportSpec node corresponds to a single import declaration.
It implements the ast.Spec interface.
Visiting Path could make more sense about the import path. Let's go there.

ast.BasicLit

*ast.BasicLit {
.  ValuePos: dummy.go:3:8
.  Kind: STRING
.  Value: "\"fmt\""
}

An ast.BasicLit node represents a literal of basic type.
It implements the ast.Expr interface.
This contains a type of token and token.INT, token.FLOAT, token.IMAG, token.CHAR, or token.STRING can be used.

From ast.ImportSpec and ast.BasicLit, we can see it has imported package called "fmt".

We can't dive any deeper, let's get back to the top level again.

Function Declarations

ast.FuncDecl

*ast.FuncDecl {
.  Name: *ast.Ident {/* Omission */}
.  Type: *ast.FuncType {/* Omission */}
.  Body: *ast.BlockStmt {/* Omission */}
}

An ast.FuncDecl node represents a function declaration.
It implements only the ast.Node interface.
Let's take a look at them in order from Name, representing a function name.

ast.Ident

*ast.Ident {
.  NamePos: dummy.go:5:6
.  Name: "greet"
.  Obj: *ast.Object {
.  .  Kind: func
.  .  Name: "greet"
.  .  Decl: *(obj @ 0)
.  }
}

The second time this has appeared, let me skip the basic explanation.

Noteworthy is the *ast.Object.
It represents the object to which the identifier refers, but why is this needed?
As you know, Go has a concept of scope, which is the extent of source text in which the identifier denotes the specified constant, type, variable, function, label, or package.

The Decl field indicates where the identifier was declared so that it identifies the scope of the identifier.
Identifiers that point to the identical object share the identical *ast.Object.

ast.FuncType

*ast.FuncType {
.  Func: dummy.go:5:1
.  Params: *ast.FieldList {/* Omission */}
}

Go back to being a parent one generation older, an ast.FuncType contains a function signature including parameters, results, and position of "func" keyword.

ast.FieldList

*ast.FieldList {
.  Opening: dummy.go:5:11
.  List: nil
.  Closing: dummy.go:5:12
}

An ast.FieldList node represents a list of Fields, enclosed by parentheses or braces.
Function parameters would be shown here if they are defined, but this time none, so no information.

List field is a slice of *ast.Field that contains a pair of identifiers and types.
It is highly versatile and is used for a variety of Nodes, including *ast.StructType, *ast.InterfaceType, and here.
That is, it's needed when mapping a type to an identifier as shown below:

foo int
bar string

Let's loop back to *ast.FuncDecl again and dive a bit into Body, the last field.

ast.BlockStmt

*ast.BlockStmt {
.  Lbrace: dummy.go:5:14
.  List: []ast.Stmt (len = 1) {
.  .  0: *ast.ExprStmt {/* Omission */}
.  }
.  Rbrace: dummy.go:7:1
}

An ast.BlockStmt node represents a braced statement list.
It implements the ast.Stmt interface.
It does have a list of statements. What an imaginable node!

ast.ExprStmt

*ast.ExprStmt {
.  X: *ast.CallExpr {/* Omission */}
}

ast.ExprStmt represents an expression in a statement list.
It implements the ast.Stmt interface and contains a single ast.Expr.

ast.CallExpr

*ast.CallExpr {
.  Fun: *ast.SelectorExpr {/* Omission */}
.  Lparen: dummy.go:6:13
.  Args: []ast.Expr (len = 1) {
.  .  0: *ast.BasicLit {/* Omission */}
.  }
.  Ellipsis: -
.  Rparen: dummy.go:6:28
}

ast.CallExpr represents an expression that calls a function.
The fields to look at are Fun, the function to call and Args, the list of arguments to pass to it.

ast.SelectorExpr

*ast.SelectorExpr {
.  X: *ast.Ident {
.  .  NamePos: dummy.go:6:2
.  .  Name: "fmt"
.  }
.  Sel: *ast.Ident {
.  .  NamePos: dummy.go:6:6
.  .  Name: "Println"
.  }
}

ast.SelectorExpr represents an expression followed by a selector.
Simply put, it means fmt.Println.

ast.BasicLit

*ast.BasicLit {
.  ValuePos: dummy.go:6:14
.  Kind: STRING
.  Value: "\"Hello, World\""
}

No longer needed an explanation, Hello, World!

Bottom Line

I've left out some of the fields in the node types I've introduced, and there are still many other node types.

Nevertheless, I'd say it's significant to actually walk the walk even if it's a bit rough. And above all, it’s quite a bit of fun.
Copy and paste the code shown in the section "How to walk", and have a try to walk around on your PC.

Digging deeper into the analysis of Go-code

Ryo Nakao — Fri, 25 Sep 2020 23:57:30 +0000

This article is originally posted here.

The analysis of source code at the syntactic level can help you with your coding in a variety of ways. For that, the text is almost always converted to AST first to make it easier to handle in most languages.

As some of you may know, Go has a powerful package go/parser, with it, you can convert source code to AST relatively easily.
However, I couldn't help but be curious about how it is working, and I realized my mind could only be satisfied by getting started to read the API implementation.
In this article, I will walk you through how it is converted, by reading the implementation of its API.

Even those unfamiliar with the Go language don't need to close the tab in the browsers, as this is a generic enough article to understand how programming languages are analyzed.

This article is also the first step in understanding the compiler and interpreter, as well as delving into static analysis.

AST

Let's start with some of the knowledge you need to read the implementation.
What is AST(Abstract Syntax Tree)? According to Wikipedia:

In computer science, an abstract syntax tree (AST), or just syntax tree, is a tree representation of the abstract syntactic structure of source code written in a programming language. Each node of the tree denotes a construct occurring in the source code.

Most compilers and interpreters use AST as an internal representation of the source code; AST typically omits semicolons, line feed characters, white spaces, braces, square brackets, and round brackets from the syntax tree, etc.

What you can do with AST:

Source code analysis
Code generation
Can be rewritten

How to convert to AST

Plain text is quite straightforward for us, but from a machine, nothing is tougher to handle. Therefore, you have to first do lexical analysis the text with a lexer. The general flow is to pass it to a parser and retrieve the AST.

I'd prefer to point out here that there is not a single universal AST format which can be used by any parser.
For instance, x + 2 in Go is represented by:

*ast.BinaryExpr {
.  X: *ast.Ident {
.  .  NamePos: 1
.  .  Name: "x"
.  .  Obj: *ast.Object {
.  .  .  Kind: bad
.  .  .  Name: ""
.  .  }
.  }
.  OpPos: 3
.  Op: +
.  Y: *ast.BasicLit {
.  .  ValuePos: 5
.  .  Kind: INT
.  .  Value: "2"
.  }
}

Lexical analysis

As mentioned earlier, the analysis typically starts by passing the text to a lexer and then fetching the tokens. A token is a string with an assigned and thus identified meaning.
go/scanner.Scanner is in charge of the lexer in Go.

What is the identified meaning? Seeing is believing.

Let's say you write:

package main

const s = "foo"

This is what happens when you tokenize it:

PACKAGE(package)
IDENT(main)
CONST(const)
IDENT(s)
ASSIGN(=)
STRING("foo")
EOF()

All tokens in Go are defined here.

Peeling away at the parsing API

To convert a Go source file to AST, just call go/parser.ParseFile as shown below:

fset := token.NewFileSet()
f, _ := parser.ParseFile(fset, "foo.go", nil, parser.ParseComments)

Now that we've figured out the conversion steps by the previous chapter, let's actually read the internal implementation of that method! (The version of Go we refer to is 1.14.1).

Scanner.Scan() — a method for Lexical Analysis

How does Go perform lexical analysis? As previously mentioned, go/scanner.Scanner is in charge of the lexer in Go.
Thus at first, let's take a closer look at that Scanner.Scan() method — which is called by parser.ParseFile() internally.

scanner/scanner.go

func (s *Scanner) Scan() (pos token.Pos, tok token.Token, lit string) {
... // Omission
    switch ch := s.ch; {
    case isLetter(ch):
        lit = s.scanIdentifier()
        if len(lit) > 1 {
            // keywords are longer than one letter - avoid lookup otherwise
            tok = token.Lookup(lit)
            switch tok {
            case token.IDENT, token.BREAK, token.CONTINUE, token.FALLTHROUGH, token.RETURN:
                insertSemi = true
            }
        } else {
... // Omission
}

ch is the current character held by Scanner. Scanner.Scan() advances to next character by calling Scanner.next() and populates ch, as long as it is available as an identifier name.
The code above is for the case where ch is a letter; It pauses its advance as soon as it encounters a character that cannot be used as an identifier and then determines the type of token.

There are different ways to determine where does a single token start and where does it end, depending on the character. For instance, in the case of String, it continues to advance until " appears:

scanner/scanner.go

case '"':
    insertSemi = true
    tok = token.STRING
    lit = s.scanString()


func (s *Scanner) scanString() string {
    // '"' opening already consumed
    offs := s.offset - 1

    for {
        ch := s.ch
        if ch == '\n' || ch < 0 {
            s.error(offs, "string literal not terminated")
            break
        }
        s.next()
        if ch == '"' {
            break
        }
        if ch == '\\' {
            s.scanEscape('"')
        }
    }

    return string(s.src[offs:s.offset])
}

Finally, the Scanner.Scan() method returns a token that has been identified.

Parsing

Before taking a look parsing a file, let's check the file structure in Go.
According to The Go Programming Language Specification - Source file organization:

Each source file consists of a package clause defining the package to which it belongs, followed by a possibly empty set of import declarations that declare packages whose contents it wishes to use, followed by a possibly empty set of declarations of functions, types, variables, and constants.

That is, the structure is:

A package clause
Import declarations
Top level declarations

After parsing a package clause and import declarations, parser.parseFile() repeats the parsing of the declaration to the end of the file.

parser/parser.go

for p.tok != token.EOF {
    decls = append(decls, p.parseDecl(declStart))
}

So let's look at parser.parseDecl next.

parser.parseDecl() — a method to parse the syntax of a declaration

parser.parseDecl() returns ast.Decl, the node of the syntax tree representing the declaration in the Go source code.

parser/parser.go

func (p *parser) parseDecl(sync map[token.Token]bool) ast.Decl {
    if p.trace {
        defer un(trace(p, "Declaration"))
    }

    var f parseSpecFunction
    switch p.tok {
    case token.CONST, token.VAR:
        f = p.parseValueSpec

    case token.TYPE:
        f = p.parseTypeSpec

    case token.FUNC:
        return p.parseFuncDecl()

    default:
        pos := p.pos
        p.errorExpected(pos, "declaration")
        p.advance(sync)
        return &ast.BadDecl{From: pos, To: p.pos}
    }

    return p.parseGenDecl(p.tok, f)
}

It goes through the tokens and process them differently for each keyword. Let's deep dive into parseFuncDecl().

parser/parser.go

if p.tok == token.LPAREN {
    recv = p.parseParameters(scope, false)
}

ident := p.parseIdent()

params, results := p.parseSignature(scope)

var body *ast.BlockStmt
if p.tok == token.LBRACE {
    body = p.parseBody(scope)
    p.expectSemi()
} else if p.tok == token.SEMICOLON {
    p.next()

Internally, it advances the token by calling Scanner.Scan() — which we saw in detail earlier.

token.LPAREN represents (, so you can see that it starts parsing the parameters as soon as ( is found.

token.LBRACE represents {, so you can see that it starts parsing the function body as soon as { is found.

.

.

.

Oops, it's going to take forever at this rate...

Summary

Parsing the tokens by myself has made me feel closer to the compiler and interpreter that I used to feel horrible about.
I'd love to dabble in Writing A Compiler In Go and Writing An Interpreter In Go as well.