DEV Community: Masayoshi Mizutani

Redacting Sensitive Data in Go's slog: A Practical Guide with masq

Masayoshi Mizutani — Sun, 18 Jan 2026 00:24:41 +0000

Logging is essential for debugging, monitoring, and auditing. But here's the catch—logs can accidentally expose sensitive data like passwords, API tokens, or personal information. Once sensitive data lands in your logs, removing it becomes a nightmare, especially when logs are immutable by design for compliance reasons.

In this post, I'll show you how to automatically redact sensitive values from your structured logs using Go's standard log/slog package and my open-source library masq.

The Problem: Sensitive Data Leaks in Logs

Consider a typical scenario: you're logging a user struct for debugging purposes.

type User struct {
    ID       string
    Email    string
    APIToken string
}

func main() {
    user := User{
        ID:       "u123",
        Email:    "alice@example.com",
        APIToken: "sk-secret-token-12345",
    }
    slog.Info("user logged in", "user", user)
}

Output:

level=INFO msg="user logged in" user="{ID:u123 Email:alice@example.com APIToken:sk-secret-token-12345}"

Oops. The API token is now in your logs, potentially accessible to anyone with log access, stored for months or years, and nearly impossible to delete.

slog's Built-in Solution: LogValuer

Go's slog package (part of the standard library since Go 1.21) provides LogValuer interface for customizing how values appear in logs:

type APIToken string

func (APIToken) LogValue() slog.Value {
    return slog.StringValue("[REDACTED]")
}

func main() {
    token := APIToken("sk-secret-token-12345")
    slog.Info("token received", "token", token)
}

Output:

level=INFO msg="token received" token=[REDACTED]

This works for direct values. However, LogValuer doesn't work for struct fields:

type APIToken string

func (APIToken) LogValue() slog.Value {
    return slog.StringValue("[REDACTED]")
}

type Credentials struct {
    UserID string
    Token  APIToken
}

func main() {
    creds := Credentials{
        UserID: "u123",
        Token:  "sk-secret-token-12345",
    }
    slog.Info("credentials", "creds", creds)
}

Output (token is exposed!):

level=INFO msg=credentials creds="{UserID:u123 Token:sk-secret-token-12345}"

When you log a struct, slog uses reflection and bypasses the LogValue() method on nested fields. This is a significant limitation for real-world applications.

Enter masq: Automatic Deep Redaction

I built masq to solve this problem. It hooks into slog's ReplaceAttr option and recursively inspects all logged values—including nested structs—to redact sensitive data.

Basic Usage

package main

import (
    "log/slog"
    "os"

    "github.com/m-mizutani/masq"
)

type EmailAddr string

type User struct {
    ID    string
    Email EmailAddr
}

func main() {
    // Create a logger with masq redaction
    logger := slog.New(
        slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
            ReplaceAttr: masq.New(masq.WithType[EmailAddr]()),
        }),
    )

    user := User{
        ID:    "u123",
        Email: "alice@example.com",
    }

    logger.Info("user registered", "user", user)
}

Output:

{
  "time": "2026-01-18T12:00:00.000Z",
  "level": "INFO",
  "msg": "user registered",
  "user": {
    "ID": "u123",
    "Email": "[FILTERED]"
  }
}

The email is automatically redacted, even though it's nested inside the struct.

Redaction Strategies

masq offers multiple ways to identify sensitive data:

1. By Custom Type

Define sensitive data as distinct types and redact them:

type Password string
type CreditCard string

masq.New(
    masq.WithType[Password](),
    masq.WithType[CreditCard](),
)

2. By Struct Tag

Mark sensitive fields with a struct tag:

type User struct {
    ID       string
    Password string `masq:"secret"`
    SSN      string `masq:"secret"`
}

masq.New(masq.WithTag("secret"))

3. By Field Name

Target specific field names:

masq.New(
    masq.WithFieldName("Password"),
    masq.WithFieldName("APIKey"),
)

4. By Field Prefix

Redact all fields starting with a prefix:

type Config struct {
    SecretKey      string  // redacted
    SecretToken    string  // redacted
    PublicEndpoint string  // not redacted
}

masq.New(masq.WithFieldPrefix("Secret"))

5. By Regex Pattern

Match values against patterns (useful for credit cards, phone numbers, etc.):

import "regexp"

// Redact potential credit card numbers (16 digits)
cardPattern := regexp.MustCompile(`\b\d{16}\b`)

masq.New(masq.WithRegex(cardPattern))

6. By String Content

Redact any value containing a specific string:

masq.New(masq.WithContain("Bearer "))

Combining Multiple Strategies

You can combine multiple redaction rules:

logger := slog.New(
    slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
        ReplaceAttr: masq.New(
            // Redact by type
            masq.WithType[Password](),
            masq.WithType[APIToken](),

            // Redact by struct tag
            masq.WithTag("secret"),

            // Redact fields with "Secret" prefix
            masq.WithFieldPrefix("Secret"),

            // Redact phone numbers
            masq.WithRegex(regexp.MustCompile(`^\+[1-9]\d{10,14}$`)),
        ),
    }),
)

Real-World Example

Here's a complete example showing masq in a typical web application context:

package main

import (
    "log/slog"
    "os"
    "regexp"

    "github.com/m-mizutani/masq"
)

type (
    Password  string
    AuthToken string
)

type LoginRequest struct {
    Username string
    Password Password
}

type UserSession struct {
    UserID      string
    AuthToken   AuthToken
    Email       string `masq:"pii"`
    PhoneNumber string
}

func main() {
    // Configure comprehensive redaction
    logger := slog.New(
        slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
            ReplaceAttr: masq.New(
                masq.WithType[Password](),
                masq.WithType[AuthToken](),
                masq.WithTag("pii"),
                masq.WithRegex(regexp.MustCompile(`^\+[1-9]\d{10,14}$`)), // phone numbers
            ),
        }),
    )
    slog.SetDefault(logger)

    // Simulate login flow
    req := LoginRequest{
        Username: "alice",
        Password: "super-secret-password",
    }
    slog.Info("login attempt", "request", req)

    session := UserSession{
        UserID:      "u123",
        AuthToken:   "tok_abc123xyz",
        Email:       "alice@example.com",
        PhoneNumber: "+14155551234",
    }
    slog.Info("session created", "session", session)
}

Output:

{"time":"2026-01-18T12:00:00.000Z","level":"INFO","msg":"login attempt","request":{"Username":"alice","Password":"[FILTERED]"}}
{"time":"2026-01-18T12:00:00.000Z","level":"INFO","msg":"session created","session":{"UserID":"u123","AuthToken":"[FILTERED]","Email":"[FILTERED]","PhoneNumber":"[FILTERED]"}}

All sensitive data is automatically redacted while preserving the structure and non-sensitive fields.

Best Practices

Define custom types for sensitive data: Instead of using string for passwords or tokens, create distinct types like type Password string. This makes redaction explicit and catches issues at compile time.
Use struct tags for external data: When working with third-party structs or database models, use the masq:"secret" tag to mark sensitive fields.
Apply regex patterns carefully: Regex matching runs on every string value, so use specific patterns to avoid performance issues.
Test your redaction: Write tests that verify sensitive data doesn't appear in log output.
Default to redaction: When in doubt, redact. It's easier to remove redaction than to clean up leaked data.

Limitations

Private map fields: masq cannot reliably clone embedded private map types—they become nil. Use struct fields for sensitive data.
Performance: Deep inspection has some overhead. For high-throughput systems, consider sampling or async logging.

Conclusion

Preventing sensitive data leaks in logs is crucial for security and compliance. While slog's LogValuer helps with direct values, masq extends this protection to nested structs and provides flexible redaction strategies.

Give masq a try and let me know what you think!

If you found this helpful, feel free to star the repo on GitHub or share your feedback in the comments below.

Eliminating Sensitive Values from Logs using Slog, the Prospective Official Structured Logger for Go

Masayoshi Mizutani — Sun, 09 Jul 2023 23:37:49 +0000

In the Go language, the log package has been used for official log output for a long time. However, in recent cloud environments and the like, structured logging is almost mandatory, and in response to such trends, the official structured log package slog has been proposed. It is already listed in the release notes for Go 1.21, which is expected to be released in August 2023, and it is almost certain that it will officially be incorporated in 1.21.

Background: Absolutely do not want to output secret values to the log

Logs output by online services are used for various purposes. For example, logs are aggregated to generate alerts for service operation monitoring, to analyze for service improvement, to detect unauthorized access for service security monitoring, and so on. Also, logs are often used for auditing, and various log-related mechanisms are built on the premise that logs cannot be deleted, at least during the storage period.

The more information is output to the log, the wider its use. It's especially advantageous to have more clues in debugging and troubleshooting. However, the data handled by the service may contain information that should not be carelessly output, such as authentication information like passwords and tokens, personally identifiable information like names, phone numbers, addresses, and bank account information and credit card numbers. In this article, we will conveniently call these "secret values".

As mentioned earlier, logs are often stored for a certain period of time and cannot be deleted during that period. Therefore, if secret information is output to the log, it will require a lot of effort to delete it. You may need to take measures such as limiting the number of people who can access it, but this reduces the situations in which it can be used and makes it impossible to utilize the log.

It's perfectly reasonable to say, "You shouldn't output secret values to the log in the first place," but consider the case where a struct is output to the log, and the fields of that struct, or even nested structs, may contain secret values. Even if you are careful in selecting the values to output to the log, the struct itself may be modified later to add fields that contain secret values. It's very difficult to cover all such cases. On the other hand, considering the requirement to leave as much information as possible in the log, it's not a very realistic approach to say, "Don't output structs to the log."

`slog.LogValuer`

slog provides an interface called LogValuer, and by implementing this, you can customize the value to output to the log. This can be used to hide or mask values, for example.

type Token string

func (Token) LogValue() slog.Value {
    return slog.StringValue("REDACTED_TOKEN")
}

func main() {
    t := Token("ThisIsSecretToken")
    slog.Info("permission granted", "token", t)
}

When this code is executed, the value ThisIsSecretToken is not output, and is replaced with REDACTED_TOKEN.

time=2009-11-10T23:00:00.000Z level=INFO msg=Access token=REDACTED_TOKEN

This mechanism seems like a good solution to the problem mentioned earlier, but it doesn't apply to fields contained in structs. For example, LogValuer does not work in the following example.

type Token string

func (Token) LogValue() slog.Value {
    return slog.StringValue("REDACTED_TOKEN")
}

type AccessLog struct {
    User  string
    Token Token
}

func main() {
    l := AccessLog{
        User:  "mizutani",
        Token: "ThisIsSecretToken",
    }
    slog.Info("Access", "log", l)
}

As in the following example, the value you wanted to hide is output as is.

time=2009-11-10T23:00:00.000Z level=INFO msg=Access log="{User:mizutani Token:ThisIsSecretToken}"

Therefore, a different approach is needed to more precisely hide secret values.

Using `ReplaceAttr`

TextHandler and JSONHandler provided by slog have an option called ReplaceAttr. This calls a callback before outputting a value passed as an attribute value, allowing you to replace the value.

type Token string

type AccessLog struct {
    User  string
    Token Token
}

func redact(_ []string, a slog.Attr) slog.Attr {
    l, ok := a.Value.Any().(AccessLog)
    if !ok {
        return a
    }
    return slog.Any(a.Key, AccessLog{
        User:  l.User,
        Token: "REDACTED_TOKEN",
    })
}

func main() {
    l := AccessLog{
        User:  "mizutani",
        Token: "ThisIsSecretToken",
    }

    logger := slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{ReplaceAttr: redact}))
    logger.Info("Access", "log", l)
}

By writing it this way, you can hide secret values.

time=2009-11-10T23:00:00.000Z level=INFO msg=Access log="{User:mizutani Token:REDACTED_TOKEN}"

However, it's not practical at all to write such a judgment logic. Therefore, we need a mechanism that can hide in a more general way.

masq

After a long introduction, let me introduce a package to redact secret values, m-mizutani/masq .

https://github.com/m-mizutani/masq

masq is a package for generating callbacks for ReplaceAttr. For example, if you want to hide a value of type EmailAddr, you write as follows.

u := struct {
    ID    string
    Email EmailAddr
}{
    ID:    "u123",
    Email: "mizutani@hey.com",
}

logger := slog.New(slog.HandlerOptions{
    ReplaceAttr: masq.New(masq.WithType[EmailAddr]()),
}.NewJSONHandler(os.Stdout))

logger.Info("hello", slog.Any("user", u))

This will output as follows (formatted with the jq command).

{
  "time": "2022-12-25T09:00:00.123456789",
  "level": "INFO",
  "msg": "hello",
  "user": {
    "ID": "u123",
    "Email": "[REDACTED]" // ← hidden
  }
}

In this way, you can hide values that match the type, even if they are in the fields of a struct.

masq.New has various options available, and generates a callback for ReplaceAttr based on the specified options. The following options are available:

WithType[T](): Hides values that match type T.
WithString(s string): Hides values that match string s.
WithRegex(re regexp.Regex): Hides values that match regexp re.
WithTag(tag string): Hides values that match the masq field tag of a struct. For example, if you specify secret, it will hide fields tagged with masq:"secret".
WithFieldName(name string): Hides values that match the field name name of a struct.
WithFieldPrefix(prefix string): Hides values that start with the field name prefix of a struct.

These can be combined in multiple ways, as shown below.

logger := slog.New(slog.HandlerOptions{
    ReplaceAttr: masq.New(
        // Hides the type AccessToken
        masq.WithType[AccessToken](),

        // Hides strings that start with 14 to 16 digits
        masq.WithRegex(regexp.MustCompile(`^\+[1-9]\d{14,16}$`)),

        // Hides fields tagged with masq:"secret"
        masq.WithTag("secret"),

        // Hides fields whose names start with Secret
        masq.WithFieldPrefix("Secret"),
    ),
}.NewJSONHandler(out))

For detailed usage, please refer to the README.

Summary

In fact, this mechanism has been provided in the zlog package for some time. However, since slog has finally reached a practical stage, we implemented masq as a new package to match slog.

As mentioned earlier, slog also has a mechanism called LogValuer, so using that is also an option. I think the best choice is to make the appropriate choice according to your use case, and I would be happy if you could include masq as one of your choices.

Goast: Generic static analysis for Go Abstract Syntax Tree by OPA/Rego

Masayoshi Mizutani — Tue, 13 Sep 2022 02:24:37 +0000

TL; DR

Go language has various static analysis tools, but to check your own rules, you need to create your own tools each time.
To enable more general checks with a single tool, I created a tool to analyze the Go language AST (Abstract Syntax Tree) with Rego, a general-purpose policy language.

https://github.com/m-mizutani/goast

The rule "always take context.Context as the first argument" checked by CI.

Motivation

Various static analysis tools are available for the Go language, and existing static analysis tools can check general best practices. For example, gosec is a tool to check secure Go coding, and I use it myself. However, coding rules in software development are not only based on best practices, but can also be software- or team-specific. For example

All functions must take context.Context as an argument in some package
All functions must call an audit logging function at least once in some package
The User structure must be initialized by the function NewUser().
Some function must be called only in a specific package.

These rules can be checked by humans during review, but it is difficult to rely on humans alone because humans are always capable of making mistakes. Also, even if the review checking works when there are only a few rules, the more rules there are, the more likely they are to be missed inadvertently due to distraction. It also makes it difficult to focus on the essential reviews.

The Go language provides official tools and frameworks for static analysis, making it easy to create your own tools. On the other hand, however, creating an analysis tool for each rule seems to be a bit difficult from the standpoint of implementation and maintenance costs. Therefore, I thought it would be better to create a tool that can check static code as universally as possible.

Separate "rule" and "implementation" by Rego

Although not only for static analysis, one of the key points in creating a versatile checking tool is how to let users describe rules. It is too costly to create an original description language, and if the rules are given in structural data such as YAML or JSON, the expressive power will be limited and the versatility will be reduced.

A useful tool for such applications is the policy description language Rego. Rego is a general-purpose language that can be used to evaluate structured data by OPA. Some of the most popular uses include checking the status of resources used in cloud environments, checking the content of Infrastructure as Code descriptions, and checking authorization for access to servers. Please see this document for more detail of Rego.

By using Rego, the checking implementation and rules can be completely separated. The implementation is responsible for reading files, reading policies, passing data for evaluation, and outputting evaluation results, while the rules are written only in Rego. This allows a separation of interest between those who implement the tool and those who think about the rules.

Implementation

Then, I implemented generic static analysis tool for Go language, goast.

https://github.com/m-mizutani/goast

The tool reads Go code and evaluates the AST (Abstruct Syntax Tree, syntax abstract tree), an abstract representation of the code, by a policy written in Rego. The parser package is used to get the AST of the Go source code, which is then evaluated by Rego's policy. The evaluation can pass the AST of the whole file only once, or provide a mode to evaluate it node by node of the AST.

Let's look AST of Go code

I am also a beginner in AST in Go, so I can't imagine AST at all just by looking at the code. Therefore, I added a function to goast to dump AST for confirmation.

package main

import "fmt"

func main() {
        fmt.Println("hello")
}

goast can output AST dump by a following command.

$ goast dump --line 6  examples/println/main.go | jq
{
  "Path": "examples/println/main.go",
  "Node": {
    "X": {
      "Fun": {
        "X": {
          "NamePos": 44,
          "Name": "fmt",
          "Obj": null
        },
        "Sel": {
          "NamePos": 48,
          "Name": "Println",
          "Obj": null
        }
      },
      "Lparen": 55,
      "Args": [
        {
          "ValuePos": 56,
          "Kind": 9,
          "Value": "\"hello\""
        }
      ],
      "Ellipsis": 0,
      "Rparen": 63
    }
  },
  "Kind": "ExprStmt"
}

AST structural data tends to be relatively large, and even the 7 lines of code described above would be 1,408 characters of JSON data. Therefore, for ease of reading, only the sixth line of the code (fmt.Println("hello")) is output. Path is the path of the read file, Node is a dump of ast.Node passed by ast.Inspect, and Kind is the type information of Node. s type information.

As you can imagine, here .Node.X.Fun represents information about the calling function, and .Node.X.Args represents the arguments. For example, you could use this to describe a rule such as "prohibit the invocation of a particular function". You can also use following contexts as additional condition for example.

Allow/Prohibit calls within a specific package
Allow/Prohibit certain arguments
Allow/Prohibit direct passing of literals

Describe a rule

Now let's write Rego rules from the output AST. This time, let's describe a simple rule to forbid calling fmt.Println.

package goast

fail[res] {
    input.Kind == "ExprStmt"
    input.Node.X.Fun.X.Name == "fmt"
    input.Node.X.Fun.Sel.Name == "Println"

    res := {
        "msg": "do not use fmt.Println",
        "pos": input.Node.X.Fun.X.NamePos,
        "sev": "ERROR",
    }
}

goast's rule schema is following.

package must be goast
Input: input has metadata such asPath、Kind and Node as actual AST.
Output: Put following structure data into fail if violation detected
- msg (string): Detail message of violation
- pos (int): Number to indicate position in the source code file
- sev (string): Severity, choose one from INFO, WARNING, and ERROR

First, the three lines at the beginning of the rule detect the fmt.Println that was just dumped, and since the message in the format just dumped is passed directly to Rego as input, inspection of Kind, Node.X.Fun.X.Name and Node.X.Fun.Sel.Name Name to determine that it is an expression of a function call.

If you are not familiar with ASTs, it may be difficult to understand what pos means, but in this case it is a number that indicates the number of bytes from the beginning of the file and is stored in some fields such as NamePos and ValuePos. By putting the number in a response, goast will convert it to the number of lines in the file where the violation occurred, and the final output will indicate the number of lines.

You can detect violations by saving the Go code as main.go and the rule as policy.rego and running

$ goast eval -p policy.rego main.go
[main.go:6] - do not use fmt.Println

        Detected 1 violations

Also, goast supports JSON format output.

$ goast eval -f json -p policy.rego main.go
{
  "diagnostics": [
    {
      "message": "do not use fmt.Println",
      "location": {
        "path": "main.go",
        "range": {
          "start": {
            "line": 6,
            "column": 2
          }
        }
      }
    }
  ],
  "source": {
    "name": "goast",
    "url": "https://github.com/m-mizutani/goast"
  }
}

Static analysis in CI

Static analysis should be performed continuously by CI (Continuous Integration) to prevent unintentional inclusion of code. The JSON output schema is compatible with reviewdog and can be used as is in reviewdog.

We also have goast-action available for use with GitHub Actions, which allows you to perform static inspection on Pull Requests with the following workflow.

name: goast

on:
  pull_request:

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - name: checkout
        uses: actions/checkout@v2
      - uses: reviewdog/action-setup@v1
      - name: goast
        uses: m-mizutani/goast-action@main
        with:
          policy: ./policy  # Directory of rule files written in Rego
          format: json      # Output format, "text" or "json"
          output: fail.json # File name for output
          source: ./pkg     # Directory of Go source code to be checked
      - name: report
        env:
          REVIEWDOG_GITHUB_API_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: cat fail.json | reviewdog -reporter=github-pr-review -f rdjson

By the workflow, a comment such as the following image will be submitted by GitHub Actions.

Conclusion

I believe that static analysis with Go AST and Rego allows developers more flexible static analysis for Go language. It would be helpful to develop more secure software.

However, I thought it's not possible to cover all static inspections by AST that shows the entire source code and a general-purpose policy language. For example, Rego is not good at writing rules that track changes in state, so it is not very suitable for use cases such as "how a variable is referenced or changed".

I have just started using it myself in practice, so I am still in the process of exploring various ways to use it. I welcome feature suggestions and discussion, so please feel free to comment or create an issue in the repository.

Control notification of every GitHub event with OPA/Rego

Masayoshi Mizutani — Sun, 13 Mar 2022 01:11:27 +0000

TL;DR

GitHub already provides official notifications and comment-based notification tools, but there are more notification use cases
OPA/Rego allows for good separation of notification rules and implementation
I implemented a tool as a PoC that can notify me of all GitHub events.

https://github.com/m-mizutani/ghnotify

GitHub notification

There are many ways to be notified of events on GitHub, not only official email notifications and [Slack integration], but various tools make this possible. In particular, tokite inspects strings in Issues and Pull Requests (PRs) and notifies Slack if there are comments containing specific keywords. Specific use cases are following:

Find comments that are not directly mentions by ID but have your name on them.
Find internal Issues in which keywords related to the topic you are interested in (in your case, for example, "security," "personal information," "authentication," etc.) appear.

GitHub also provides notifications of various events via webhook, and in addition to comments, it may be possible to streamline development and increase transparency in GitHub operations by receiving notifications for the following events.

Results of GitHub Actions (notifications when a particular repository or a particular step succeeds or fails)
Issue new Deploy key
Push to a specific branch
Granting or deleting specific labels
Creating and archiving repositories
Team changes
Installing or uninstalling GitHub App

Problem about "notification rule"

It would be nice to leverage the notifications provided by GitHub, but there is a challenge in realizing this: "Different organizations and individuals want to receive different notifications. As mentioned above, there are also "specific" behaviors for "specific" resources. This is determined by organizational or individual policies, so creating a tool that provides useful notifications for one organization will be full of noise for another. It is logically possible to have different implementations for different organizations or individuals, but it doesn't seem to make much sense.

In order to do notifications to each organizations and individuals, tokite allows rules to be written that specify the body of comments and issue attributes. This can be done by

Focusing to issues and PRs, and
Focusing to the attributes (body, label, title, etc.) that determine whether you want to be notified and how to check them.

Creating a system of rules and implementing them requires a certain amount of effort, so by focusing on items that are likely to be commonly used, the return on investment is increased. But of course, it is very possible that the method of checking does not fit or that the attribute values you want are not included. If the cost of implementation is extremely low, it would be preferable to address all events.

Separating of Policy and Implementation

Open Policy Agent (OPA) and Rego are available as one solution to this challenge. To give a very rough explanation, Rego is a language that provides only the function of "inputting structural data (such as JSON) and outputting new structural data," and OPA is the execution engine for this function. Usual use cases are to "input request contents and output the allow/deny" or "input cloud configuration information and output whether it complies with the organization's policy", but OPA/Rego is highly versatile and can be used not only for information security/compliance but also another systematic rule engine.

In this case, by entrusting OPA with the function of "inputting GitHub event data and outputting notification contents". It is possible to separate implementation and rules (policies). The figure below illustrates this workflow.

OPA's policy determination process can be done by 1) processing imported Rego file with runtime in Go or 2) inquiry to OPA server. It can be chosen by organization or individual usage.

OPA can handle any form of structural data as input. Therefore, even if there are multiple schemas for each event like GitHub, there is no need to implement separate processing for each schema, and it is possible to "just throw it in as input". The implementation receiving the notifications can then give up concern about the schema.

The same is true for notifications: the only instruction returned from OPA is "send a notification with this text or item". Therefore, the notification part is also almost schema-neutral, just format the message and hit the API.

So,

No need to implement dependencies on GitHub notification event types, so all events can be handled without additional implementation costs
Separation of implementation and policy ensures that changes to notification rules do not affect implementation

ghnotify

So, I implemented a tool to filter notification of GitHub event by OPA/Rego.

m-mizutani / ghnotify

General GitHub event notification tool to Slack with Open Policy Agent and Rego

ghnotify

ghnotify is a general GitHub event notification tool to Slack with Open Policy Agent and Rego. There are a lot of notification tools from GitHub to Slack. However, in most case, notification rules are deeply integrated with source code implementation and customization of notification rule by end-user is limited.

ghnotify uses generic policy language Rego and OPA as runtime to separate implementation and policy completely. Therefore, ghnotify can handle and notify all type of GitHub event, not only issue/PR comments but also such as following events according to your Rego policy.

GitHub Actions success/failure
deploy key creation
push to specified branch
add/remove a label
repository creation, archived, transferred
team modification
a new GitHub App installation

Setup

1) Retrieve Bot User OAuth Token of Slack

Create your Slack bot and keep OAuth Tokens for Your Workspace in OAuth & Permissions page.

2) Creating Rego policy

ghnotify evaluates received…

View on GitHub

The specific usage is detailed in the README, but for example, the following rules could be written.

notify[msg] {
    input.name == "issue_comment"
    contains(input.event.comment.body, "mizutani")
    msg := {
        "channel": "#notify-mizutani",
        "text": "Hello, mizutani",
        "body": input.event.comment.body,
    }
}

If the word mizutani is included in the comments of an issue in the monitored repository, the following notification will be sent via Slack.

There are 3 usages for the tool.

Case 1) Run ghnotify as a web service and receive webhooks
- Case 1.1) Query an OPA server running separately
- Case 1.2) Read the policy file included with the ghnotify web service
Case 2) Catching events on GitHub Actions

Case 1 requires setting up a GitHub App or Webhook and deploying ghnotify as a web service, which is more time-consuming, but it is useful when you want to monitor many repositories because you can receive notifications for the entire Organization. Case 2, on the other hand, is easier to set up, but only allows you to monitor one repository. The author uses the following configuration.

Policy examples

What other specific policies can we write? I would like to briefly introduce the following.

Specific labels were assigned to the PR

notify[msg] {
    input.name == "pull_requests"
    input.event.action == "labeled"
    input.event.label.name == "breaking-change"

    msg := {
        "channel": "#alert",
        "text": "A new breaking change PR",
    }
}

GitHub Actions for a specific repository failed

notify[msg] {
    input.name == "workflow_run"
    input.event.action == "completed"
    input.event.repository.full_name == "m-mizutani/nanika"
    input.event.workflow.name == "deploy"
    input.event.workflow_run.conclusion != "success"

    msg := {
        "text": "nanika deployment failed",
    }
}

New Deploy Key issued

notify[msg] {
    input.name == "deploy_key"
    input.event.action == "created"
    msg := {
        "channel": "#alert",
        "text": "A new deploy key created",
        "fields": [
            {
                "name": "title",
                "value": input.event.key.title,
            },
            {
                "name": "read_only",
                "value": input.event.key.read_only,
            },
        ],
    }
}

Create, delete, and publish repositories

notify[msg] {
    input.name == "repository"
    input.event.action == "created"

    msg := {
        "channel": "#alert",
        "text": "repository created",
    }
}

notify[msg] {
    input.name == "repository"
    input.event.action == "deleted"

    msg := {
        "channel": "#alert",
        "text": "repository created",
    }
}

notify[msg] {
    input.name == "repository"
    input.event.action == "publicized"

    msg := {
        "channel": "#alert",
        "text": "repository publicized",
    }
}

Conclusion

OPA and Rego are mainly used in the field of information security and compliance because they are "policy languages," but the development philosophy of OPA and Rego is "to separate the logic of implementation and decision making" (Policy Decoupling).
Personally, I believe that there are many tools and systems that can take advantage of this idea of separation of implementation and policy, and I would like to continue to consider its application.

zlog: Secure logger in Go to prevent output of sensitive/secret values

Masayoshi Mizutani — Mon, 01 Nov 2021 23:27:45 +0000

TL; DR
I created Go logger zlog that prevent outputting secret values to the log.
https://github.com/m-mizutani/zlog

Background: Server side logging problem with secret values

It is common for many server-side services, including web services, to output and record logs about the behavior of the service. By continuously outputting logs, they can be used for troubleshooting, debugging, responding to security incidents, auditing, and clues for performance improvement. The more information contained in the log, the more clues to solve problems, so it is useful to post as much information as possible (although there is a limit), or to be able to increase the amount of information through configuration.

On the other hand, however, there is some information that is undesirable to output on the server side.

Credential: Passwords, API tokens, session tokens, and other information that can be used to obtain the privileges of another user. A common mistake is that when configuration-related information is output for debugging, the authentication information for using another service is also output, or when a request to a service is output to the log as is, the API token is mixed in.
Personal identifiable information: For example, the name, phone number, and email address of a user who uses the service. It is often the case that personal information is mistakenly included when an object containing user information is output to the log.

For the sake of explanation, I'll refer to the above as "secret values", and while it's not good to have secret values output by CLI tools, it's more difficult from the perspective of log management and operation on the server side.

Many viewers: Server-side logs are useful for troubleshooting purposes if they can be viewed by many members of the development and operation team. This is why permissions are often set to a wide range, which means that multiple people can view them if you're running a team. It is not good to allow multiple people to freely view authentication information that can be used to impersonate another user, and personal information is restricted by many laws and regulations to be viewed only by the minimum number of people required to handle personal information for business purposes. Long term storage.
Long retention: Logs tend to be retained for a long time for auditing and security purposes. For example, PCI DSS requires credit card companies to retain audit trail history for at least one year, and the SANS SIEM Operations Guide also recommends retaining logs for at least one year. It is difficult to delete a part of the log.
Difficulty in partial deletion: Even if you are aware that confidential values are being output to the log by mistake, from the perspective of log management and operation, there are many cases where the operation of "deleting only the log containing confidential values" is not expected or not possible. For example, in AWS, it is considered a best practice to store a large number of logs as objects in S3, and it tends to be difficult to partially exclude logs that contain confidential values. Also, from an auditing point of view, I have the impression that many log storage and operation services and products are designed in such a way that they cannot be easily modified or deleted (even if they could, the history would be left somewhere).

Therefore, it is better not to output secret values to the log on the server side, because it will cause various kinds of wear and tear. However, as mentioned above, it is not uncommon to inadvertently output them (I often do it myself). In particular, it is difficult to be aware of what is contained in the nested fields in the structure each time, and I do not want to think about it every time.

One tool that addresses this issue is an OSS called blouson. It has a function to hide the values of field names with secure_ prefix in Ruby on Rails and the corresponding values from SQL statements included in Exceptions. Since I have been developing exclusively in Go recently, I wanted to use a tool with a similar function in Go.

zlog implementation

So, I implemented zlog.

The Go language comes with a standard logging package called log and a logger interface, but this time I implemented it completely separately. The reason for this is that the concept of log level is not well known, and structured logging is not supported. In particular, when considering server-side output, it is desirable to output structured logs as much as possible, since, for example, AWS's CloudWatch Logs and GCP's Cloud Logging support logs in JSON format, and JSON schemas can be used for searching.

Some famous existing libraries that support structured logging are zap, logrus, zerolog, but none of them implement such a function to hide the secret value.

How to hide a secret value

Basically, when you input a variable (or a structure) you want to log into the With() chain method, the filter in the Filters array checks if it contains the secret value, and hides the field if it is found to contain it. The following five use cases are assumed.

Specifying a specific value

This function is designed to hide limited and predetermined secret values, such as API tokens used by the application itself to call external services. This is expected to not only hide specific fields, but also to deal with some values that are introduced by string operations such as fmt.Sprintf.

const issuedToken = "abcd1234"
authHeader := "Authorization: Bearer " + issuedToken

logger := newExampleLogger()
logger.Filters = []zlog.Filter{
    filter.Value(issuedToken),
}
logger.With("auth", authHeader).Info("send header")
// Output:  [info] send header
// "auth" => "Authorization: Bearer [filtered]"

Specify a specific field name.

This filter hides the secret value if it matches the field name of the specified structure. In addition to exact match, a forward match filter named filter.FieldPrefix is also provided. This allows you to hide the secret value only if the field name has a Secure prefix, just like blouson.

type myRecord struct {
    ID    string
    EMail string
}
record := myRecord{
    ID:    "m-mizutani",
    EMail: "mizutani@hey.com",
}

logger.Filters = []zlog.Filter{
    filter.Field("EMail"),
}
logger.With("record", record).Info("Got record")
// Output:  [info] Got record
// "record" => zlog_test.myRecord{
//   ID:    "m-mizutani",
//   EMail: "[filtered]",
// }

Specifying a custom defining type

The Go language allows you to create your own defining types from existing types, which can be used to distinguish your usage from existing ones (e.g. key of value in context package). To use this mechanism, you can define a type you want to keep secret and specify it with a Filter to prevent it from being displayed. The advantage of this method is that copying a value from a custom type to the original type requires a cast, making it easier for the developer to notice unintentional copying. (Of course, this is not a perfect solution because you can still copy by casting.

I think this method is useful for use cases where you need to use secret values between multiple structures.

type password string
type myRecord struct {
    ID       string
    Password password
}
record := myRecord{
    ID:       "m-mizutani",
    Password: "abcd1234",
}

logger.Filters = []zlog.Filter{
    filter.Type(password("")),
}
logger.With("record", record).Info("Got record")
// Output:  [info] Got record
// "record" => zlog_test.myRecord{
//   ID:       "m-mizutani",
//   Password: "[filtered]",
// }

Specify by tag in a structure

It is possible to hide the fields of a structure with a specific tag. The key name of the tag is fixed to zlog and you can specify multiple values to hide. If nothing is specified, secret is targeted by default.

This is useful if you want to hide values without changing the current implementation, but do not want the management of target field names to be centralized in zlog.

type myRecord struct {
    ID    string
    EMail string `zlog:"secret"`
}
record := myRecord{
    ID:    "m-mizutani",
    EMail: "mizutani@hey.com",
}

logger.Filters = []zlog.Filter{
    filter.Tag(),
}
logger.With("record", record).Info("Got record")
// Output:  [info] Got record
// "record" => zlog_test.myRecord{
//   ID:    "m-mizutani",
//   EMail: "[filtered]",
// }

Specify (what appears to be) personal information.

This is an experimental effort and not a very reliable method, but it may be of some value. It is a way to detect and hide personal information that should not be output based on a predefined pattern like many DLP (Data Leakage Protection) solutions.

In the following example, we use a filter that we wrote to detect Japanese phone numbers. The content is just a regular expression. This method does not have as many patterns as the existing DLP solutions, and the patterns are not accurate enough, but we hope to expand it in the future if necessary.

type myRecord struct {
    ID    string
    Phone string
}
record := myRecord{
    ID:    "m-mizutani",
    Phone: "090-0000-0000",
}

logger.Filters = []zlog.Filter{
    filter.PhoneNumber(),
}
logger.With("record", record).Info("Got record")
// Output:  [info] Got record
// "record" => zlog_test.myRecord{
//   ID:    "m-mizutani",
//   Phone: "[filtered]",
// }

Create your own filters

You can create your own filters using methods implemented according to the zlog.Filter interface. The two required methods are as follows

ReplaceString(s string) string: This method is called if the value to be checked is of type string. The argument is the value to be tested and the return value is the value to be replaced. If nothing needs to be done, the argument is returned as is. It is assumed that you want to hide a part of the string.
ShouldMask(fieldName string, value interface{}, tag string) bool: The field name (or key name if it is a map type) of the value to check, the value itself, and if the structure is tagged with zlog, the information is passed as arguments. if the structure is tagged with zlog. If the return value is false, nothing is done; if it is true, the entire field is hidden. If the return value is false, nothing is done, and if it is true, the entire field is hidden. The hidden value will be replaced with the value [filtered] if the original type is string, otherwise it will be empty.

Comparison with other implementations.

Finally, I would like to conclude with a comparison with other implementations. If you are interested in using it, I hope this will be helpful.

Advantage

The ability to hide values in a variety of ways makes it possible to tailor the implementation to the use case.
For example, github.com/gyozatech/noodlog provides a similar function, but it assumes conversion to JSON, and type information is lost in the output stage. Since zlog basically performs hiding while preserving the types, you can freely choose the format of the final output.
The advantage of zlog over DLP solutions is that you don't have to remove the secret values from the stored logs.

Disadvantage

Computation and memory usage is lower than other loggers due to deep copying of variables and structures that can be modified to display data.
- This makes it unsuitable for services where logging has a direct impact on performance.
- Also, it may cause a heavy load when trying to display huge data.
The implementation is still in its infancy and does not yet fully implement the functionality of a modern logger.

golambda: A suite of Go utilities for AWS Lambda functions to ease adopting best practices

Masayoshi Mizutani — Sun, 27 Dec 2020 01:43:27 +0000

I often use Go language + AWS Lambda at work, especially developing backend processing for security monitoring related infrastructure. While advancing the development, there were various tips that said "this is convenient", but since the processing was too detailed, I used it for development by copying it between projects. However, as the number of projects managed has increased, the behavior has become different, and the number of tips has accumulated to some extent, so I cut it out as a package.

https://github.com/m-mizutani/golambda

Basically, I assume Lambda for data processing pipeline and a little integration, and implement the following four functions.

Event decapsulation
Structured logging
Error handling
Secret values management

Implemented features

Event decapsulation

AWS Lambda can have event source(s) and be invoked by trigger notifications from them. When triggered, the Lambda function is started by passing the data structure of the event source such as SQS and SNS. Therefore, it is necessary to extract the data to be used by yourself from various structural data. If you specify callback (Handler in the example below) in the function golambda.Start(), the necessary information is stored in golambda.Event and can be retrieved from there.

package main

import (
    "strings"

    "github.com/m-mizutani/golambda"
)

type MyEvent struct {
    Message string `json:"message"`
}

// Handler concat messages of SQS
func Handler(event golambda.Event) (interface{}, error) {
    var response []string

    // Extract body of SQS
    bodies, err := event.DecapSQSBody()
    if err != nil {
        return nil, err
    }

    // Handling as multiple data because of batch SQS messages
    for _, body := range bodies {
        var msg MyEvent
        if err := body.Bind(&msg); err != nil {
            return nil, err
        }

        // Save messages to slice
        response = append(response, msg.Message)
    }

    // concat
    return strings.Join(response, ":"), nil
}

func main() {
    golambda.Start(Handler)
}

This sample code is located in the ./example/deployable directory where you can deploy and try it out.

Contrary to the function DecapXxx that implements the process of retrieving data, the process of embedding data is prepared asEncapXxx. This allows you to write a test for the above Lambda Function as follows:

package main_test

import (
    "testing"

    "github.com/m-mizutani/golambda"
    "github.com/stretchr/testify/require"

    main "github.com/m-mizutani/golambda/example/decapEvent"
)

func TestHandler(t *testing.T) {
    var event golambda.Event
    messages := []main.MyEvent{
        {
            Message: "blue",
        },
        {
            Message: "orange",
        },
    }
    // Inject event data
    require.NoError(t, event.EncapSQS(messages))

    resp, err := main.Handler(event)
    require.NoError(t, err)
    require.Equal(t, "blue:orange", resp)
}

Currently, golambda supports SQS, SNS, and SNS over SQS (SQS queue that subscribes to SNS), but I'm planning to implement DynamoDB stream and Kinesis stream later.

Structured Logging

Standard log output destination of Lambda function is CloudWatch Logs, but Logs and Insights (log viewer) supports JSON-formatted logs. Therefore, it is useful to have a logging tool that can output in JSON format instead of using the Go language standard log package.

Logging on Lambda, including the log output format, generally has the same requirements. General logging tools have many options for output methods and formats, but we don't need to change the settings for each Lambda function. Also, in most cases, only the variables necessary for explaining the message + context are sufficient for the output content, so I implemented a wrapper with such a simplification in golambda. In the actual output part, zerolog is used. Actually, it was good to expose the logger created with zerolog as it is, but I thought it would be easier for me to narrow down what I could do, so I dared to wrap it.

A global variable called Logger is exported so that messages for each log level such as Trace, Debug, Info, and Error can be output. We have Set, which allows you to permanently embed any variable, and With, which allows you to add values in a method chain.

// ------------
// Use With() to embed temporary variables
v1 := "say hello"
golambda.Logger.With("var1", v1).Info("Hello, hello, hello")
/* Output:
{
    "level": "info",
    "lambda.requestID": "565389dc-c13f-4fc0-b113-xxxxxxxxxxxx",
    "time": "2020-12-13T02:44:30Z",
    "var1": "say hello",
    "message": "Hello, hello, hello"
}
*/

// ------------
// Use Set () to embed variables that you want to output permanently, such as request ID
golambda.Logger.Set("myRequestID", myRequestID)
// ~~~~~~~ snip ~~~~~~
golambda.Logger.Error("oops")
/* Output:
{
    "level": "error",
    "lambda.requestID": "565389dc-c13f-4fc0-b113-xxxxxxxxxxxx",
    "time": "2020-11-12T02:44:30Z",
    "myRequestID": "xxxxxxxxxxxxxxxxx",
    "message": "oops"
}
*/

In addition, CloudWatch Logs is relatively expensive for writing logs, and if you constantly output detailed logs, it will greatly affect the cost. Therefore, it is usually convenient to output only the minimum log so that detailed output can be performed only when troubleshooting or debugging. In golambda, the log output level can be tampered with externally by setting the LOG_LEVEL environment variable. (Because only environment variables can be easily changed from the AWS console etc.)

Error Handling

AWS Lambda should be implemented so that each function is as single feature as possible, and to implement complicated workflow, multiple Lambda can be combined with SNS, SQS, Kinesis Stream, Step Functions, etc. Therefore, if an error occurs in the middle of processing, do not try to recover forcibly in the Lambda code, but return the error as straightforwardly as possible to make it easier to notice by external monitoring, or benefit from Lambda's own retry function. It will be easier to receive.

On the other hand, Lambda itself does not handle errors very carefully, so you need to prepare your own error handling. As mentioned earlier, it is convenient to configure the Lambda function so that if something happens, it will just return an error and fail. So, in most cases, if an error occurs, if the main function ( Handler() in the example described later) returns an error, it will output all the information about the error, and it will be output here and there. There is no need to write a process to output a log at the location where an error occurs or to skip an error somewhere.

golambda mainly handles the following two errors called by golambda.Start().

Output detailed log of the error generated by golambda.NewError or golambda.WrapError
Send the error to the exception monitoring service (Sentry)

Output detailed log of the error

From my experience, when an error occurs, there are two main things you want to know for debugging: "where it happened" and "what kind of situation it happened".

Strategies to find out where the error occurred include adding a context using the Wrap function, or having a stack trace like the github.com/pkg/errors package. In the case of Lambda, if you implement it as simple as possible, in most cases you can find out where the error occurred and how it occurred in the stack trace.

Also, by knowing the contents of the variable that caused the error, you can understand the conditions for reproducing the error. This can be dealt with by logging the variables that are likely to be relevant each time an error occurs, but it will result in poor log visibility across multiple output lines (especially if the call is deep). Also, you have to simply write the log output code repeatedly, which makes it redundant, and it is difficult to write it simply, and it is troublesome when you want to make changes related to log output.

Therefore, for the error generated by golambda.NewError() or golambda.WrapError(), the variable related to the error can be routed by the function With(). The entity is simply stored in the variable map [string] interface {} in the form of key / value. When the main logic (Handler() in the example below) returns an error generated by golambda.NewError() or golambda.WrapError(), the variables stored byWith()and the error Prints the stack trace of the generated function to CloudWatch Logs. Below is an example of the code.

package main

import (
    "github.com/m-mizutani/golambda"
)

// Handler is exported for test
func Handler(event golambda.Event) (interface{}, error) {
    trigger := "something wrong"
    return nil, golambda.NewError("oops").With("trigger", trigger)
}

func main() {
    golambda.Start(Handler)
}

Lambda with above code will output a log with the variables stored in With inerror.values and the stack trace in error.stacktrace, as shown below. The stack trace is also output as text in the %+v format of github.com/pkg/errors, but golambda.Error supports JSON format according to the output of the structured log.

{
    "level": "error",
    "lambda.requestID": "565389dc-c13f-4fc0-b113-f903909dbd45",
    "error.values": {
        "trigger": "something wrong"
    },
    "error.stacktrace": [
        {
            "func": "main.Handler",
            "file": "xxx/your/project/src/main.go",
            "line": 10
        },
        {
            "func": "github.com/m-mizutani/golambda.Start.func1",
            "file": "xxx/github.com/m-mizutani/golambda/lambda.go",
            "line": 127
        }
    ],
    "time": "2020-12-13T02:42:48Z",
    "message": "oops"
}

Send the error to the exception monitoring service

There is no particular reason why using Sentry, but it is desirable to use some kind of exception monitoring service not only for API but also for Lambda function like Web application. The reasons are as follows.

Since it is not possible to determine whether the log ended normally or abnormally from the log output to CloudWatch Logs by default, it is difficult to extract only the log of the execution that ended abnormally.
CloudWatch Logs doesn't have a function to group errors, so it's difficult to find one that has a different type of error in only one out of 100 errors.

Both can be solved to some extent by devising the error log output method, but it is recommended to use the exception monitoring service obediently because you have to be careful and implement it.

golambda sends the error returned by the main logic to Sentry by specifying the Sentry's DSN (Data Source Name) as the environment variableSENTRY_DSN (Sentry + Go Details). It doesn't matter which error you send, but the errors generated by golambda.NewError orgolambda.WrapError implement a function called StackTrace () that is compatible with github.com/pkg/errors. Therefore, the stack trace is also displayed on the Sentry side.

This is the same as what is output to CloudWatch Logs, but since you can also check it on the Sentry side screen, "View notification" → "View Sentry screen" → "Search logs with CloudWatch Logs and check details" You may be able to guess the error in the second step. Also, the search for CloudWatch Logs is slightly heavy UI, so if you don't have to search, it's better...

By the way, when you send an error to Sentry, the Sentry event ID is embedded in the CloudWatch Logs log as error.sentryEventID, so you can search from the Sentry error.

Secret values management

In Lambda, parameters that change depending on the execution environment are often stored in environment variables and used. If it is an AWS account used by an individual, it is sufficient to store it in an environment variable, but in an AWS account shared by multiple people, by separating the secret value and the environment variable, Lambda information You can separate the person (or Role) who can only refer to the secret value and the person (or Role) who can also refer to the secret value. Even if it is used by an individual, if it deals with truly dangerous information, there may be cases where the authority is separated so that even if some access key is leaked, it will not die instantly.

In my case, I often use AWS Secrets Manager to separate permissions [^ use-param-store]. Retrieving the value from Secrets Manager is relatively easy by calling the API, but I got tired of writing the same process about 100 times, so I modularized it. You can get the value by adding the json meta tag to the field of the structure.

type mySecret struct {
    Token string `json:"token"`
}
var secret mySecret
if err := golambda.GetSecretValues(os.Getenv("SECRET_ARN"), &secret); err != nil {
    log.Fatal("Failed: ", err)
}

Not implemented features

I thought it would be useful, but I didn't implement followings.

Execute arbitrary processing just before timeout: Lambda will shutdown silently after the configured maximum execution time, so there is a technique to call some processing just before timeout to output performance analysis information. However, I was not in trouble in may cases by silent shutdown of Lambda timeout because I could find out bottleneck by a few logs.
Tracing: Powertools in Python provide the ability to measure performance on AWS X-Ray using annotations and so on. If you try to do this with Go, you can enjoy it more than Use official SDK. At the moment, I didn't come up with a better way in Go than [Use the official SDK], so I didn't work on it.

How to build distributable serverless application module by AWS CDK

Masayoshi Mizutani — Wed, 29 Jul 2020 08:43:54 +0000

Background

We used to create serverless applications using AWS SAM (Serverless Application Model), but the serverless configuration information and the parameters required for it tended to be tightly coupled, making it difficult to create an application that can be used anywhere. Although there were functions such as Parameters that allowed you to embed values in definition files afterwards, there was a sense of being forced to use YAML and JSON, and there was a lot of wear and tear.

On the other hand, AWS CDK (Cloud Development Kit) allows you to write Constrcut, which is the equivalent of a definition file, in code and is quite flexible. I've created several serverless applications as distributable module, and I'll summarize my findings in an article.

Please see https://docs.aws.amazon.com/cdk/latest/guide/home.html for more detail of AWS CDK.

How to build

Create a new CDK project

Let's create a module named onigiri, which is a normal CDK project. Let's start by creating a normal CDK project.

% mkdir onigiri
% cd onigiri
% cdk init --language typescript

Write CDK Construct

Construct is the template for the Stack that will eventually be deployed in CloudFormation. It can be written in the file lib/onigiri-stack.ts, which will be generated when you init with the name onigiri. In the following example, a Lambda Function and a DynamoDB are deployed in a construct where a Lambda Function is queried and its result is written to DynamoDB.

import * as cdk from "@aws-cdk/core";
import * as dynamodb from "@aws-cdk/aws-dynamodb";
import * as lambda from "@aws-cdk/aws-lambda";
import * as events from "@aws-cdk/aws-events";
import * as eventsTargets from "@aws-cdk/aws-events-targets";
import { NodejsFunction } from "@aws-cdk/aws-lambda-nodejs";

import * as path from "path";

export interface properties extends cdk.StackProps {
  someParameter: string;
}

export class OnigiriStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props: properties) {
    super(scope, id, props);

    const cacheTable = new dynamodb.Table(this, "cacheTable", {
      partitionKey: { name: "pk", type: dynamodb.AttributeType.STRING },
      billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
    });

    const lambdaQuery = new NodejsFunction(this, "query", {
      entry: path.join(__dirname, "lambda/query.js"),
      handler: "main",
      environment: {
        TABLE_NAME: cacheTable.tableName,
        SOME_PARAM: props.someParameter,
      },
    });

    cacheTable.grantReadWriteData(lambdaQuery);

    new events.Rule(this, "periodicQuery", {
      schedule: events.Schedule.rate(cdk.Duration.minutes(10)),
      targets: [new eventsTargets.LambdaFunction(lambdaQuery)],
    });
  }
}

Here are some tips for writing Construct

As we will be using TypeScript to create the Lambda function, we will use the NodejsFunction in the @aws-cdk/aws-lambda-nodejs package to deploy the modules under node_modules at the same time. We use the NodejsFunction in the @aws-cdk/aws-lambda-nodejs package to deploy the following modules at the same time. It's also possible to store a node_modules in a separate layer. /node_modules` in another layer, but I'll spare you the trouble.
- Although the path parameter of NodejsFunction can be relative to this file, the standard of relative path is changed when distributed as a module, so __dirname is joined to specify absolute path.
- The path is specified as a .js file. This is because it doesn't work well if you try to transcompile the module via NodejsFunction after the module is distributed.
When you add a CDK module such as @aws-cdk/aws-lambda, make sure to install it with the same version of aws-cdk/core added at init time. If the version is slightly out of alignment, the tsc error occurs.
If you use it as a single CDK project, you should add CDK modules like @aws-cdk/aws-lambda as devDependencies as npm i --save-dev @aws-cdk/aws-lambda.

Also, as it's good to have variable parameters in the distribution construct, I've extended the cdk.StackProps argument from the OnigiriStack to create an interface called properties. This allows you to pass in some environment variables and external resources as arguments.

Create a Lambda Function at the same time. In this case, I will create a Lambda Function file in . /onigiri/lib/lambda/query.ts, where the Lambda Function file is located.

Edit package.json

Make the following changes to package.json.

Added files: ["lib"]: to place transcompiled code under lib.
Add "main": ". /Add lib/onigiri-stack.js"`: Construct code can now be read from the distribution project.
Added "type": ". /lib/onigiri-stack.d.ts" added: Ibid.
Added "prepare": "tsc" in the scripts: to make it possible to transcompile the package when it is published or installed from github.

You can also set the output of transcompilation to dist and you can adjust it as you like. See https://github.com/m-mizutani/dnstrack/blob/master/package.json for a sample.

Deployment

The purpose of this article is to distribute your CDK construct as a module, but you may want to deploy it as a test. One way to do this is to make bin/onigiri.ts deployable with environment variables. As an example, let's set up bin/onigiri.ts as follows.

#!/usr/bin/env node
import "source-map-support/register";
import * as cdk from "@aws-cdk/core";
import { OnigiriStack } from "../lib/onigiri-stack";

const stackName = process.env.ONIGIRI_STACK_NAME || "onigiri";
const app = new cdk.App();
new OnigiriStack(app, stackName, {
  someParameter: process.env.ONIGIRI_PARAM!,
});

By doing so, you can specify an environment variable to deploy directly from the developing repository, but avoid embedding parameters in the development environment.

% env ONIGIRI_STACK_NAME=onigiri-1 ONIGIRI_PARAM=nanika cdk deploy

Publish and distribute the pacakge

You can reuse your CDK constructs in external projects by publishing them with npm publish and npm i onigiri. Also you can install the pacakge from github by npm i your-github-account/onigiri .

After that, create another construct to deploy the package.

#!/usr/bin/env node
import "source-map-support/register";
import * as cdk from "@aws-cdk/core";
import { OnigiriStack } from "onigiri";

const app = new cdk.App();
new OnigiriStack(app, "wagaya-no-onigiri", {
  someParameter: "nanika-sugoi-guzai",
});

Sample

The following is a construct made according to the above method, for your reference.

DNS track: https://github.com/m-mizutani/dnstrack

altenv: Powerful CLI Environment Variable Manager

Masayoshi Mizutani — Mon, 04 May 2020 09:24:14 +0000

I just created a tool to manage environment variables in CLI: altenv

https://github.com/m-mizutani/altenv

Motivation

In my business, I'm developing various tools and system. However we felt that the more projects we had to deal with, the more complicated it became to manage the setting values used for development and deployment. In particular, some tools are open to the public, so if you try to separate the implementation and settings properly (especially in the case of local development settings). So I just want a universal tool to manage these varialbles.

Simply put, altenv reads environment variables according to the contents of options and configuration files, and executes the specified commands with the environment variables. A simple example is following.

$ cat test.env
DBNAME=hoge
$ altenv -e test.env npm run server
# running npm server with environment variable DBNAME=hoge

Use cases

Details of usage is wrote in the repository and let me roughly describe the usage for each use case. You can specify various options on the command line, but the configuration file $ HOME / .altenv is read by default. So I'd like to introduce the configuration there.

Use different environment variable(s) by project directory.

In configuration file, wordir.xxx can be used to define environment variables for each working directory. (xxx is just a label)

[workdir.proj1]
dirpath = "/Users/mizutani/.ghq/github.com/xxx/project1"
define = ["DB_NAME=mydb1"]

[workdir.proj2]
dirpath = "/Users/mizutani/.ghq/github.com/yyy/project2"
define = ["DB_NAME=mydb2"]

When current working directory (CWD) is under dirpath, configurations
in the table is enabled. Specifically, it looks like following. (-r dryrun option only output a list of imported environment variables)

$ cd /Users/mizutani/.ghq/github.com/xxx/project1
$ altenv -r dryrun
DB_NAME=mydb1
$ cd ../../yyy/project2
$ altenv -r dryrun
DB_NAME=mydb2

Referencing environment variables from an external repository or file sharing service

If you use only define configuration to define environment variables, all environment variables need to be written in $HOME/.altenv. It difficult to share common environment variables with your team member. So, by referencing an external file, you can share environment variables using git repository and file sharing services (e.g. Google File Stream, Dropbox, etc.). NOTE: the git repository assumes that you will pull the latest file yourself.

[workdir.proj1]
dirpath = "/Users/mizutani/.ghq/github.com/xxx/project1"
envfile = ["/Users/mizutani/Google Drive File Stream/Shared drives/MyTeam/config/project1.env"]

[workdir.proj2]
dirpath = "/Users/mizutani/.ghq/github.com/yyy/project2"
envfile = ["/Users/mizutani/.ghq/github.com/yyy/configs/project2.env"]

With this setting, files in the Shared Drive of Google Drive will be referenced when in the project1 directory, and files in the config management repository will be referenced when in the project2 directory.

Switching Staging and Production

For example, when accessing both staging environment and production environment even in the same directory, it is necessary to switch the environment variable to be used. By preparing a table called profile.xxx, you can select the settings you want to use each time. The xxx in profile is the profile name as it is.

[profile.proj1-stg]
envfile = ["/path/to/proj1/stg.env"]

[profile.proj1-prd]
envfile = ["/path/to/proj1/prd.env"]

With this setting, the environment variables read by the -p option can be switched.

$ altenv -p proj1-stg ./deploy.sh
# The deploy.sh is executed after the environment variables for staging are read.

Use secret value as environment variable (macOS only)

By saving environment variables in the macOS Keychain, you can manage secret values such as API keys more safely than saving plain text in a file. This feature is inspired by envchain.

With the option -r update-keychain -w <namespace>, the read environment variables are written to Keychain. <namespace> means just namespace and you can select namespace when both of reading and writing.

For example, if you have already saved the file in plain text, you can export it to Keychain as follows.

$ altenv -e credential.env -r update-keychain -w mycred

Alternatively, if it has already been read in the environment variable, you can write it to Keychain as follows. For example, let me save Environment Variables for AWS CLI tool to Keychain.

$ env | grep -e "^AWS_ " | altenv -i env -r update-keychain -w aws-cli

You can also copy and paste them out one at a time.

$ altenv --prompt AWS_SECRET_ACCESS_KEY -r update-keychain -w aws-cli
Enter AWS_SECRET_ACCESS_KEY Value:
# No-echo because assuming secret value

-k <namespace> option imports saved environment variable in Keychain.

$ altenv -k aws-cli -r dryrun
AWS_SECRET_ACCESS_KEY=xxxxxxx
AWS_ACCESS_KEY_ID=xxxx
$ altenv -k aws-cli aws s3 ls
2019-11-21 19:02:20 mybucket-1
2019-11-21 19:02:22 mybucket-2
...

How to build Web App with Go + gin-gonic + Vue

Masayoshi Mizutani — Mon, 13 Jan 2020 00:39:47 +0000

This article describes how to build and develop Web application with Go + gin-gonic (Web Application Framework in Go) + Vue.

(Edited) Sample code repository: https://github.com/m-mizutani/web-app-go

Steps

Configure webpack

At first, install yarn packages

$ yarn init
$ yarn add -D @babel/cli @babel/core @babel/preset-env babel-loader webpack webpack-cli webpack-dev-server html-webpack-plugin vue-loader vue-template-compiler css-loader vue-style-loader sass-loader
$ yarn add babel-polyfill vue  node-sass axios

After that, create webpack.config.js file.

const path = require('path');
const VueLoaderPlugin = require("vue-loader/lib/plugin");

module.exports = {
  mode: "development",
  entry: ["babel-polyfill", path.resolve("src", "js", "index.js")],
  output: {
    filename: "bundle.js",
    path: path.join(__dirname, "static/js/"),
    publicPath: "/js"
  },
  module: {
    rules: [
      {
        test: /\.vue$/,
        loader: "vue-loader"
      },
      {
        test: /\.js$/,
        loader: "babel-loader"
      },
      {
        test: /\.s[ac]ss$/i,
        use: [
          // Creates `style` nodes from JS strings
          'vue-style-loader',
          // Translates CSS into CommonJS
          'css-loader',
          // Compiles Sass to CSS
          'sass-loader',
        ],
      }
    ]
  },
  resolve: {
    extensions: [".js", "json", "jsx", "vue"],
    alias: {
      vue$: "vue/dist/vue.esm.js"
    }
  },
  devServer: {
    contentBase: "static",
    proxy: {
      "/api": "http://localhost:9080"
    }
  },
  plugins: [new VueLoaderPlugin()]
};

Additoinaly, put following setting into package.json.

  "scripts": {
    "start": "webpack-dev-server",
    "build": "webpack --optimize-minimize"
  },

Add web assets

Required following files:

src/css/main.scss
src/js/index.js
src/js/app.vue
static/index.html

main.scss

body {
  font-family: "Helvetica Neue", "Helvetica", Helvetica, Arial, sans-serif;
}

index.js

import '../css/main.scss'

import _ from 'lodash';
import "babel-polyfill";
import Vue from "vue";

import App from "./app.vue";

new Vue({
    el: "#app",
    render: h => h(App)
});

app.vue

<template>
  <div>
    <div>MyApp</div>
    <button v-on:click="showMessage">This is test</button>
    <div>{{message}}</div>
  </div>
</template>
<script>
import axios from "axios";
const appData = {
  message: ""
};
export default {
  data() {
    return appData;
  },
  methods: {
    showMessage: showMessage
  }
};
function showMessage() {
  axios.get("/api/v1/hello").then(res => {
    console.log(res);
    appData.message = res.data.message;
  });
}
</script>
<style>
</style>

index.html

<!doctype html>
<html>
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width" />
    <title>MyApp</title>
  </head>
  <body>
    <div id="app"></div>
    <script src="/js/bundle.js"></script>
  </body>
</html>

Add go module

Initialize go module.

$ go mod init github.com/m-mizutani/web-app-go

After that, add main.go.

package main

import (
    "fmt"
    "os"

    "github.com/gin-contrib/static"
    "github.com/gin-gonic/gin"
    "github.com/sirupsen/logrus"
)

var logger = logrus.New()

var logLevelMap = map[string]logrus.Level{
    "trace": logrus.TraceLevel,
    "debug": logrus.DebugLevel,
    "info":  logrus.InfoLevel,
    "warn":  logrus.WarnLevel,
    "error": logrus.ErrorLevel,
}

type arguments struct {
    LogLevel       string
    BindAddress    string
    BindPort       int
    StaticContents string
}

func runServer(args arguments) error {
    level, ok := logLevelMap[args.LogLevel]
    if !ok {
        return fmt.Errorf("Invalid log level: %s", args.LogLevel)
    }
    logger.SetLevel(level)
    logger.SetFormatter(&logrus.JSONFormatter{})

    logger.WithFields(logrus.Fields{
        "args": args,
    }).Info("Given options")

    r := gin.Default()

    r.Use(static.Serve("/", static.LocalFile(args.StaticContents, false)))
    r.GET("/api/v1/hello", func(c *gin.Context) {
        c.String(200, `{"message":"hello, hello, hello"}`)
    })

    if err := r.Run(fmt.Sprintf("%s:%d", args.BindAddress, args.BindPort)); err != nil {
        return err
    }

    return nil
}

func main() {
    args := arguments{
        LogLevel: "info",
        BindAddress: "0.0.0.0",
        BindPort: 9080,
        StaticContents: "./static",
        }

    if err := runServer(args); err != nil {
        logger.WithError(err).Fatal("Server exits with error")
    }
}

Development

Run go server

$ go run .

air command with .air.conf like following is recommended for hot reloading.

[build]
include_ext = ["go", "tpl", "tmpl", "html"]
exclude_dir = ["src", "tmp", "node_modules"]
delay = 1000 # ms
stop_on_error = true
log = "air_errors.log"

And run air

$ air -c .air.conf

Start webpack development server

$ npm run start

Open browser

Open http://localhost:8080 to confirm Web UI.

Deploy

$ npm run build
$ go build -o sercer
$ cp -r server static /path/to/server

Created Go library to acquire and manage blacklists of IP addresses and domain names

Masayoshi Mizutani — Fri, 03 Jan 2020 07:22:38 +0000

It is almost the title, but I have created a library called badman for the purpose of obtaining blacklists published on various sites and using it for comparison with traffic logs etc. badman stands for Blacklisted Address and Dmain name Manager.

https://github.com/m-mizutani/badman

A little while ago, if you wanted to monitor your network security, I think you often used a network based IDS such as snort. However, in recent years, most communications have been on HTTPS. Then, network based IDS has not been a very effective method at present. One of the alternatives is to store network flow information (IP address, port number, protocol, data size exchanged, etc.) and DNS query + response logs, and then verifies if communication to malware's Command & Control server or fraudulent site has occurred. With this method, even if the communication such as the Web is encrypted, you can discover if there was suspicious communication. However, if you download the list from the blacklist provider every time you check the communication logs that occur continuously, it will put a load on the provider and your network. Therefore, I created this library in the hope that it could be used once and then reused over the medium to long term.

The reason for implementing the code as library is that the method of inputting logs and the contents of logs were considered to differ greatly depending on the environment. Specific architecture examples will be described later.

How to use

If you are familiar with Go language, I think that if you see the following code, you will get an image.

package main

import (
    "bufio"
    "log"
    "os"

    "github.com/m-mizutani/badman"
    "github.com/m-mizutani/badman/source"
)

func main() {
    man := badman.New()

    if err := man.Download(source.DefaultSet); err != nil {
        log.Fatal("Fail to download:", err)
    }

    // ipaddrs_in_traffic_logs.txt contains IP address line by line
    fd, err := os.Open("ipaddrs_in_traffic_logs.txt")
    if err != nil {
        log.Fatal("Fail to open a file:", err)
    }
    defer fd.Close()

    scanner := bufio.NewScanner(fd)
    for scanner.Scan() {
        entities, err := man.Lookup(scanner.Text())
        if err != nil {
            log.Fatal("Fail to lookup:", err)
        }

        if len(entities) > 0 {
            log.Printf("Matched %s in %s list (reason: %s)\n",
                entities[0].Name, entities[0].Src, entities[0].Reason)
        }
    }
}

Here's what this code does:

Download multiple lists from sites offering blacklists and create your own blacklist repository
Open ipaddrs_in_traffic_logs.txt which contains the list of IP addresses and extract one line (one IP address) at a time
Verify that the retrieved IP address is included in the repository

In this sample, the download of the blacklist of 1. will be executed every time, but originally it is assumed that the downloaded blacklist data is saved locally once and reused. There are two ways to save: A) Write the serialized data to a file and read the file from next time, B) Persistent data store (currently packaged only in AWS DynamoDB) as backend. See README in the repository for detailed usage.

Architecture examples

Assuming use on AWS, I present two example architectures. It is assumed that each of them will create and run their own programs using badman as a library.

Serverless architecture

In this case, we use two Lambda functions. The first (left) function gets the blacklist periodically and stores the serialized blacklist data in S3. The second (right) Lambda function is called by the ObjectCreated event when the traffic log file is uploaded to S3. The Lambda function then downloads both the serialized blacklist data and the log file and checks if the IP address in the traffic log is on the blacklist. If present, Lambda will notify the administrator via a communication tool such as Slack.

On the other hand, the disadvantage is delay. Depending on the log flow, there will be a delay in the buffering time since logs will be compiled to some extent before uploading to S3. As a rule of thumb, it takes about a few minutes to a dozen minutes. If you need real-time performance, you may want to adopt the following server model.

Server based architecture

In this architecture, a constantly running program runs on the host (AWS EC2 in this example). The main advantage is stream processing for real-time performance. As mentioned earlier, in the serverless model, it is necessary to collect logs to some extent before uploading them to S3, so a delay of several minutes occurs from the generation of logs to the completion of processing. Minimize delays by using a constantly running server program to constantly flush data.

Assuming that this program runs fluentd's forward service (the protocol is [https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1]) And receive traffic logs via fluentd. Then use badman to check the traffic log for the blacklisted IP addresses. Blacklists are expected to be updated regularly. Use DynamoDB as a repository for storing data so that you can recover if the host running the program (in this case, EC2) crashes.

The disadvantage of this architecture is that it is difficult to allocate resources. Perhaps the log flow is a bottleneck, so you need to configure your host resources for that maximum. You should also be aware of the potential for a significant increase in log traffic due to trouble or unusual events. In order to solve this, it is necessary to use it in combination with a mechanism such as automatic scaling.

Precautions for use

Each site offering a blacklist has a different policy. Please note that the usage conditions may not be met depending on the environment or organization. In order to limit the blacklist service sites that can be used, users can also select sites to use themselves. You can also import data from your own site or a private data store by implementing a structure that satisfies the interface provided by badman yourself.

The policy of each site is also summarized in README, so please refer to that .

A log format analyzing tool from existing text log file

Masayoshi Mizutani — Thu, 24 May 2018 00:54:33 +0000

NOTE: This article was written mainly by machine translation. I apologize if this is hard to understand.

I created a log format analysing tool in Go and this post introduce it.

What is "log format analysis"?

In this post, "log format" means like format string in C, Go or etc.

log.Printf("Requested from %s", ipAddr)

This code output following logs.

2018/05/23 23:25:00 Requested from 10.0.2.1
2018/05/23 23:25:10 Requested from 192.168.1.5
2018/05/23 23:25:24 Requested from 10.0.1.5

The original format is embedded in %s as IP address and output as text like below. Since this example is very simple, it is easy to guess from the bottom to the top, but as the content becomes more complicated it is rarely common to say, 'What is this value or fixed statement?' The tool created this time is to infer the format statement (close to) above from the output below. We are implementing two functions for This tool: 1) Estimate the format from already output log file , and 2) Use the estimated format and see where the log corresponding to that format appeared in the log file.

Why the tool is required?

This tool is unnecessary in environments that handle only normalized and structured log data, but it is useful in the following situations.

When you want to grasp the whole picture of the log : I think that it is particularly frequent in the context of security analysis, but there are times when it is necessary to see a large amount of logs that have never been seen before and to draw knowledge from there . Even if you try to view with the less command at all, it is severe for humans, so what kind of logs are there as a whole? And what kind of distribution do you do? If you find out that it is, the analysis will be much easier. Especially in the security analysis, it is not the log concerning the usual service accounting for 99% of the whole in many cases, it is the point where something abnormality happened. Since it is easy for errors and processing that are not normally seen to occur, logs when abnormality happens, if you can grasp where logs with abnormal logs = unusual formats appear, we will analyze them first by focusing on them You can make a foothold.
If you need to reuse logs output in text format : If you are already running a service etc. and you want to output logs in text, add the log to the regular expression etc. It is necessary to extract the value being processed. If you have a specification, it is nice, but if not, you can see the source code or write a regular expression → make sure it is exhaustive → fix the regular expression, you have to repeat things like it is rather troublesome ((I think there are tsukkom that such an environment is more funny, but it was rarely a situation, especially in the former position). This tool does not estimate the regular expression of the value to be extracted, but it will be easier to work because it can cover how far it can be done from the existing logs.

Usage

If you already have Go language environment, you can install it by go get github.com/m-mizutani/logptn

https://github.com/m-mizutani/logptn

For example, I try to use this tool with following logs.

$ cat test.log
Feb  1 07:56:49 pylon sshd[5153]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.0.3  user=root
Feb  1 07:56:51 pylon sshd[5153]: Failed password for root from 192.168.0.3 port 7176 ssh2
Feb  1 07:56:51 pylon sshd[5153]: Connection closed by 192.168.0.3 [preauth]
Feb  1 08:01:26 pylon sshd[5156]: Invalid user upload from 192.168.0.3
Feb  1 08:01:26 pylon sshd[5156]: input_userauth_request: invalid user upload [preauth]
Feb  1 08:01:26 pylon sshd[5156]: pam_unix(sshd:auth): check pass; user unknown
Feb  1 08:01:26 pylon sshd[5156]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.0.3
Feb  1 08:01:28 pylon sshd[5156]: Failed password for invalid user upload from 192.168.0.3 port 51058 ssh2
Feb  1 08:01:28 pylon sshd[5156]: Connection closed by 192.168.0.3 [preauth]
Feb  1 08:05:01 pylon CRON[5159]: pam_unix(cron:session): session opened for user root by (uid=0)
Feb  1 08:05:01 pylon CRON[5159]: pam_unix(cron:session): session closed for user root
Feb  1 08:05:54 pylon sshd[5162]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.0.3  user=root
Feb  1 08:05:56 pylon sshd[5162]: Failed password for root from 192.168.0.3 port 33005 ssh2
Feb  1 08:05:56 pylon sshd[5162]: Connection closed by 192.168.0.3 [preauth]
Feb  1 08:10:28 pylon sshd[5165]: Invalid user mythtv from 192.168.0.3
Feb  1 08:10:28 pylon sshd[5165]: input_userauth_request: invalid user mythtv [preauth]
Feb  1 08:10:28 pylon sshd[5165]: pam_unix(sshd:auth): check pass; user unknown
Feb  1 08:10:28 pylon sshd[5165]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.0.3
Feb  1 08:10:30 pylon sshd[5165]: Failed password for invalid user mythtv from 192.168.0.3 port 59978 ssh2
Feb  1 08:10:30 pylon sshd[5165]: Connection closed by 192.168.0.3 [preauth]
Feb  1 08:15:01 pylon CRON[5168]: pam_unix(cron:session): session opened for user root by (uid=0)
Feb  1 08:15:01 pylon CRON[5168]: pam_unix(cron:session): session closed for user root
Feb  1 08:15:26 pylon sshd[5171]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.2.3.4  user=root
Feb  1 08:15:28 pylon sshd[5171]: Failed password for root from 10.2.3.4 port 60733 ssh2
Feb  1 08:15:28 pylon sshd[5171]: Connection closed by 10.2.3.4 [preauth]
Feb  1 08:17:01 pylon CRON[5173]: pam_unix(cron:session): session opened for user root by (uid=0)
Feb  1 08:17:01 pylon CRON[5173]: pam_unix(cron:session): session closed for user root
Feb  1 08:20:35 pylon sshd[5177]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.2.3.4  user=root
Feb  1 08:20:37 pylon sshd[5177]: Failed password for root from 10.2.3.4 port 44877 ssh2
Feb  1 08:20:37 pylon sshd[5177]: Connection closed by 10.2.3.4 [preauth]
Feb  1 08:25:01 pylon CRON[5180]: pam_unix(cron:session): session opened for user root by (uid=0)
Feb  1 08:25:01 pylon CRON[5180]: pam_unix(cron:session): session closed for user root
Feb  1 08:25:16 pylon sshd[5183]: Invalid user user from 10.2.3.4

From above log data, the tool output following result.

./logptn test.log
2018/05/20 13:30:55 arg:test.log
     4 [4ffb267b] Feb  1 *:*:* pylon sshd[*]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=*  user=root
     4 [845f4659] Feb  1 *:*:* pylon sshd[*]: Failed password for root from * port * ssh2
     6 [847ccf35] Feb  1 *:*:* pylon sshd[*]: Connection closed by * [preauth]
     3 [de051cd9] Feb  1 08:*:* pylon sshd[*]: Invalid user * from *
     2 [8e9e2a13] Feb  1 08:*:* pylon sshd[*]: input_userauth_request: invalid user * [preauth]
     2 [22190c74] Feb  1 08:*:* pylon sshd[*]: pam_unix(sshd:auth): check pass; user unknown
     2 [83fba2bf] Feb  1 08:*:* pylon sshd[*]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.0.3
     2 [f1ba83ea] Feb  1 08:*:* pylon sshd[*]: Failed password for invalid user * from 192.168.0.3 port * ssh2
     4 [e4a6f815] Feb  1 08:*:01 pylon CRON[*]: pam_unix(cron:session): session opened for user root by (uid=0)
     4 [5256845b] Feb  1 08:*:01 pylon CRON[*]: pam_unix(cron:session): session closed for user root

In this output, it is "the number of times that format appeared" "format ID" "estimated format" from the left. Also, in the estimated format, the part that is supposed to be embedded as a value is replaced by the symbol *. In this example there are few samples, so the IP address part is not *, but as the number of samples increases it will also be replaced with *. The above is output in human readable text format, but it can also be output in json format so that it can be handled by another program.

./logptn test.log -d sjson | jq . 
{
  "formats": [
    {
      "segments": [
        "Feb  1 ",
        null,
        ":",
        null,
        ":",
        null,
        " pylon sshd[",
        null,
        "]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=",
        null,
        "  user=root"
      ],
      "count": 4
    },
(snip)

In addition, you can display in HTML format how many lines of that log format the format appeared.

$ ./logptn  ./var/log/secure -d heatmap -o secure.html

With the above command you can create a heat map showing the format of the log and what line it occurred. The heat map is the format estimated on the left, the top header is the number of lines (the line number to the line number), and the right is the total log number. The image below is a little small and hard to see, but the HTML file itself can also be downloaded from here.

Large size image

Performance

The calculation amount is O (NM), N is the total number of logs included in the log file, M is the estimated number of formats. I tried with various log files, but M converges to about 10 to 100, so N is the total number of logs affected. Although it measures only miscellaneous, I moved it with MacBookPro Early 2015 (2.7 GHz Intel Core i 5) for data that is about M = 40 and ran around 30,000 logs / sec. Perhaps it can be optimized more in terms of code, but I have not done so far yet.

Design & Implementation of Modern Router with Docker + Linux for home and small office

Masayoshi Mizutani — Tue, 09 Jan 2018 15:18:11 +0000

This article describes a router built by Docker + Linux for the home and small office.

Design

Principle

Modulability

Except for work that we have to do with the Linux kernel, I will make each service work as independently as possible.

Compared to commercially available broadband routers, machines with Linux can do anything because of its extremely high degree of freedom, but there is a problem that the environment becomes "dirty" - While keeping minor changes, dependencies etc of services and saved files will gradually become unknown - Especially if you are trying to insert or delete a new service, garbage will remain
Although there are not many opportunities to build a router, it is desirable to have a configuration that separates the service from the OS so that it can be easily migrated as much as possible when replacing the machine

Low running cost

External services (SaaS etc.) that are convenient for operation have increased compared to the past. Although I would like to actively use these things, I assume the home and small office to the end, so I do not expect to multiply the daily running cost.

Specifically, we aim to keep the monthly amount to about several hundred yen. It is seriously conscious that we can use a variety of better services if we put out money, but we design it to be the minimum necessary function based on this policy.

Just for engineers

In the first place I think that it is about IT engineers to make routers by themselves, but if it is dare to say, the purpose is to fulfil the desire that engineers want to tinker with themselves. Therefore, it is not designed for the purpose of replacing existing broadband router etc. For ordinary purposes it is far cheaper and easier to buy and set up a broadband router.

Assumptions

Hardware - x86 machine (Although it may be configured with Raspberry Pi, there is no mind to work hard with a slight performance uncertainty & ARM) - I have more than two NICs - RAM can be used with some margin (assuming about 4 GB) - Assumed that the HDD is not so large, leaving less permanent data
Required services - Packet forwarding (home network -> Internet) - Firewall - DNS cache server - DHCP server - Communication monitoring

Architecture

Based on the above requirements etc., it was designed as shown in the figure below.

Package forwarding, firewalls and other parts directly related to the network to make Linux OS obediently obedient
Other services are basically modularized by configuring with Docker, making it easier to add / change / delete services - DNS cache server (unbound) - DHCP server (kea) - Monitor communication (softflowd, dns - gazer)
Use external services as appropriate - Metrics monitoring (mackerel) - Save log (AWS S3)

Implementation

Preparing the machine

Machines prepared this time will be below.

XCY Intel Celeron J1900 Barebones (2 Ghz quad-core 4 threads) Gigabit LAN * 4 Fanless small space-saving (4 G RAM 32 G SSD)
https://www.amazon.com/dp/B01N6MDE01

4 NICs of Ethernet are pierced, and the CPU has enough performance to use as a router with 2 GHz quad core, 4 G memory, 32 G SSD. Since there are four NICs, you can play with separate segments, and the aircraft is also small, so it is suitable for installation as a router.

Of course, even if it is not this machine, there is also a configuration such as sticking another NIC into the old desktop PC and using it. This time it will be explained on the premise that it will be installed using this XCY.

Installing Linux as a host

OS Installation

Since the machine of XCY used this time is an ordinary x86 machine, there is no problem in the same procedure as installing on PC or server. As the picture shows, two USB mouths are pecking, so we will install CD / DVD or USB memory etc in there. I have not tried it, but since the COM serial port is on, I can install it even if I use it.

This time I used the CD / DVD drive, connected the keyboard to another USB, and installed while copying the VGA output on the back to the screen. When using a USB CD / DVD drive, it is necessary to tamper with BIOS boot priority ... (memorization is ambiguous because it was doing it for a while) But immediately after startup, put it in the BIOS with the ESC key.

In addition, this time I chose ubuntu Linux 16.04 as the host OS. After that, we will explain it on the premise.

Package Installation

$ sudo apt update
$ sudo apt upgrade -y
$ sudo apt install -y iptables docker.io docker-compose pppoeconf git

Since the host OS side does not want to put extra things as much as possible, the installation package is minimized as much as possible.

Interface configuration

As shown in the design drawing, one NIC is used on the Internet side and the other NIC is used for the internal network side. In ubuntu Linux 16.04 (or Linux kernel 4.4.0-104-generic), the interface names written in the real machine and the device name of the OS correspond as follows.

LAN1 : enp1s0
LAN2 : enp2s0
LAN3 : enp3s0
LAN4 : enp4s0

For this time, set LAN1 (enp1s0) to the Internet side and LAN2 (enp2s0) to the internal network side. The network configuration is as follows.

Network: 10.0.0.0/24
Range of IP addresses used for DHCP: from 10.0.0.129 to 10.0.0.254
IP address of internal network side: 10.0.0.1

The settings of /etc/network/interfaces at this stage are as follows.

auto lo
iface lo inet loopback

auto enp2s0
iface enp2s0 inet static
    address 10.0.0.1
    netmask 255.255.255.0

Setting up PPPoE

Grasp the user name and password of PPPoE and execute the following command.

$ sudo pppoeconf

If you run pppoeconf you will recommend setting as nice, so if you answer the questions accordingly, it prepares it under /etc/ppp/. Please be aware that it is necessary to connect with the PPPoE-compatible device with the LAN cable at the time of executing the pppoeconf command.

NAT & F/W Configuration

Add a following line into /etc/sysctl.conf.

net.ipv4.ip_forward=1

Edit /etc/rc.local.

/sbin/iptables-restore < /etc/network/iptables
exit 0

Originally, it seems to be a good idea to put a script to read iptables under /etc/network/if-pre-up.d/, but after doing that, the docker service executed after the interface startup will be executed by iptables Since it seems to overwrite the setting, execute iptables-restore with /etc/rc.local.

I saved and edited the result of iptables-save in /etc/network/iptables. Mixing the original docker setting, setting up the setting to free the port for service, and so on are finally as follows (/etc/network/iptables).

*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]

# eth0 is WAN interface, eth1 is LAN interface, ppp0 is the PPPoE connection
-A POSTROUTING -o ppp0 -j MASQUERADE
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN

COMMIT

*filter
:INPUT DROP [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION -j RETURN

-A DOCKER -i ppp0 -p udp -m state --state ESTABLISHED -j ACCEPT
-A DOCKER -i ppp0 -p tcp -m state --state RELATED,ESTABLISHED -j ACCEPT
-A DOCKER -i ppp0 -j DROP

-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p udp -m state --state ESTABLISHED -j ACCEPT

-A INPUT -p tcp --dport 22 -s 10.0.0.0/24 -j ACCEPT
-A INPUT -p icmp -s 10.0.0.0/24 -j ACCEPT

COMMIT

rsyslog Configuation

As will be described later, since all logs are saved in S3 of AWS via fluentd, rsyslog of host OS is also set to skip log to fluentd. Create a new /etc/rsyslog.d/10-remote.conf and edit it as follows.

*.* @127.0.0.1:5514

Monitoring Service Installation

For monitoring service, use Mackerel. Similar services are Datadog and New Relic. Although I did not compare them properly, I thought that Mackrel seems to be relatively simple and easy to use, I chose custom metrics on the free frame so that it seems to be easy to use.

Although it was possible to monitor from the Docker container, it seemed confusing to forcibly reference the data on the host from the container, so I obediently installed the agent on the host OS. The basic monitoring agent can be installed by executing the following script as one line.

$ wget -q -O - https://mackerel.io/file/script/setup-all-apt-v2.sh | MACKEREL_APIKEY='2R8VBzXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' sh

For details, refer to the following page.
https://mackerel.io/orgs/mizutani-home/instruction-agent

Although it seems to be able to set up 10 monitoring rules as well, we only move rules that detect default connectivity check and CPU, Disk, Memory overuse.

Service Configuration

Services

fluentd (log collection)

You know, the log collecting tool fluentd. In this router, logs are concentrated on fluentd and managed collectively. Major sources and destinations are as follows.

Sources - Log of standard output and standard error sent from logging driver from each docker container - Host OS syslog - Netflow log sent from softflowd - DNS query / reply log sent from dns-gazer
Destinations - Logs are basically stored on AWS S3 - Send some metrics to mackerel to monitor (not implemented in this commentary, but will be implemented in the future)

As for the log, netflow log and DNS log can be stored even if it does not go 3MB a day. In the Tokyo region, the fee is \ $ 0.025 / GB (first 50 TB). Even if it gets about 1 GB in 1 year, it is 10 GB / month for 10 years monthly * \ $ 0.025 / month = 30 yen or less per month, calculated to fit the original goal.

unbound (DNS cache server)

Speaking of DNS servers for a long time, bind is famous, but if you want just to run it as a cache server, there's not much reason to use bind that implements the full DNS feature. unbound seems to be a DNS server created by focusing mainly on the function as a cache server, with its advantages such as cache pollution tolerance, performance, ease of setting, etc. It is. The setting is also very simple and it works as a DNS cache server with this setting contents only.

server:
  port: 53
  interface: 0.0.0.0
  access-control: 10.0.0.0/24 allow

  logfile: ""
  verbosity: 2

kea (DHCP server)

DHCP which dynamically sets IP address also used to be so-called isc-dhcp-server which was conventionally implemented by Internet Systems Consortium, but recently ISC itself has implemented a DHCP server named "Modern Open Source DHCPv4 & DHCPv6 Server" kea. Facebook published a blog that they also uses kea at data center. There seems to be various introduction results.

Kea seems to have the following features compared to conventional ISC DHCP server.

APIs such as RESTful configuration change are provided
RDBS (PostgreSQL, MySQL) is available for lease DB
DHCPv4 and DHCPv6 are separated

Although I do not use the setting change etc. function this time, I try to use MySQL as a lease DB.

Also as an aside, it is very big as source code, it takes a tremendous amount of time to build from the source code. (Long time compiling which takes a few hours on the 2013 version of MacBook Pro) We recommend using a binary version when using it.

softflowd (Flow monitoring)

softflowd monitors the network interface and displays the communication flow information (source and destination IP address, IP protocol, source and destination port number, transmission / reception data Quantity, number of packets, start / end time etc).

In fact, nprobe provided by ntopng got more detailed information. However, to use it as a proper operation it was necessary to purchase a license. I gave up thinking that it is not suitable for this design that I implement this cheaply.

I'm operating with softflowd for the moment, but I'm planning to replace it with another tool or a tool I develop in the future.

dns-gazer (DNS monitoring)

I use the tool called dns-gazer to log inquiries in DNS (it is developed by myself). As well as softflowd, it monitors the network interface and creates DNS query and reply logs. It outputs the log directly to fluentd as forward format, and this time we save the log to AWS S3. There is dnscap as a similar tool to capture DNS communication, but since there was a difficulty in the log output method and format, I decided to develop as a separate tool did.

DNS logs are also acquired for security purposes. Even if it can not do advanced analysis so far, it simply can detect infection of malware by confirming that it was communicating with the domain name and IP address posted in blacklist later. In addition, it is also used for the purpose of enabling forensics at the time of incidents (although it is almost impossible to do such things in a normal home environment) in some cases.

Actually I wanted to introduce the utilisation of this log together, but this part are not mature yet. For the time being, this time it is only to leave the log properly.

Preparation of external service

AWS S3

As mentioned earlier, we will use the AWS S3 bucket for saving the log file this time. I will skip about account creation steps.

When the console becomes usable with an account, first create a bucket. This time we will proceed with the premise that we prepared S3 bucket called home-network.

Once we have created the bucket, we will get the API key. Although it is not impossible even if you use the API key of your account, we recommend that you create another user dedicated to saving logs to S3 as the authority.

After creating the user (or while doing it), create a policy to write to S3 and attach it to that user. The policy example to be set up is as follows. The s3:GetObject and s3:ListBucket privileges are also granted so that s3:PutObject and fluentd's out_s3 plugin to create log files can check the existence of files on the bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::home-network",
                "arn:aws:s3:::home-network/*"
            ]
        }
    ]
}

Once you have set the policy, you can retrieve and save the information of AWSAccessKeyId andAWSSecretKey.

Setting up docker-compose

From here, I will explain on the premise that we will build the environment using docke-compose configuration prepared by me.

$ sudo mkdir -p /opt
$ sudo chown $USER /opt
$ cd /opt
$ git clone https://github.com/m-mizutani/docker-based-home-router
$ cd docker-based-home-router

Then, create a file named config.json.

$ cp config.template.json config.json
$ vim config.json

And rewrite the necessary parts.

{
  "s3": {
    "key": "AKXXXXXXXXXXXXXXXXXXXXXX",
    "secret": "Ib346xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "region": "ap-northeast-1",
    "bucket_name": "home-network"
  },
  "dhcp": {
    "interfaces": [
      "enp0s2"
    ],
    "pools": [
      "10.0.0.129 - 10.0.0.254"
    ]
  },
  "network": {
    "internal_gateway": "10.0.0.1",
    "subnet": "10.0.0.0/24"
  },
  "monitor": {
    "interface": "enp0s2"
  },
  "mackerel": {
    "api_key": "2R8xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  }
}

I will put a commentary for a moment.

Interface names under dhcp/interfaces/ and monitor/interface are basically the same. The former is the setting of kea, the latter is the monitoring interface of softflowd and dns-gazer.
dhcp Please note that the following two items are list type (once inserting multiple values into the list it should work, but it is not verified at the moment)

If you can create config.json, runsetup.py. *I confirmed it with Python 3.6. * When it is executed, various setting files are automatically generated.

$ ./setup.py
2018-01-06 03:57:57,279 [DEBUG] config path: /opt/docker-based-home-router/config.json
2018-01-06 03:57:57,282 [INFO] creating fluentd config: /opt/docker-based-home-router/fluentd/fluent.conf
2018-01-06 03:57:57,286 [DEBUG] Found Mackerel API KEY in config
2018-01-06 03:57:57,286 [DEBUG] Found Mackerel ID file: /var/lib/mackerel-agent/id
2018-01-06 03:57:57,286 [INFO] creating kea config file: /opt/docker-based-home-router/kea/kea-config.json
2018-01-06 03:57:57,289 [INFO] creating env file for mysql: /opt/docker-based-home-router/mysql.env
2018-01-06 03:57:57,289 [INFO] creating env file for dns-gazer: /opt/docker-based-home-router/dns-gazer.env

When the file is automatically generated, build the image.

$ sudo docker-compose build

Now you are ready to set it up.

docker-compose.yml

As for the composition of docker-compose and the details of the contents of each image, basically it is a feeling that you should read the code, but I will cover points precisely.

  unbound:
    build: ./unbound
    restart: always    
    ports:
    - "53:53/udp"
    depends_on:
    - fluentd
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        tag: "docker.{{.ImageName}}.{{.ID}}"

Not limited to unbound, all logging drivers are concentrated in the fluentd container and are skipped to S3. So we are specifying the fluentd container for every depends_on. I tried to make the tag docker.<Image name>.<Container ID>.

  kea:
    build: ./kea
    restart: always    
    network_mode: host

The kea needs to receive broadcast packets that occur at the client's DHCP discover. Since the docker container is basically in the internal network of host OS, DHCP discover packets do not reach the container in the normal configuration. Although I seemed possible if I could do it a lot, this time I corresponded simply by setting a setting equivalent to --net=host.

  kea-db:
    build: ./kea-db
    restart: always    
    ports:
    - 127.0.0.1:3306:3306
    env_file:
    - ./mysql.env
    volumes:
    - kea-db-vol:/var/lib/mysql

On the other hand, if the container of kea is in the state of net = host, mysql will expose 3306/tcp to local and kea will 127.0.0.1:3306 because it can not connect with the mysql container with links It is configured to connect.

The database initialization script is put in the image of mysql called kea-db. Also, since the database file is persisted by setting volume, the same lease DB will be inherited even if it is restarted.

  fluentd:
    build: ./fluentd
    restart: always
    ports:
    - 127.0.0.1:24224:24224
    - 127.0.0.1:5514:5514/udp
    - 127.0.0.1:2055:2055/udp
    volumes:
    - fluentd-buffer:/var/log/fluentd

Fluentd also takes over the directory for the buffer even after volume mount and rebooting the container. Each port is expose for the following purposes.

24224: Logging of logging driver + dns-gazer received
5514: Receive rsyslog
2055: Receive netflow

Make docker-compose to start up at machine startup

Wherever you are ready, add a startup script to /etc/rc.local at the end. Finally it will be as follows.

/sbin/iptables-restore < /etc/network/iptables
docker-compose -p router -f /opt/docker-based-home-router/docker-compose.yml --verbose up -d
exit 0

After that, if you say sudo /etc/rc.local etc, all the components should start up. Since logging flows to the background with this startup command, when debugging is necessary, it is a good idea to run the docker-compose command with the -d option turned off.

$ sudo docker-compose -p router -f /opt/docker-based-home-router/docker-compose.yml --verbose up

Experience

Since it is a router, the result is not visible and it is not easy to understand the result, but in the sense that it is working with this feeling for the time being.

It is a monitoring screen on Mackrel. Although I run two traffic monitoring processes, the traffic volume is also totally low (since the maximum instantaneous wind speed of last night is 1 M Byte / ss = 8 Mbps), CPU is not being consumed at all. The cache has been going down quite a bit when memory restarted last night, but almost 1 GB is being used.

Acknowledgements

I would like to thank Google Translator.

DEV Community: Masayoshi Mizutani

Redacting Sensitive Data in Go's slog: A Practical Guide with masq

The Problem: Sensitive Data Leaks in Logs

slog's Built-in Solution: LogValuer

Enter masq: Automatic Deep Redaction

Basic Usage

Redaction Strategies

1. By Custom Type

2. By Struct Tag

3. By Field Name

4. By Field Prefix

5. By Regex Pattern

6. By String Content

Combining Multiple Strategies

Real-World Example

Best Practices

Limitations

Conclusion

Eliminating Sensitive Values from Logs using Slog, the Prospective Official Structured Logger for Go

Background: Absolutely do not want to output secret values to the log

slog.LogValuer

Using ReplaceAttr

masq

Summary

Goast: Generic static analysis for Go Abstract Syntax Tree by OPA/Rego

TL; DR

Motivation

Separate "rule" and "implementation" by Rego

Implementation

Let's look AST of Go code

Describe a rule

Static analysis in CI

Conclusion

Control notification of every GitHub event with OPA/Rego

TL;DR

GitHub notification

Problem about "notification rule"

Separating of Policy and Implementation

ghnotify

m-mizutani / ghnotify

General GitHub event notification tool to Slack with Open Policy Agent and Rego

ghnotify

Setup

1) Retrieve Bot User OAuth Token of Slack

2) Creating Rego policy

Policy examples

Specific labels were assigned to the PR

GitHub Actions for a specific repository failed

New Deploy Key issued

Create, delete, and publish repositories

Conclusion

zlog: Secure logger in Go to prevent output of sensitive/secret values

Background: Server side logging problem with secret values

zlog implementation

How to hide a secret value

Specifying a specific value

Specify a specific field name.

Specifying a custom defining type

Specify by tag in a structure

Specify (what appears to be) personal information.

Create your own filters

Comparison with other implementations.

Advantage

Disadvantage

golambda: A suite of Go utilities for AWS Lambda functions to ease adopting best practices

Implemented features

Event decapsulation

Structured Logging

Error Handling

Output detailed log of the error

Send the error to the exception monitoring service

Secret values management

Not implemented features

How to build distributable serverless application module by AWS CDK

Background

How to build

Create a new CDK project

Write CDK Construct

Edit package.json

Deployment

`slog.LogValuer`

Using `ReplaceAttr`