zlog: Secure logger in Go to prevent output of sensitive/secret values

TL; DR
I created Go logger zlog that prevent outputting secret values to the log.
https://github.com/m-mizutani/zlog

Background: Server side logging problem with secret values

It is common for many server-side services, including web services, to output and record logs about the behavior of the service. By continuously outputting logs, they can be used for troubleshooting, debugging, responding to security incidents, auditing, and clues for performance improvement. The more information contained in the log, the more clues to solve problems, so it is useful to post as much information as possible (although there is a limit), or to be able to increase the amount of information through configuration.

On the other hand, however, there is some information that is undesirable to output on the server side.

• Credential: Passwords, API tokens, session tokens, and other information that can be used to obtain the privileges of another user. A common mistake is that when configuration-related information is output for debugging, the authentication information for using another service is also output, or when a request to a service is output to the log as is, the API token is mixed in.
• Personal identifiable information: For example, the name, phone number, and email address of a user who uses the service. It is often the case that personal information is mistakenly included when an object containing user information is output to the log.

For the sake of explanation, I'll refer to the above as "secret values", and while it's not good to have secret values output by CLI tools, it's more difficult from the perspective of log management and operation on the server side.

• Many viewers: Server-side logs are useful for troubleshooting purposes if they can be viewed by many members of the development and operation team. This is why permissions are often set to a wide range, which means that multiple people can view them if you're running a team. It is not good to allow multiple people to freely view authentication information that can be used to impersonate another user, and personal information is restricted by many laws and regulations to be viewed only by the minimum number of people required to handle personal information for business purposes. Long term storage.
• Long retention: Logs tend to be retained for a long time for auditing and security purposes. For example, PCI DSS requires credit card companies to retain audit trail history for at least one year, and the SANS SIEM Operations Guide also recommends retaining logs for at least one year. It is difficult to delete a part of the log.
• Difficulty in partial deletion: Even if you are aware that confidential values are being output to the log by mistake, from the perspective of log management and operation, there are many cases where the operation of "deleting only the log containing confidential values" is not expected or not possible. For example, in AWS, it is considered a best practice to store a large number of logs as objects in S3, and it tends to be difficult to partially exclude logs that contain confidential values. Also, from an auditing point of view, I have the impression that many log storage and operation services and products are designed in such a way that they cannot be easily modified or deleted (even if they could, the history would be left somewhere).

Therefore, it is better not to output secret values to the log on the server side, because it will cause various kinds of wear and tear. However, as mentioned above, it is not uncommon to inadvertently output them (I often do it myself). In particular, it is difficult to be aware of what is contained in the nested fields in the structure each time, and I do not want to think about it every time.

One tool that addresses this issue is an OSS called blouson. It has a function to hide the values of field names with secure_ prefix in Ruby on Rails and the corresponding values from SQL statements included in Exceptions. Since I have been developing exclusively in Go recently, I wanted to use a tool with a similar function in Go.

zlog implementation

So, I implemented zlog.

The Go language comes with a standard logging package called log and a logger interface, but this time I implemented it completely separately. The reason for this is that the concept of log level is not well known, and structured logging is not supported. In particular, when considering server-side output, it is desirable to output structured logs as much as possible, since, for example, AWS's CloudWatch Logs and GCP's Cloud Logging support logs in JSON format, and JSON schemas can be used for searching.

Some famous existing libraries that support structured logging are zap, logrus, zerolog, but none of them implement such a function to hide the secret value.

How to hide a secret value

Basically, when you input a variable (or a structure) you want to log into the With() chain method, the filter in the Filters array checks if it contains the secret value, and hides the field if it is found to contain it. The following five use cases are assumed.

Specifying a specific value

This function is designed to hide limited and predetermined secret values, such as API tokens used by the application itself to call external services. This is expected to not only hide specific fields, but also to deal with some values that are introduced by string operations such as fmt.Sprintf.

const issuedToken = "abcd1234"
authHeader := "Authorization: Bearer " + issuedToken

logger := newExampleLogger()
logger.Filters = []zlog.Filter{
    filter.Value(issuedToken),
}
logger.With("auth", authHeader).Info("send header")
// Output:  [info] send header
// "auth" => "Authorization: Bearer [filtered]"

Specify a specific field name.

This filter hides the secret value if it matches the field name of the specified structure. In addition to exact match, a forward match filter named filter.FieldPrefix is also provided. This allows you to hide the secret value only if the field name has a Secure prefix, just like blouson.

type myRecord struct {
    ID    string
    EMail string
}
record := myRecord{
    ID:    "m-mizutani",
    EMail: "[e-mail]",
}

logger.Filters = []zlog.Filter{
    filter.Field("EMail"),
}
logger.With("record", record).Info("Got record")
// Output:  [info] Got record
// "record" => zlog_test.myRecord{
//   ID:    "m-mizutani",
//   EMail: "[filtered]",
// }

Specifying a custom defining type

The Go language allows you to create your own defining types from existing types, which can be used to distinguish your usage from existing ones (e.g. key of value in context package). To use this mechanism, you can define a type you want to keep secret and specify it with a Filter to prevent it from being displayed. The advantage of this method is that copying a value from a custom type to the original type requires a cast, making it easier for the developer to notice unintentional copying. (Of course, this is not a perfect solution because you can still copy by casting.

I think this method is useful for use cases where you need to use secret values between multiple structures.

type password string
type myRecord struct {
    ID       string
    Password password
}
record := myRecord{
    ID:       "m-mizutani",
    Password: "abcd1234",
}

logger.Filters = []zlog.Filter{
    filter.Type(password("")),
}
logger.With("record", record).Info("Got record")
// Output:  [info] Got record
// "record" => zlog_test.myRecord{
//   ID:       "m-mizutani",
//   Password: "[filtered]",
// }

Specify by tag in a structure

It is possible to hide the fields of a structure with a specific tag. The key name of the tag is fixed to zlog and you can specify multiple values to hide. If nothing is specified, secret is targeted by default.

This is useful if you want to hide values without changing the current implementation, but do not want the management of target field names to be centralized in zlog.

type myRecord struct {
    ID    string
    EMail string `zlog:"secret"`
}
record := myRecord{
    ID:    "m-mizutani",
    EMail: "[e-mail]",
}

logger.Filters = []zlog.Filter{
    filter.Tag(),
}
logger.With("record", record).Info("Got record")
// Output:  [info] Got record
// "record" => zlog_test.myRecord{
//   ID:    "m-mizutani",
//   EMail: "[filtered]",
// }

Specify (what appears to be) personal information.

This is an experimental effort and not a very reliable method, but it may be of some value. It is a way to detect and hide personal information that should not be output based on a predefined pattern like many DLP (Data Leakage Protection) solutions.

In the following example, we use a filter that we wrote to detect Japanese phone numbers. The content is just a regular expression. This method does not have as many patterns as the existing DLP solutions, and the patterns are not accurate enough, but we hope to expand it in the future if necessary.

type myRecord struct {
    ID    string
    Phone string
}
record := myRecord{
    ID:    "m-mizutani",
    Phone: "090-0000-0000",
}

logger.Filters = []zlog.Filter{
    filter.PhoneNumber(),
}
logger.With("record", record).Info("Got record")
// Output:  [info] Got record
// "record" => zlog_test.myRecord{
//   ID:    "m-mizutani",
//   Phone: "[filtered]",
// }

Create your own filters

You can create your own filters using methods implemented according to the zlog.Filter interface. The two required methods are as follows

• ReplaceString(s string) string: This method is called if the value to be checked is of type string. The argument is the value to be tested and the return value is the value to be replaced. If nothing needs to be done, the argument is returned as is. It is assumed that you want to hide a part of the string.
• ShouldMask(fieldName string, value interface{}, tag string) bool: The field name (or key name if it is a map type) of the value to check, the value itself, and if the structure is tagged with zlog, the information is passed as arguments. if the structure is tagged with zlog. If the return value is false, nothing is done; if it is true, the entire field is hidden. If the return value is false, nothing is done, and if it is true, the entire field is hidden. The hidden value will be replaced with the value [filtered] if the original type is string, otherwise it will be empty.

Comparison with other implementations.

Finally, I would like to conclude with a comparison with other implementations. If you are interested in using it, I hope this will be helpful.

Advantage

• The ability to hide values in a variety of ways makes it possible to tailor the implementation to the use case.
• For example, github.com/gyozatech/noodlog provides a similar function, but it assumes conversion to JSON, and type information is lost in the output stage. Since zlog basically performs hiding while preserving the types, you can freely choose the format of the final output.
• The advantage of zlog over DLP solutions is that you don't have to remove the secret values from the stored logs.

Disadvantage

• Computation and memory usage is lower than other loggers due to deep copying of variables and structures that can be modified to display data.
  • This makes it unsuitable for services where logging has a direct impact on performance.
  • Also, it may cause a heavy load when trying to display huge data.
• The implementation is still in its infancy and does not yet fully implement the functionality of a modern logger.