AWS CloudWatch Contributor Insights. Is it useful?

Check logs with Contributor Insights

Right! So we have logs. We can use CloudWatch Logs Insights to query it, we can observe it through CloudWatch logs, and we can do many, many things with them now. Finally, we have them!

AWS gives us another tool, called Contributor Insights. This service can be used to analyze logs, create time series and display this data. That's quite powerful!

Go to CloudWatch service, Contributor insights. Click Create rule. In the dropdown Log group(s) select our API logs. For the first time, select Sample rule, API Gateway and Calls by route and HTTP method (CLF). In fact, it doesn't matter if you select JSON or CLF (in this case).

Keep rest of configuration as it is.

Log format: CLF (or JSON if you selected it earlier)  
Contribution: ip  
Aggregation: Count

Click Next and give the rule name. I named mine as APIGW_byCount.

After you click Create rule, you have to wait a few minutes (AWS claims 5) to see any data.

This will build the analysis and aggregation. On the screen you see four different IPs (are they IPs, really? We'll see).

The example above is autogenerated.

Let's see, if we can to create custom one.

Go to editor, click Syntax to change the screen to text editor.

Our example looks like this:

{
    "Schema": {
        "Name": "CloudWatchLogRule",
        "Version": 1
    },
    "LogGroupNames": [
        "API-Gateway-Execution-Logs_9fcv1s573l/Prod"
    ],
    "LogFormat": "JSON",
    "Contribution": {
        "Keys": [
            "$.ip"
        ],
        "Filters": []
    },
    "AggregateOn": "Count"
}

Here you can start playing with your query.

Let's visualise what we queried

It is time to visualise our work. Click Actions, View in CloudWatch Metrics and select Unique contributors.

In time I received a few more requests from different IPs, and below you can see the visualisation.

Let's make it useful

Well... What we see now is... not really useful, right? Right. This visualisation is not really about IPs. It is some mix of many elements. You know what? Forget this. Let's do it right.

What we need to do, is work on the samples provided by AWS and rewrite them, or create our own. Or, and this is what we will do, we will rewrite the logs which API is generating. Well, we should do it, anyway.

Ok. Let's go to the API service, select the proper API, again, got to Stages, Prod and select the Logs/Tracing tab.

Click Enable Access Logging, and use Insert Example for JSON. You will have default log (which actually is different than default default log ;) ). Let's add one more field, like in example below.

{ "requestId":"$context.requestId", "ip": "$context.identity.sourceIp", "caller":"$context.identity.caller", "user":"$context.identity.user","requestTime":"$context.requestTime", "httpMethod":"$context.httpMethod","resourcePath":"$context.resourcePath", "status":"$context.status","protocol":"$context.protocol", "responseLength":"$context.responseLength","traceId":"$context.xrayTraceId" }

On the end of the string you can find traceId field.

I also slightly modified the Log Destination ARN in order to differentiate the behavior between the logs. It will be more readable for you.

Ok, now let's be sure, that the syntax for rule is like this:

{
    "AggregateOn": "Count",
    "Contribution": {
        "Filters": [],
        "Keys": [
            "$.ip"
        ]
    },
    "LogFormat": "JSON",
    "LogGroupNames": [
        "API-Gateway-Execution-Logs_9fcv1s573l"
    ],
    "Schema": {
        "Name": "CloudWatchLogRule",
        "Version": 1
    }
}

Be also sure, you use the proper (new one!) log group (remember, yours will be different than mine!)

First of all, you can see totally different log. Shorter, more clearly structurized.

And how it looks in Contributor Insights? Well, great!

SAM template

It is time to modify the template.

AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31
Description: simple Lambda

Resources:
  accessLogGroup:
    Type: AWS::Logs::LogGroup

  lambdaDemoApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: Prod
      Description: 'Prod stage'
      TracingEnabled: true
      MethodSettings:
      - HttpMethod: '*'
        LoggingLevel: INFO
        ResourcePath: '/*'
        MetricsEnabled: true
        DataTraceEnabled: true
      AccessLogSetting:
        DestinationArn: !GetAtt accessLogGroup.Arn
        Format: >-
          '{ "requestId":"$context.requestId", 
          "ip": "$context.identity.sourceIp", 
          "caller":"$context.identity.caller", 
          "user":"$context.identity.user", 
          "requestTime":"$context.requestTime", 
          "httpMethod":"$context.httpMethod", 
          "resourcePath":"$context.resourcePath", 
          "status":"$context.status", 
          "protocol":"$context.protocol", 
          "responseLength":"$context.responseLength", 
          "traceId":"$context.xrayTraceId" }'

  lambdaFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: simplefunction.handler
      CodeUri: lambdafunction/
      Runtime: python3.8
      Policies:
        - CloudWatchLambdaInsightsExecutionRolePolicy
      AutoPublishAlias: live
      Description: Simple demo function
      MemorySize: 128
      Timeout: 10
      Tracing: Active
      Layers:
        - !Sub "arn:aws:lambda:${AWS::Region}:580247275435:layer:LambdaInsightsExtension:14"
      Events:
        simpleApi:
          Type: Api
          Properties:
            RestApiId: 
              !Ref lambdaDemoApi
            Path: /
            Method: get

As you can see, I added one resource - AccessLogGroup and AccessLogSettings in Properties of lambdaDemoApi.

---

Now you can go crazy. There is many possibilities to build API Logs and to visualize it. Now you know :)

---

Cover image by Hebi B. from Pixabay