DEV Community: mnlth

Linting OpenAPI specs using Spectral

mnlth — Sun, 09 May 2021 14:20:58 +0000

Introduction

This post details on linting swagger specification using spectral. When it comes to following standards, its always conversational and prone to deviations with human aspect. How good would it be to automate the verification of standards.

Its such a relief to hear about a positive move towards lint option on json and yaml variant of API specs. Yes, Spectral provides lint'n options to json or yaml objects. Unlike regular linters which checks for syntax, spectral provides an option to create your own ruleset to apply on the openapi swagger specification file.

Spectral comes with built-in functions with provision to create custom functions. These custom functions can perform pattern checks, parameter checks, provided keys in an object and many more.

Usage

There are a few options available for installation and usage based on requirements. Lets look into these

node

    npm install -g @stoplight/spectral

yarn

    yarn global add @stoplight/spectral

run as binaries

    curl -L 
https://raw.githack.com/stoplightio/spectral/master/scripts/install.sh | sh

docker

Running docker image locally requires file to be copied on to volume mount

    docker run --rm -it -v $(pwd):/tmp stoplight/spectral lint "/tmp/file.yaml"

If you want to run docker from CI, use docker command

    docker run --rm -it stoplight/spectral lint "${url}"

Example:

Let me show simple usage of running docker locally on employee-openapi-spec.yaml using the command

docker run --rm -it -v $(pwd):/tmp stoplight/spectral lint "/tmp/employee-openapi-spec.yaml

openapi: 3.0.1
info:
  title: Employees API Spec
  description: Working with employee details
  version: v1
servers:
- url: http://localhost:8080/employees
  description: The employee api
tags:
- name: employee details api spec
paths:
  /api/v1/employees:
    summary: Employees
    description: API to work with employee details
    get:
      tags:
      - employees
      summary: Get list of employees
      description: Retrieve the list of employees and their details
      operationId: getEmployees
      responses:
        "200":
          description: request to get list of employees
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/EmployeeResponse'

And the output looks like this

OpenAPI 3.x detected

/tmp/employee-openapi-spec.yaml
   2:6   warning  info-contact                   Info object should contain `contact` object.
  17:9   warning  operation-tag-defined          Operation tags should be defined in global tags.

✖ 2 problems (0 errors, 2 warnings, 0 infos, 0 hints)

Features

With the ability to add linter enables to detect issues with API early on, as early as design phase. Isn't this an amazing benefit to shift left on design issues and save time for more precious work later.

Spectral has three key building blocks:

Rulesets - act as a container for rules and functions.
Rules - filters the object to a set of target values and specifies the function that is used to evaluate these values.
Functions - accept a value and return any issues with its value.

As mentioned earlier, spectral comes with basic set of functions and a predefined set of style guides for openapi specs. Rules are created using one or more of these basic in-built functions to add onto the style guides. These additional ones could be organisational guardrails which can be centrally managed and enforced onto all the APIs being designed. These could be anything from

HTTP Basic is not allowed at this company

to checking

Are all operations secured with a security schema

Working with Rulesets

Let's dive into exploring rulesets and working with custom functions. Since spectral supports linting of json and yaml objects, rulesets can also be in either of the formats. Often the ruleset filename goes like .spectral.yaml which consists of two parts, rules and functions.

In-built Spectral "oas" ruleset, OAS being shorthand for the OpenAPI Specification. In the above example, we saw a warning on info-contact. This is one of the default ruleset of oas.

Creating any custom rules requires .spectral.yaml file in current working directory. Here's how a simple custom rule looks like.

Simple ruleset example

rules:
  my-rule-one:
    description: Tags must have a description.
    given: $.tags[*]
    severity: error
    then:
      field: description
      function: truthy

The rule is applied on the json element tags defined under given and the function is truthy with reporting severity as error. This means, if any of the specification tags doesn't contain description, linter will throw error.

Spectral cli command to apply the rule:

spectral lint employee-openapi-spec.yaml

. This command will now use custom rules file instead of default spectral ruleset.

Extending ruleset

extends: spectral:oas

Just by adding extends statement to the spec, you can import other default rulesets. So when the above created custom rule combined with default ruleset looks like:

extends: spectral:oas

rules:
  my-rule-one:
    description: Tags must have a description.
    given: $.tags[*]
    severity: error
    then:
      field: description
      function: truthy

This is essentially to instruct spectral to apply default rules plus the additional custom rule.

Custom ruleset

extends: spectral:oas
rules:
  operation-2xx-response: warn

Change the severity - update the default severity of rule by overwriting it. Different severity levels that can be used are error, warn, info, hint, and off.

Turn off a rule

extends: [[spectral:oas, all]]
rules:
  my-rule-one: off

Here all the default spectral rules are enabled and only custom rule my-rule-one is disabled.

Turn on a rule

extends: [[spectral:oas, off]]
rules:
  my-rule-one: true

This is quite opposite to disabling a rule. Default rules are all disabled using off flag and only my-rule-one is enabled.

Conclusion

There are a few more options to customise the rules to cover most of the organisational requirements. Also, it is a good start to see a working lint option available for swagger which removes the overhead of manually verifying and controlling the common design pattern for APIs. I'm pretty impressed and suggest trying this to see the benefit of automating early design flaw detection. Hope you find this useful.

References

Spectral documentation
OpenAPI Rules

Functionbeat for Streaming Cloudwatch Events to ElasticSearch

mnlth — Mon, 11 May 2020 02:45:03 +0000

Photo by Bjørn Christian Tørrissen

With various serverless and opensource options available, it's easier to spin up a concept into some kind of working model with comparatively quicker than earlier times. With this advancement it becomes essential to be able to effectively monitoring the different components and solution.
What makes monitoring easier than having single logcentral to be able to get a view of required metrics.

Functionbeat for Monitoring

Yes, as an engineer, I would like to setup a dashboard with all required information. And functionbeat helps in achieving the same with as simple as writing a lambda to collect the logs from different sources. Functionbeat is one of Elastic's beat family allowing you to be able to stream logs from Kinesis, SQS, Cloudwatch (as of today) to single logcentral.

Collect the Cloudwatch Logs

What we are focusing here is, functionbeat to read each row of cloudwatch logs and stream it to elasticsearch. With functionbeat deployed as serverless lambda to AWS, you should be able to achieve above.

In this post, we focus on collecting logs from cloudwatch group and shipping it to elasticsearch.
lambda -> Cloudwatch Logs -> Functionbeat -> Elasticsearch

Setup a lambda with cloudwatch monitoring

A lambda function with any business function generating logs and metrics to cloudwatch.

Setup Functionbeat lambda

Download functionbeat distribution
https://www.elastic.co/guide/en/beats/functionbeat/current/functionbeat-installation.html
Before deploying cloudwatch, ensure you have AWS profile set-up. Verify that the aws account and region are set correctly using
aws configure
Create S3 bucket from AWS console for the functionbeat to be uploaded to. The reference to created S3 bucket is provided as

  functionbeat.provider.aws.endpoint: "s3.amazonaws.com"
  functionbeat.provider.aws.deploy_bucket: "functionbeat-deploy"

4. For the functionbeat to be deployed, lambda should have required permissions to the AWS resources. Create IAM role with access permissions to
ec2, es, s3
For detailed IAM cloudformation code refer to sample

5. Configure functionbeat in functionbeat.yaml and setup lambda to add triggers to cloudwatch group defined in #1. functionbeat-cloudwatch is the name of lambda function and type representing the trigger type with actual values listed under triggers configuration. We could configure multiple cloudwatch log group triggers as comma separated values.

  functionbeat.provider.aws.functions:
    - name: fb-cloudwatch
      enabled: true
      type: cloudwatch_logs

      role: arn:aws:iam::xxawsaccountidxx:role/lambda.functionbeat.role

      triggers:
      - log_group_name: /aws/lambda/sample-lambda-events

If you would want to deploy the lambda as part of private cloud set-up, look at configuring virtual_private_cloud with subnet details.

Now, lets look at configure the output of these logs to be shipped to. It only requires to configure the elasticsearch host and port to be configured. Provided the AWS connectivity to elasticsearch is already established, hook into the host. Else, use the local elasticsearch distribution to verify.

  output.elasticsearch:
    hosts: ["localhost:9200"]
    protocol: http

If the elasticsearch setup requires credentials, the same can be configured under output.elasticsearch with username and password. To verify the setup so far, we could simply run the command

./functionbeat test config -e

Finally, we are ready to deploy functionbeat to AWS. Deploy command uploads the functionbeat to s3 bucket functionbeat-deploy created as part of #3. Use the command:

./functionbeat -v -e -d "*" deploy fb-cloudwatch

This would create cloudformation stack in AWS creating required lambda and resources. Once successfully deployed, you should be able to navigate to cloudwatch and look at the functionbeat fb-cloudwatch logs. You could enable debug logs on functionbeat to be able to view detailed logs, add config logging.level: debug to functionbeat.yaml.

With any additional changes, you can update the functionbeat using update command without requiring to recreate the whole function.
./functionbeat -v -e -d "*" update fb-cloudwatch

Improvements

As part of the lambda function collecting and streaming logs, we could pick the required fields from log source. This could be configured as processors to the function. Say, if its required to drop field noMuchUse from log group and enrich with host metadata, set up processor

  processors:
    - include_fields:
      fields: ["id"]
    - drop_fields: 
      fields: ["id.noMuchUse"]
    - add_host_metadata:
      netinfo.enabled: false

References:
Functionbeat
Github