ApiBlaze: Parsing and Searching Elements in OpenApi Specs

ApiBlaze is a tool to explore API specifications: Search for a keyword, filter for objects, properties, or endpoints, and immediately see descriptions and code examples. ApiBlaze helps you to answer a specific question about an API lightning fast. You can try it here: apiblaze.admantium.com.

The format of an OpenAPI spec is a verbose, metadata rich JSON structure. Searching in this structure is not ideal, especially when a long list of object properties need to be traversed, or when the response or request objects of API endpoints need to be searched. Therefore, ApiBlaze generates an internal data structure that makes it easier for parsing.

In this blog post, I cover the details of building this structure, fulfilling this core requirement of ApiBlaze:

• SEL01 - Distinguish objects, properties and endpoints
• SEL02 - Search for API Elements by keywords

This article originally appeared at my blog.

Challenges when Parsing OpenAPI Spec

An OpenAPI spec is a JSON or YAML file. It contains several information, starting with the general description, metadata about the swagger version, and security information about how to access the API (typically a bearer token). We need to address two challenges: References to data structures, and nested data structures.

The information that is relevant for ApiBlaze are the paths (API endpoints) and the definitions (data structures for requests and responses). And here, OpenAPI allows the definition and usage of references. For example, you have a standard response object for all API endpoints. Instead of defining this data structure at all endpoints, you define it once in the definitions section, and then include a reference from the endpoint to this definition.

Lets consider an example from the Kubernetes spec for a pod. A pod has the spec property, which is defined as a $ref to io.k8s.api.core.v1.PodSpec. And this object has the property affinity, which is again a Sref to io.k8s.api.core.v1.Affinity. And so on. These references need to be resolved.

"io.k8s.api.core.v1.Pod": {
  "properties": {
    "spec": {
      "$ref": "#/definitions/io.k8s.api.core.v1.PodSpec",
      "description": "Specification of the desired behavior of the pod. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status"
    },
    // ...
  }
}

 "io.k8s.api.core.v1.PodSpec": {
    "description": "PodSpec is a description of a pod.",
    "properties": {
      "affinity": {
        "$ref": "#/definitions/io.k8s.api.core.v1.Affinity",
        "description": "If specified, the pod's scheduling constraints"
      },
      // ...
    }
  }

  "io.k8s.api.core.v1.Affinity": {
    "description": "Affinity is a group of affinity scheduling rules.",
    "properties": {
      "nodeAffinity": {
        "$ref": "#/definitions/io.k8s.api.core.v1.NodeAffinity",
        "description": "Describes node affinity scheduling rules for the pod."
      },
      // ...
    }
  }

The second challenge are nested data structures, Once the references are resolved, we will have a nested object. If we want to access the Pod Affinity from the example above, we would need the path ["io.k8s.api.core.v1.Pod"].properties.spec.properties.affinity.properties.nodeAffinity. This should be simplified to ["io.k8s.api.core.v1.Pod"].spec.affinity.nodeAffinity.

Let’s see how to solve these two challenges.

Resolving References

An OpenAPI specification is a YAML or JSON file. For parsing this structure, we use the library OpenPAI swagger parser. This library provides the helpful function dereference to substitute all refs in a spec. Here is an example how to use the library:

async function loadApiModel (specFile) {
  const dereferencedSpec = await SwaggerParser.dereference(specFile)

  return {
    definitions: dereferencedSpec.definitions,
    endpoints: dereferencedSpec.paths
  }
}

const apiSpec = await loadApiModel('./k8s.json')

This function returns the object { definitions: {}, endpoints: {}}, on which we continue with searching.

Searching

Searching is done with these steps:

1. Check the passed keyword(s)
2. Create a regular expression
3. Find matching occurrences of this keywords in objects, properties and endpoints
4. Sort all results by their score

Here is an excerpt of the search function:

search(keyword) {
  keyword.match(/ /) ? keyword = keyword.split(' ').map(tag => `(${tag})`).join('|') : ''

  return [
    ...this._search(this.indexedObjects, keyword),
    ...this._search(this.indexedProperties, keyword),
    ...this._search(this.indexedEndpoints, keyword)
  ].sort((a,b) => b.score - a.score)
}

_search(list, keyword) {
  return [
    ...list
    .filter(i => JSON.stringify(i).match(new RegExp(keyword), 'ig'))
    .map(i => Object.assign(i, {score: JSON.stringify(i).match(new RegExp(keyword, 'ig')).length}))
  ]
}

Resolving nested Structures

The nested data structure is not problem when searching. But once the search results are accessible in the UI, users can select individual entries to see a copy-able representation of the code.

Transforming the nested structure to a simpler one is handled by the drillDown function. It recursively calls itself to resolve all the properties key and replace them with their children.

_drillDownObject(item) {
    item.properties = this._drillDownProperties({}, this.definitions[item.containingObject].properties)
    return item
  }

  _drillDownProperties (stack, properties) {
    if (!properties || !Object.entries(properties)) {
      return stack;
    }
    for (let [key, values] of Object.entries(properties)) {
      switch (values.type) {
        case 'string':
          stack[key] = { _type: 'string', _description: values.description};
          break;
        case 'object':
          stack[key] = { _type: 'object', _description: values.description, ...this._drillDownProperties({}, values.properties) };
          break;
        case 'array':
          stack[key] = { _type: 'array', _description: values.description, ...this._drillDownProperties({}, values.items.properties) };
          break;
      }
    }
    return stack;
  }

This results in this structure:

{
  name: "Pod",
  containingObject: "io.k8s.api.core.v1.Pod"
  properties: {
    affinity: {
      nodeAffinity: {
        type: "string",
        description: "Describes node affinity scheduling rules for the pod."
      }
    }
  }
}

Review: ApiBlaze Project Requirements

With the changes of this article, we obtain the following status of ApiBlaze's requirements:

• Searching for APIS
  • ✅ SEA01 - Search for APIs by Keyword
  • ✅ SEA02 - Show search results in a popup
  • ✅ SEA03 - Select a search results with arrow keys, enter and mouse click
• Searching API Elements
  • ✅ SEL01 - Distinguish objects, properties and endpoints
  • ✅ SEL02 - Search for API Elements by keywords
• Framework
  • ✅ FRAME01 - Controller & Routing
  • ✅ FRAME02 – Stateful Pages & Components
  • ✅ FRAME03 - Actions
  • ✅ FRAME04 – Optimized Bundling
• Technologies
  • ✅ TECH01 - Use PlainJS & Custom Framework
  • ✅ TECH02 - Use SAAS for CSS
  • ✅ TECH03 - Use WebSockets to Connect Frontend and Backend

Conclusion

OpenAPI specifications are structured into sections for definitions (objects and properties) and paths (endpoints). Once Inside these definitions, references are used to point to commonly recurring structures, preventing repetition. These references need to be resolved in order to obtain a simple, parse able representation of the OpenAPI specification. For this challenge, we use the OpenAPI swagger parser. The resulting structure can be searched. Searching means to look for the entered keywords, count the occurrences, and rank results according to their score. When a user selects a result, we also want to show a copy-able representation of the code. However, traversing nested objects is tedious because of long objects paths. And therefore, a custom function processes this structure to design a simpler to traverse structure from which the code representation is created.