DEV Community: Eugene Chernyshov

Analyzing AST in Go with JSON tools

Eugene Chernyshov — Mon, 05 Sep 2022 19:10:35 +0000

There are many specific tasks that could significantly improve and automate your ongoing maintenance of big-project. Some of them require building tools that can analyze or change source code created by developers.

For example, such tools could be:

gathering metadata from comments,
gathering strings that need to be translated,
understanding the structure of the code to calculate some complexity metrics or build explanatory diagrams,
or even apply some automatic code optimization and refactoring patterns

Solving such tasks seems to lead us to complicated topics of compilers and parsers. But in 2022 every modern programming language comes with batteries included. The structure of code in form of AST that is ready to be searched and manipulated is presented as a built-in library. Basically, parsing files with code and searching for specific things is not much harder as do the same for JSON or XML.

In this article, we will cover AST analysis in Go.

Existing approaches

In golang there is a standard package ast that provides structs of AST nodes and functions for parsing source files. It is quite easy and straightforward for experienced go developers to write code for the tool. Also, there is printer package that can convert AST back into source code.

Here is a list of articles describing how to manipulate AST in golang:

One small aspect, you need to know the structure of golang AST. For me, when I first dive into the topic, the problem was to understand how nodes are combined together and figure out what exactly I need to search in terms of node structure. Of course, you can print AST using built-in capabilities. You will get output in some strange format:

   0  *ast.File {
   1  .  Package: 1:1
   2  .  Name: *ast.Ident {
   3  .  .  NamePos: 1:9
   4  .  .  Name: "main"
   5  .  }
   6  .  Decls: []ast.Decl (len = 2) {
   7  .  .  0: *ast.GenDecl {
   8  .  .  .  TokPos: 3:1
   9  .  .  .  Tok: import
  10  .  .  .  Lparen: 3:8
....

Also, since the format is very specific, you can’t use any tools to navigate it, except text search. Tools like goast-viewer can help with this, but capabilities are limited.

Proposed solution

I started thinking of the library that would allow us to convert AST into some very conventional format like JSON. JSON is easy to manipulate, and many tools (like jq) and approaches exist to search and modify JSON.

So, what I end up with is asty

Asty is a small library written in go that allows parsing source code and presenting it in JSON structure. But, moreover, it allows also to do the reverse conversion. It means that now you can manipulate go code with a tool or algorithm developed with any programming language.

You can use it as go package, as a standalone executable, or even as a docker container. Try this page to experiment with asty in web assembly.

Example go code:

package main

import "fmt"

func main() {
    fmt.Println("hello world")
}

Example JSON output:

{
  "NodeType": "File",
  "Name": {
    "NodeType": "Ident",
    "Name": "main"
  },
  "Decls": [
    {
      "NodeType": "GenDecl",
      "Tok": "import",
      "Specs": [
        {
          "NodeType": "ImportSpec",
          "Name": null,
          "Path": {
            "NodeType": "BasicLit",
            "Kind": "STRING",
            "Value": "\"fmt\""
          }
        }
      ]
    },
    {
      "NodeType": "FuncDecl",
      "Recv": null,
      "Name": {
        "NodeType": "Ident",
        "Name": "main"
      },
      "Type": {
        "NodeType": "FuncType",
        "TypeParams": null,
        "Params": {
          "NodeType": "FieldList",
          "List": null
        },
        "Results": null
      },
      "Body": {
        "NodeType": "BlockStmt",
        "List": [
          {
            "NodeType": "ExprStmt",
            "X": {
              "NodeType": "CallExpr",
              "Fun": {
                "NodeType": "SelectorExpr",
                "X": {
                  "NodeType": "Ident",
                  "Name": "fmt"
                },
                "Sel": {
                  "NodeType": "Ident",
                  "Name": "Println"
                }
              },
              "Args": [
                {
                  "NodeType": "BasicLit",
                  "Kind": "STRING",
                  "Value": "\"hello world\""
                }
              ]
            }
          }
        ]
      }
    }
  ]
}

asty is also capable to output comments, positions of tokens in original source, and reference ids. In some places, AST of go is not actually a tree but rather a DAG. So nodes may have the same ids specified in JSON.

Development principles and constraints

In the development of asty I tried to follow some rules:

Make JSON output as close to real golang structures as possible. There is no additional logic introduced. No normalization. No reinterpretation. The only things that were introduced are the names of some enum values. Even names of fields are preserved in the same way they exist in go ast package.
Make it very explicit. No reflection. No listing of fields. This is done to facilitate future maintenance. If something will be changed in future versions of golang this code will probably break compile time. Literally, asty contains 2 copies for each AST node struct to define marshaling and unmarshaling of JSON.
Keep polymorphism in JSON structure. If some field references an expression then a particular type will be discriminated from the object type name stored in a separate field NodeType. It is tricky to achieve so if you want something like this for other tasks I would recommend checking out this example https://github.com/karaatanassov/go_polymorphic_json

Further work

I am looking for cooperation with other developers interested in language tools development. Meanwhile, you can check another repository with examples where I experiment with AST JSON in python.

Calculating Text Width at Server-Side

Eugene Chernyshov — Mon, 05 Sep 2022 00:20:52 +0000

How is text width established when content is user-generated and the information is rendered server-side? Let's take a look at text width at the server-side.

Introduction

Creators of reports and web pages are aware of the importance of readability. Text box width impacts the readability and usability of any web-based experience, but how should a developer establish text width when the content is user-generated and the information is to be delivered server-side?

Opinion on the ideal line length and text box width differs among designers and developers, and the ideal width is further complicated by the array of devices, layouts, and fonts available. Popular guidance suggests a line length of 45-85 characters, but text box width is not measured by character count alone. Font widths vary widely, and in the user-generated text, the fonts available are practically endless.

My team ran into this challenge when building a new project with a requirement to support the export of diagrams as images. The product being built provides various sophisticated views on data obtained from many sources and uses the React library for diagram rendering in SVG, which offers excellent capabilities inside the Node.js environment. However, the Node.js runtime does not offer a text renderer and the information about font properties is not easily accessible.

The project also contains many micro-services for gathering information from a range of sources and in a variety of data formats. Text can be added by the user and the output is sophisticated visualizations that power the entire product. The bottom line is we process the text but do not control it.

When preparing text width for a browser experience, the browser has capabilities built in to estimate the width of each font and can cut or split text wherever it needs to. The issue is on the server-side, and the development team needed to find or build an instrument that would estimate the font sizes.

Exploring a Solution

The simplest solution that comes to mind is to create a table with the width of each character and then the width of the text in pixels, which will be a sum of widths for all the characters it contains.

That’s it! End of story! Well, not exactly.

Characters in different fonts and font sizes all have different widths. Additional factors like font weight and font presentation, such as the angle of incline when italicized, also impact width. For this specific project, the developers needed to consider the range of character width in different languages, including Cyrillic, Arabic, Latin, Hebrew, Asian, and so forth. This created a much more complex set of requirements for the table-based solution. Also, it is for user-generated text, which will mean the potential for a wide variety of text lengths that must be optimized into the codebase.

Another requirement of the project is the ability to generate an image that could be exportable in different formats, including presentation software or word processing formats. The product analyzes data using Natural Language Processing (NLP) and Artificial Intelligence (AI), generating diagrams and charts with various filters and queries and customizable by the user.

Text Width Mapping Tool

What we ended up creating took just one weekend. In fact, it is a rather tiny (yet smart) move that really helped solve the challenge. The development team was very collaborative, working off a Kanban board and working each task through until it was complete. The first day was spent building one part of it, and the second day was spent improvising and testing.

There was one project on GitHub that was very close to what this project needed, but it lacked some of the customization and flexibility required. Other insights were pulled from suggestions on Q&A forums. As with anything, sometimes you can spend a lot of time trying to struggle with an existing instrument or fix it to make it work for you. In this case, I quickly realized we needed to develop our own solution.

We created a function for calculating text width in pixels for server-side rendering. The development built a tool to aid with table generation that would be optimized in the codebase. It started with a single HTML page with vanilla JavaScript. It was not very pretty, but it ran everywhere it needed to and it does the job. In that tool, users can select font, size, weight, presentation, and the set of characters they will use as a Unicode page. With input at the top, users can experiment with characters and test the table. If a character is missing, it will be highlighted with red. In the text area at the very bottom, the result table is generated.

In font rendering, each character’s width might be fractional, so we decided to store numbers as floats. Numbers are stored in JavaScript in a single-precision number, so to make the code more concise, Base64 was used for encoding for numbers. When encoded, each number is presented with two Base64 characters, one for the integer and one for the fractional.

In the tool, the table is generated for a single font, size, and weight. When several fonts are used in one project, the user could manage tables by developing a registry of font tables, or by merging them into one dictionary manually. For our project, we did not predefine a solution for simplicity.

As stated above, visualizations are one of the key features of the product being developed so the impact was immediate. My team was happy because we found a simple and manageable solution and did not overcomplicate it. The product owner was happy because we did exactly what they wanted according to the requirements defined. The project manager was also satisfied that the time estimates were met. Most importantly, potential customers will be happy because it all comes together beautifully inside the product.

Open Source Solution

As a result, I decided to make this an open-source project. Calculating text width could be a part of a lot of different tasks, and if another developer is seeking something like this, I would be happy to know they found it useful. Some aspects of our implementation make it unique, but it is very flexible and could be adapted to the particular needs of another project by simply adjusting the fonts, the sizes, the font weights, and other parameters.

Cover image was taken from this page