Creating Terraform-like configuration languages with HCL and Go

In the past year and a half, I've been working on Atlas, a database schema management tool that we're developing at Ariga. As part of this effort, I worked on implementing the infrastructure for the Atlas DDL, a data definition language that is the basis for Atlas's declarative style workflow for managing database schemas.

The Atlas language is based on HCL, a toolkit for creating configuration languages with a neat and simple information model and syntax. HCL was created at HashiCorp and used in popular tools such as Terraform and Nomad. We chose HCL as the basis of our configuration language for multiple reasons:

• It has a base syntax that is clear and concise, easily readable by humans and machines.
• Popularized by Terraform and other projects in the DevOps / Infrastructure-as-Code space, we thought it would feel familiar to practitioners which are the one of the core audiences for our tool.
• It's written in Go, making it super easy to integrate with the rest of our codebase at Ariga.
• It has great support for extending the basic syntax into a full-blown DSL using functions, expressions and context variables.

PCL: The Pizza Configuration Language

In the rest of this post, we will demonstrate how to create a basic configuration language using HCL and Go. To make this discussion entertaining, let's imagine that we are creating a new PaC (Pizza-as-Code) product that lets users define their pizza in simple HCL-based configuration files and send them as orders to their nearby pizza place.

Orders and Contacts

Let's start building our PaC configuration language by letting users define where they want their pizza delivered and who is the hungry contact waiting for the pizza to arrive. We're aiming for something like:

contact {
  name  = "Sherlock Holmes"
  phone = "[phone]"
}
address {
  street  = "221B Baker St"
  city    = "London"
  country = "England"
}

To capture this configuration, we will define a Go struct Order with sub-structs for capturing the Contact and Address:

type (
    Order struct {
        Contact *Contact `hcl:"contact,block"`
        Address *Address `hcl:"address,block"`
    }
    Contact struct {
        Name  string `hcl:"name"`
        Phone string `hcl:"phone"`
    }
    Address struct {
        Street  string `hcl:"street"`
        City    string `hcl:"city"`
        Country string `hcl:"country"`
    }
)

The Go HCL codebase contains two packages with a fairly high-level API for decoding HCL documents into Go structs: hclsimple (GoDoc)
and gohcl (GoDoc). Both packages rely on the user supplying Go struct field tags to map from the configuration file to the struct fields.

We will start the example by using the simpler one, with the surprising
name, hclsimple:

func TestOrder(t *testing.T) {
    var o Order
    if err := hclsimple.DecodeFile("testdata/order.hcl", nil, &o); err != nil {
        t.Fatalf("failed: %s", err)
    }
    require.EqualValues(t, Order{
        Contact: &Contact{
            Name:  "Sherlock Holmes",
            Phone: "[phone]",
        },
        Address: &Address{
            Street:  "221B Baker St",
            City:    "London",
            Country: "England",
        },
    }, o)
}

Pizza sizes and toppings (using static values)

Next, let's add the ability to order actual pizzas in our PaC application. To describe a pizza in our configuration language users should be able to do
something like:

pizza {
  size = XL
  count = 1
  toppings = [
    olives,
    feta_cheese,
    onions,
  ]
}

Notice that to make our API more explicit, users do not pass string values to the size or toppings field, and instead they use pre-defined, static identifiers (called "variables" in the HCL internal API) such as XL or feta_cheese.

To support this kind of behavior, we can pass an hcl.EvalContext (GoDoc),
which provides the variables and functions that should be used to evaluate an expression.

To construct this context we'll create this ctx() helper function:

func ctx() *hcl.EvalContext {
    vars := make(map[string]cty.Value)
    for _, size := range []string{"S", "M", "L", "XL"} {
        vars[size] = cty.StringVal(size)
    }
    for _, topping := range []string{"olives", "onion", "feta_cheese", "garlic", "tomatoe"} {
        vars[topping] = cty.StringVal(topping)
    }
    return &hcl.EvalContext{
        Variables: vars,
    }
}

To use it we need to add the pizza block to our top level Order struct:

type (
    Order struct {
        Contact *Contact `hcl:"contact,block"`
        Address *Address `hcl:"address,block"`
        Pizzas  []*Pizza `hcl:"pizza,block"`
    }
    Pizza struct {
        Size     string   `hcl:"size"`
        Count    int      `hcl:"count,optional"`
        Toppings []string `hcl:"toppings,optional"`
    }
    // ... More types ...
)

Here's our pizza block read using ctx() in action:

func TestPizza(t *testing.T) {
    var o Order
    if err := hclsimple.DecodeFile("testdata/pizza.hcl", ctx(), &o); err != nil {
        t.Fatalf("failed: %s", err)
    }
    require.EqualValues(t, Order{
        Pizzas: []*Pizza{
            {
                Size: "XL",
                Toppings: []string{
                    "olives",
                    "feta_cheese",
                    "onion",
                },
            },
        },
    }, o)
}

How many pizzas to order? (Using functions in HCL)

The final conundrum in any pizza delivery order is of course, how many pizzas to order. To help our users out with this riddle, let's level up our DSL and add the for_diners function that will take a number of diners and calculate for the user how many pizzas should be ordered. This will look something like:

pizza {
  size     = XL
  count    = for_diners(3)
  toppings = [
    tomato
  ]
}

Based on the universally accepted heuristic that one should order 3 slices per diner and round up, we can register the following function into our EvalContext:

func ctx() *hcl.EvalContext {
    // .. Variables ..

    // Define a the "for_diners" function.
    spec := &function.Spec{
        // Return a number.
        Type: function.StaticReturnType(cty.Number),
        // Accept a single input parameter, "diners", that is not-null number.
        Params: []function.Parameter{
            {Name: "diners", Type: cty.Number, AllowNull: false},
        },
        // The function implementation.
        Impl: func (args []cty.Value, _ cty.Type) (cty.Value, error) {
            d := args[0].AsBigFloat()
            if !d.IsInt() {
                return cty.NilVal, fmt.Errorf("expected int got %q", d)
            }
            di, _ := d.Int64()
            neededSlices := di * 3
            return cty.NumberFloatVal(math.Ceil(float64(neededSlices) / 8)), nil
        },
    }
    return &hcl.EvalContext{
        Variables: vars,
        Functions: map[string]function.Function{
          "for_diners": function.New(spec),
        },
    }
}

Testing the for_diners function out:

func TestDiners(t *testing.T) {
    var o Order
    if err := hclsimple.DecodeFile("testdata/diners.hcl", ctx(), &o); err != nil {
      t.Fatalf("failed: %s", err)
    }
    // For 3 diners, we expect 2 pizzas to be ordered.
    require.EqualValues(t, 2, o.Pizzas[0].Count)
}

Wrapping up

With these features, I think we can call it a day for this prototype of the world's first Pizza-as-Code product. As the source code for these examples is available on GitHub under an Apache 2.0
license, I truly hope someone picks this up and builds this thing!

In this post we reviewed some basic things you can do to create a configuration language for your users using HCL. There's a lot of other cool features we built into the Atlas language (such as input variables, block referencing and block polymorphism), so if you're interested so if you're interested in reading more about it feel free to ping me on Twitter.

Comments

[XMachli]

import org.apache.poi.ss.usermodel.;
import org.apache.poi.xssf.usermodel.XSSFWorkbook;
import org.apache.poi.ss.usermodel.CellType;
import java.io.;
import java.util.*;
import java.util.stream.Collectors;

public class ExcelFileMerger {
public static void main(String[] args) {
String inputFolder = "path/to/input/folder";
String outputParentFolder = "path/to/output/parent"; // Specify the parent folder for merged files

    // Find all Excel files in the input folder and its subfolders
    List<File> excelFiles = findExcelFiles(inputFolder);

    // Create the output folder for merged files
    String outputFolder = outputParentFolder + File.separator + "MergedOutput";
    File outputDirectory = new File(outputFolder);
    outputDirectory.mkdirs(); // Create the "MergedOutput" directory if it doesn't exist

    // Create subfolders for .xlsx and .csv files
    String xlsxSubfolder = outputFolder + File.separator + "XLSX";
    String csvSubfolder = outputFolder + File.separator + "CSV";
    File xlsxDirectory = new File(xlsxSubfolder);
    File csvDirectory = new File(csvSubfolder);
    xlsxDirectory.mkdirs();
    csvDirectory.mkdirs();

    // Merge Excel files
    Map<String, List<File>> groupedFiles = groupFilesByName(excelFiles);
    for (Map.Entry<String, ListFile>> entry : groupedFiles.entrySet()) {
        String outputFileName = entry.getKey();
        String xlsxFilePath = xlsxSubfolder + File.separator + outputFileName;
        String csvFilePath = csvSubfolder + File.separator + outputFileName.replace(".xlsx", ".csv");

        mergeExcelFiles(entry.getValue(), xlsxFilePath);
        convertXlsxToCsv(xlsxFilePath, csvFilePath);
    }
}

private static List<File> findExcelFiles(String folderPath) {
    List<File> excelFiles = new ArrayList<>();
    File inputFolder = new File(folderPath);
    File[] files = inputFolder.listFiles();

    if (files != null) {
        for (File file : files) {
            if (file.isDirectory()) {
                excelFiles.addAll(findExcelFiles(file.getPath()));
            } else if (file.getName().endsWith(".xlsx")) {
                excelFiles.add(file);
            }
        }
    }
    return excelFiles;
}

private static Map<String, ListFile>> groupFilesByName(List<File> files) {
    Map<String, ListFile>> groupedFiles = new HashMap<>();

    for (File file : files) {
        String fileName = file.getName();

        if (groupedFiles.containsKey(fileName)) {
            groupedFiles.get(fileName).add(file);
        } else {
            ListFile> fileList = new ArrayList<>();
            fileList.add(file);
            groupedFiles.put(fileName, fileList);
        }
    }

    return groupedFiles;
}

private static void mergeExcelFiles(List<File> files, String outputFolder) {
    try {
        Workbook mergedWorkbook = new XSSFWorkbook();
        for (File file : files) {
            FileInputStream fileInputStream = new FileInputStream(file);
            Workbook workbook = new XSSFWorkbook(fileInputStream);

            for (int i = 0; i < workbook.getNumberOfSheets(); i++) {
                Sheet sheet = workbook.getSheetAt(i);
                String sheetName = sheet.getSheetName();
                Sheet mergedSheet = mergedWorkbook.getSheet(sheetName);

                if (mergedSheet == null) {
                    mergedSheet = mergedWorkbook.createSheet(sheetName);
                }

                for (int j = 0; j < sheet.getPhysicalNumberOfRows(); j++) {
                    Row row = sheet.getRow(j);
                    if (row != null) { // Check if the row is not null
                        Row mergedRow = mergedSheet.createRow(mergedSheet.getLastRowNum() + 1);

                        for (int k = 0; k < row.getPhysicalNumberOfCells(); k++) {
                            Cell cell = row.getCell(k);
                            Cell mergedCell = mergedRow.createCell(k);

                            if (cell != null) {
                                CellType cellType = cell.getCellType();
                                switch (cellType) {
                                    case STRING:
                                        mergedCell.setCellValue(cell.getStringCellValue());
                                        break;
                                    case NUMERIC:
                                        if (DateUtil.isCellDateFormatted(cell)) {
                                            mergedCell.setCellValue(cell.getDateCellValue());
                                        } else {
                                            mergedCell.setCellValue(cell.getNumericCellValue());
                                        }
                                        break;
                                    // Handle other cell types as needed
                                    default:
                                        // Handle unknown or unsupported cell types
                                        break;
                                }
                            }
                        }
                    }
                }
            }
            fileInputStream.close();
        }

        // Save the merged workbook as .xlsx
        File outputFile = new File(outputFolder);
        FileOutputStream fileOutputStream = new FileOutputStream(outputFile);
        mergedWorkbook.write(fileOutputStream);
        fileOutputStream.close();
    } catch (Exception e) {
        e.printStackTrace();
    }
}

private static void convertXlsxToCsv(String xlsxFile, String csvFile) {
    try {
        FileInputStream xlsxInputStream = new FileInputStream(xlsxFile);
        Workbook workbook = new XSSFWorkbook(xlsxInputStream);
        Sheet sheet = workbook.getSheetAt(0); // Assuming you want to convert the first sheet

        FileWriter csvWriter = new FileWriter(csvFile);

        for (int i = 0; i < sheet.getPhysicalNumberOfRows(); i++) {
            Row row = sheet.getRow(i);
            for (int j = 0; j < row.getPhysicalNumberOfCells(); j++) {
                Cell cell = row.getCell(j);
                if (j > 0) {
                    csvWriter.append(',');
                }
                if (cell != null) {
                    csvWriter.append(cell.toString());
                }
            }
            csvWriter.append('\n');
        }

        csvWriter.flush();
        csvWriter.close();
        xlsxInputStream.close();
    } catch (Exception e) {
        e.printStackTrace();
    }
}

}