Why I built a cross-language and cross-repo constant sharer, and how you can use it

What's the background?

Last year, I did a placement year as part of my degree. While I was there, my team and I were building a brand new project made up of tonnes of little microservices, most of which were built in Python. So that we could share constants cross-service, my team developed a 'common' pip package, which declared these constants and could then be imported into other projects.

Although this library worked, microservices programmed in other languages such as JavaScript couldn't use this library, so they had their constants hardcoded in. This meant that when constants needed to be changed, they needed to be changed in a lot of places and tested in a lot of places.

What did I make?

I sat down the summer after I finished this placement, and decided to make a new tool which would provide the benefits that the original 'common' library was meant to, while also allowing people to share constants cross-language.

To do this, I created conshare. This is a CLI tool, written in Golang, which takes as parameters two files: a constant declaration file and a configuration file. From this, it generates language specific files (as declared), which define the constants as requested in the file, which can be referenced in different projects.

Constant declaration file

This file declares the name of the constants that need to be shared cross-language, their types (so int, string, etc...), as well as their value.

The constant names or types which are declared in the file are not tied to a specific language. This means that the names and types are dynamically formatted to suit the language standard when generated. Take the following example:

- name: "age"
  type: "int"
  value: 5

This would be translated to the following in Golang:

const Age = 5

and to the following in Python:

AGE = 5

This tool would work perfectly in a monorepo, but maybe you don't use a monorepo structure, or you want to share constants cross-repo. To support this, the constant declaration file allows you to include a download tag, with a value of an address where another constant file can be found. When the tool is used, the file defined in the download tag will be downloaded and the constants declared in it appended onto the constants declared in the file referencing it. This means that a constant file can be hosted elsewhere, referenced in a different constant file, and extended for use in a specific project. This effectively means that you have some form of inheritance in the constant files.

Put into practice, this means that a base constant file can be hosted in some remote Git repository, and shared cross-repo.

The diagram below shows an example hierarchy.

constant-file-1.yml
│   
│       
│
└───constant-file-2.yml
    │   
    │   
    │
    └───constant-file-3.yml
    │         
    │         
    │        
    └───constant-file-4.yml 

Configuration file

This file declares the directories of the projects in your repo, as well as the language which those projects are written in. This is so that the tool knows where to place the generated constant files and what language they should be written in. The generated files are placed in a conshare package within the directories you specify, and are importable in your project.

If you use a monorepo, this is where the tool comes in really handy. If you don't use a monorepo, then this file will only be one project long, but it will still work just as well.

Encryption

Sometimes, you'll want to share constants which are sensitive. Maybe you've got some passwords which never change which you want to share. Problem is, you can't store them in source control in plain text. That can be solved with conshare! Simply encrypt your constant declaration file with AES-256 CBC, and provide an additional file to the CLI with the decryption key. The tool will decrypt your constants and work from there.

Alternatives

Of course the question could be asked: why not just use some key-value store like Redis? Good question! Using some tool like that has some associated cost with it, as well as the need to maintain and secure it.

conshare was designed to be a lightweight tool which does not require some backing infrastructure and has no associated cost.

What did I learn?

In doing this project, I learnt Golang more or less from scratch. It was nice to be able to not only learn a new language, but to apply what I was learning to an interesting project from the start. That was, of course, after the obligatory 'Hello, world!' task! I'm not for one second saying that my code is perfect, this is very much my first Golang task, however I'm glad to be have been able to create a tangible product.

I also built my CI pipeline from scratch when I did this. While I had some experience with modifying them when I was on placement, I hadn't built them from scratch or dealt with injecting secrets into them. It was nice to be able to learn how to do that, see my project build (when it validated that I had good code of course!) and be deployed on GitHub (releases for the tool itself, pages for the docs).

Future improvements

Testing! I have to be honest, I made no unit tests. That's very bad, and I know that, so I plan on fixing that in the near future. Unfortunately I didn't really have much time to develop this tool for a few reasons, so I valued functionality over testing just so that I could prove to myself that I could get these features implemented. This past academic year has also been filled to the brim with work, but seeing as I graduate this summer, I plan on using some time this summer to develop unit tests.

I'd also like to develop an IntelliJ plugin, so that when you're referencing conshare generated files it knows and doesn't nag you that you're using files which don't exist. Obviously, those files wouldn't have been generated yet, but it could read the constant declaration and config files to determine what constants would be generated and not complain at you if you reference them.

Any more info?

You can check out the repo at this link, or the docs at this link