Generate Files for gRPC in Node.js

When I started working with gRPC in Node.js, I followed Google's dynamically generated code base to create my servers and clients. And for basic development, this works really well. At least in a mono repository using relative paths to import the proto files. As we move closer to deployment though, a glaring problem has emerged.

A growing collection of proto files - many of which are sharing messages and models - that need to be imported into various gRPC servers and clients. These servers and clients will be hosted in various Kubernetes clusters.

My solution:

Create an npm package for each service cluster to house the needed proto files.

At first, I thought I would just write a bash script to copy the needed service and message/model files into an npm package and call it a day. However, I know that you can generate actual server and client from the proto files (in some languages), and I want to be ready for that when it comes to Node.js. Therefore, I decided that my npm packages would hold protoc generated files.

To do this, I needed to have protoc and grpc-tools installed on my development machine. (Note: grpc-tools needs to be installed globally for this to work).

I decided to make a bash script file to handle the actual generation. I figure we can use/modify this file later to use in a more automated process.

1. I remove any existing files. For Node.js, protoc creates two files, both of which end in _pb.js. So I used a wildcard to select and delete all files ending that way.

rm -f <path to target>/*_pb.js

1. Then I actually generate the files:

grpc_tools_node_protoc --js_out=import_style=commonjs,binary:<path to export directory> --grpc_out=generate_package_definition:<path to export directory> <proto file>

Note: I found that the script needed to change directories and run in the directory housing the proto file. I was not able to figure out how to use relative paths to that directory, though I do believe there is a way to do it.

The --grpc_out= flag supports two options:

• grpc_js: Generates code with require('@grpc/grpc-js') instead of require('grpc')
• generate_package_definition: Generates code that does not require any gRPC library, and instead generates PackageDefinition objects that can be passed to the loadPackageDefinition function provided by both the grpc and @grpc/grpc-js libraries.

I tried both flags and found that the generate_package_definition worked best in both my servers and clients.

My export directory is the npm package I had created for the cluster that needed the proto file. I am now able to use the package - and therefore the files - in both my servers and clients.

I am very pleased with the results. The script files does need to be manually run with any proto file changes, but I can automate that at a future date.