DEV Community: sdcaulley

Node.js - Streams and Promises

sdcaulley — Sun, 03 Apr 2022 13:53:56 +0000

I have been working on a project that requires reading large .csv files from the local file system and then working with the data. Node.js has some great tools for working with this, namely streams, event emitters, the readline native modules. However, all of the example code/tutorials fell into one of three categories:

print the data to the console (not useful)
write the data to a file
push the incoming data to an outside array
use an external library

I started with using the external library csv-parser. However, since it is basically a wrapper around the base Node.js technologies I listed above I has the same problems working with my data that I will list below. I eventually uninstalled it and wrote my own light-weight version.

Background

The `readline` module provides an interface for reading data from a Readable stream...one line at a time. from [Node.js Documentaion](https://nodejs.org/docs/latest-v16.x/api/readline.html)

All streams are instances of EventEmitter. from [Node.js Documentaion](https://nodejs.org/docs/latest-v16.x/api/stream.html)

Basically working with streams means listening for events with your data. And since the .on method of an EventEmitter expects a callback, everything you want to do next needs to happen in that callback. The readline module gives you the line event to listen for.

Solution #1

At first I tried the "push the incoming data to an outside array" approach.

const incomingData = [];

rl.on('line', data => [
  incomingData.push(data);
])
  .on('close', () => {
    // do something with incomingData
  });

This solution does actually work if you are only reading one file. Unfortunately, I need to loop through a directory of files and read each one, and then do something with the data. I tired all sorts of things with counters and what not, but kept running into race conditions with the loops and what needed to happen next. So not really a solution for me.

Solution #2

This solution actually came from a member of my local code mentoring meetup. This solution uses Promises.

First, I created a JavaScript class for my various .csv needs.

const fs = require('fs');
const readline = require('readline');
const path = require('path');

class CSVHelpers {
  constructor () {
    super();
  }

  /**
   * @param  {string} filePath
   * @return {promise} Array of row objects. Key: header, value: field value
   */
  read (filePath) {
    return new Promise ((resolve, reject) => {
      try {
        const reader = this._createReadStream(filePath);
        let rows = [];
        let headers = null;

        reader.on('line', row => {
          if (headers === null) {
            headers = row.split(',');
          } else {
            const rowArray = row.split(',');
            const rowObject = {};
            rowArray.forEach((item, index) => {
              rowObject[headers[index]] = item;
            });

            rows.push(rowObject);
          }
        })
          .on('close', () => {
            resolve({
              rows,
              file: filePath
            });
          });
      } catch (error) {
        reject(error);
      }
    });
  }

  /**
   * @param  {type} filePath
   * @return {type} Readline event emitter
   */
  _createReadStream (filePath) {
    const fd = fs.openSync(path.resolve(filePath));
    const fileStream = fs.createReadStream(path.resolve(filePath), {fd});
    return readline.createInterface({
      input: fileStream
    });
  }
}

module.exports = CSVHelpers;

Then in my code:

const csv = new CSVHelpers();
const dataFiles = fs.readdirSync(<pathToDirectory);

const filePromises = dataFiles.map(file => {
  return csv.read(<pathToFile>);
});

Promise.all(filePromises)
  .then(values => {
    // do something with the values.
  });

This Promise approach means I don't need to trying to next loops or callbacks.

Conclusion

I do not know if this is the best solution, but it works for my use case, and solves the race conditions I was having. If you have better ways to solve the problem, please let me know.

Not Another Hello World

sdcaulley — Sat, 02 Apr 2022 17:00:05 +0000

One of the biggest blockers for me moving from a beginner JavaScript developer to a more intermediate developer has been the lack of more advanced examples. There are an abundance of "hello wold" examples out there, but not much in the way of more advanced examples. (Really, who makes an API call only to print it to the console.)

So I am introducing a new series to for more advanced examples. For me, part of the learning experience is turning around and teaching/sharing my new knowledge with someone else. My hope is that someone else can use what I learned to improve their JavaScript as well.

Note: The name of this series is a nod to the movie "Sideways", where the Miles character rants about having to drink merlot. Here is the clip.

JavaScript Function Expressions vs Declarations

sdcaulley — Sat, 19 Mar 2022 18:01:55 +0000

One of the beauties of JavaScript is its flexibility - there is multiple ways to do just about everything. Working with functions is no exception. There are two primary ways to write functions in JavaScript - function expressions and function declarations. I am going to look at both and discussion what I feel are the similarities and differences of each, and why I choose one style over the other.

Definitions

Function Expression

A function expression is a type of JavaScript expression that once resolved, assigns its value to a variable. It typically looks like this:

const add = function (x, y) {
  return x + y;
}

console.log(add(1, 2));
// expected output: 3

or this:

const add = (x, y) => {
  return x + y;
}

console.log(add(1, 2));
// expected output: 3

The value in the variable is the function and can be called like a function declaration.

Function Declaration

A function declaration is a JavaScript statement. It is declared with the function keyword, like this:

function add (x, y) {
  return x + y;
}

console.log(add(1, 2));
// expected output: 3

Similarities

Both function declarations and function expressions have the same syntax: name, parameter, and statements.

// function declaration
function name (parameter, parameter, ...ect) {
  statements
}

// function expression
const variable = function name (parameter, parameter, ...ect) {
  statements
}

Both styles of functions are considered higher-order functions.

Differences

There are some significant differences between the two styles.

Syntax

Where both styles use the same syntax - name, parameter, and statements - function expressions consider all three to be optional, and function declarations require name. This means that function expressions can be anonymous functions.

Hoisting

Function declarations are hoisted, and function expressions are not. This means that you cannot use a function expression before you create it. The variable that the function expression is assigned to will be hoisted but not the value. This has to do with how JavaScript parses a file, and a topic for a later post.

My Preference

In my code I prefer to use function declarations. There are several reasons for this (all completely subjective):

Readability - I find function declarations to be easier to read and lead to cleaner code.
Debugging - I find functions with a name to be easier to debug, and function declarations enforce the naming of functions.
Hoisting - Since function declarations are hoisted, I can write my procedural logic at the top of my file or module and declare the functions that are used later. This leads to cleaner, easier to follow code.
Laziness - Why type const blah = function when I can just write function.

Conclusion

Using function expressions versus function declarations is going to be a personal style choice. Knowing the pros and cons of each style will help you choose which style will work best with the code you are writing.

Until next time!

Resources

JavaScript Expressions
JavaScript Statements
JavaScript Function Expression
JavaScript Function Declaration

Gotcha with Generated Files and Clients

sdcaulley — Fri, 06 Nov 2020 12:28:54 +0000

When using [grpc-tools](https://www.npmjs.com/package/grpc-tools) to create generated Node.js files, you have three options the --grpc_out flag:

grpc_js: Generates code with require('@grpc/grpc-js') instead of require('grpc')

generate_package_definition: Generates code that does not require any gRPC library, and instead generates PackageDefinition objects that can be passed to the loadPackageDefinition function provided by both the grpc and @grpc/grpc-js libraries.

no option: Generates code with require('grpc')

Using the grpc_js or no option actually triggers protoc to create a generic client in the *_grpc_pb.js file:

exports.RouteGuideClient = grpc.makeGenericClientConstructor(RouteGuideService);

What I found was that you need to use the same instance of gRPC (not just the same version) for them to work together. This will not be a problem if you are declaring you protos, generated files, and clients all in the same project. However, I have mine spread in different npm packages, so I did not use this functionality.

I Develop in Virtual Machines

sdcaulley — Sun, 01 Nov 2020 16:31:42 +0000

I am a JavaScript developer, and I use Virtual Machines for all of my development.

My set up:

Hardware:
- Lenovo ThinkPad T460s with:
- 20 GB of RAM
- 500 GB SSD hard disk
OS:
- Debian 10
Virtualization Software:
- native kvm/qeum

I set up a base VM and a separate VM for each of my projects.

The base VM has:

OS
IDE (for me this is atom)
git
wget
curl
Postman
Flameshot(screenshot software)

I will then clone the Base VM, rename it, and set it up for the desired project. I also take periodic snapshots for easy recovery.

My reasons:

Less risk to my laptop.

I have been known to do bad (unintentional) things with my code. Like that day that I somehow set up a Node.js server that spawned itself and used up all my ports in about 10 minutes. Having my dev environment separate from my actual laptop meant that I still had a working laptop, and I could restore the VM from its snapshot.

Less bloat on my laptop.

Let's face it, I really don't want node, npm, mysql, mongodb, git, ...ect on my day to day laptop. They take up space and processing power (even when idling in the background). Having them on separate VM means they are running only when I need them. And when I no longer need them, I can delete the VM.

Specific configurations for specific projects

Each project in its own VM means I can fine tune the environment without disrupting any other project. This makes it easier to track down bugs, and prepare the work for deployment.

Easy transfer.

If and when I need to move to new hardware, I can easily take my dev environment with me, instead of spending hours setting it back up again. I know for some that is part of the excitement of new hardware, but not for me.

There you have it - a brief stroll through my development set up.

Creating a gRPC Client as a Local npm Package

sdcaulley — Sun, 01 Nov 2020 15:13:28 +0000

Making a local npm package is relatively easy - you create a new Node.js project.

npm init

The sole purpose of this package is to have one client for each service that can be easily imported and used by other services. It uses the protobuf packages I created earlier.

I made the entry point to the package <name_of_service>.js, install the protobuf package needed, and gRPC:

npm install --save <path_to_package> @grpc/grpc-js

I then set up my file:

const grpc = require('@grpc/grpc-js');
const { 
  <name_of_service>Services,
  <name of message>Messages } = require('<name_of_package');
const packageDefinition = grpc.loadPackageDefinition(<name_of_service>Services);
const meta = new grpc.Metadata();

gRPC now knows to use the generated package definition, and which messages I need to use for this client.

Note on Messages
When using the generated files, you have to explicitly set up the messages. I did not have to do this originally using the raw *.proto files.

I also set up the metadata object that will be filled as needed by each operation.

I then set up a class that will be the export of the file:

class <name_of_service>Client {
  constructor(pathToService) {
    this.client = new packageDefinition.<name_of_service>(
      pathToService,
      grpc.credentials.createInsecure()
    );
  }
...

I use the constructor to import the path to the service that the client is for, and then actually instantiate the client. I also use it to import any global variables needed, typically any metadata that needs to be sent with the request.

I then write any logic needed. Because these clients are to be "universal", I keep it very simple.

I create a helper function for creating the messages needed for each operation.

async createRequestMessage(request, type) {
  let message;

  switch (type) {
    case <type>:
    message = new <name of message>Messages.<message>();
    message.set<field>(request.<field>);
  }
  return message;
}

The generated files creates a constructor class for each message it finds. So I start by instantiating a new instance of that class and using the .set* syntax to actually put my data in the class. This gets interesting with some of the field types, which I will discuss in later posts.

I then create an async function for each operation in the service.

...
async nameOfOperation(request) {
  meta.set('<key>', <value>);
  const messsage = await this.createRequestMessage(request, <type>);
  const promise = new Promise((resolve, reject) => {
      this.client.<name_of_operation>(message, meta, (err, res) => {
        if (err) {
          reject(err);
        } else {
          resolve(res);
        }
      });
    });

    return promise.then(value => {
      return value;
    });
  }

I start by setting any metadata needed by the receiving service.
Then I send my request off to be turned into a gRPC approved message.
I then create an explicit Promise to receive the response from the service. Since gRPC is synchronous by nature, I found this approach works the best.
Once the Promise as been resolved or rejected, I return the value to the function that originally called it.

And then I export the class.

module.exports = <name_of_class>;

I like to use the module.exports syntax over export default because it is explicit and easy for another developer to follow.

Because the clients are being used by multiple services for various reasons, it made sense to me to have them simply receive the request and return the response, thus allowing the service do what it needed to do with the response.

My clients can now be imported as npm packages to other services and used.

Using My npm Packages in a Node.js gRPC Server

sdcaulley — Sun, 01 Nov 2020 14:25:49 +0000

Having created a local npm package for the protobuf files in my cluster, I can now install that package and use it in a gRPC server.

I install the package:

npm install --save <path to package>

And use it like I would any other installed npm package:

const { exportName } = require('<package name>');

For the gRPC server, I will use the Services export, since it the file that has the definitions for the operations that were defined.

So, my server ends up looking like this:

const grpc = require('@grpc/grpc-js');
const <service_name>Services = require('<package_name>');
const protoDescription = grpc.loadPackageDefinition(<service name>Services);

// operation declarations as logic

async function main() {
  const server = new grpc.Server();
  server.addService(protoDescription.<name_of_service>.service, {
    // operations
  });

  server.bindAsync(
    `0.0.0.0:${config.port}`,
    grpc.ServerCredentials.createInsecure(),
    err => {
      if(err) {
        console.error('err: ', err);
      }
      server.start();
      console.log('Server Started');
    }
  );
}

main();

There is one major difference between using my generated file and directly using a proto file:

When directly using a proto file, you need to import the proto file, create a packageDefinition using @grpc/proto-loader, and feed that to grpc.loadPackageDefinition().

So the server file would look like this:

const grpc = require('@grpc/grpc-js');
const protoLoader = require('@grpc/proto-loader');
const PROTO_PATH = '<path_to_proto_file';
const packageDefinition = protoLoader.loadSync(PROTO_PATH);
const protoDescription = grpc.loadPackageDefinition(packageDefinition);

// operation declarations as logic

async function main() {
  const server = new grpc.Server();
  server.addService(protoDescription.<name_of_service>.service, {
    // operations
  });

  server.bindAsync(
    `0.0.0.0:${config.port}`,
    grpc.ServerCredentials.createInsecure(),
    err => {
      if(err) {
        console.error('err: ', err);
      }
      server.start();
      console.log('Server Started');
    }
  );
}

main();

The generated files skip this step, because I used the --grpc_out:generate_package_definition flag for the grpc-tools.

Creating a Local npm Package for Protobuf Files

sdcaulley — Sun, 01 Nov 2020 12:59:36 +0000

Making a local npm package is relatively easy - you create a new Node.js project.

npm init

In my case, the sole purpose of this package is to deliver generated protobuf files to various gRPC servers and clients.

I made my entry point main.js and used it to export references to the protobuf files. Since protoc creates two files - one for the messages and one for the services - I created a created a variable for each with the naming pattern of <service>Messages and <service>Services.

const <service>Messages = require(`<name of service>_pb.js`);
const <service>Services = require(`<name of service>_grpc_pb.js);

module.exports = {
  <service>Messages,
  <service>Services
};

This pattern makes it very easy to import the code I need into my servers and clients. Also, any of my co-workers needing to consume the packages will not need to guess as to what the name of the exports are.

I can now install these packages as part of the service, and they will update themselves as the package updates.

npm install --save <path to module>

And the package.json will have:

"<name of package>": "file:<path to package>"

Generate Files for gRPC in Node.js

sdcaulley — Sun, 01 Nov 2020 12:52:42 +0000

When I started working with gRPC in Node.js, I followed Google's dynamically generated code base to create my servers and clients. And for basic development, this works really well. At least in a mono repository using relative paths to import the proto files. As we move closer to deployment though, a glaring problem has emerged.

A growing collection of proto files - many of which are sharing messages and models - that need to be imported into various gRPC servers and clients. These servers and clients will be hosted in various Kubernetes clusters.

My solution:

Create an npm package for each service cluster to house the needed proto files.

At first, I thought I would just write a bash script to copy the needed service and message/model files into an npm package and call it a day. However, I know that you can generate actual server and client from the proto files (in some languages), and I want to be ready for that when it comes to Node.js. Therefore, I decided that my npm packages would hold protoc generated files.

To do this, I needed to have protoc and grpc-tools installed on my development machine. (Note: grpc-tools needs to be installed globally for this to work).

I decided to make a bash script file to handle the actual generation. I figure we can use/modify this file later to use in a more automated process.

I remove any existing files. For Node.js, protoc creates two files, both of which end in _pb.js. So I used a wildcard to select and delete all files ending that way.

rm -f <path to target>/*_pb.js

Then I actually generate the files:

grpc_tools_node_protoc --js_out=import_style=commonjs,binary:<path to export directory> --grpc_out=generate_package_definition:<path to export directory> <proto file>

Note: I found that the script needed to change directories and run in the directory housing the proto file. I was not able to figure out how to use relative paths to that directory, though I do believe there is a way to do it.

The --grpc_out= flag supports two options:

grpc_js: Generates code with require('@grpc/grpc-js') instead of require('grpc')

generate_package_definition: Generates code that does not require any gRPC library, and instead generates PackageDefinition objects that can be passed to the loadPackageDefinition function provided by both the grpc and @grpc/grpc-js libraries.

I tried both flags and found that the generate_package_definition worked best in both my servers and clients.

My export directory is the npm package I had created for the cluster that needed the proto file. I am now able to use the package - and therefore the files - in both my servers and clients.

I am very pleased with the results. The script files does need to be manually run with any proto file changes, but I can automate that at a future date.

gRPC and Node.js

sdcaulley — Sat, 31 Oct 2020 19:40:39 +0000

The start-up I am working with has chosen a microservice architecture for its product, and is predominately using gRPC for communication between the services. This has proven to be a tremendous challenge and growth opportunity for me - having not worked before with either microservices or gRPC.

I have found that there is not written about out using gRPC and Node.js and want to share what I have learned. This will be a series of articles about what I have found - mostly by trial and error. There is no particular order to this series, and most of the posts will be structured in a problem/solution type format.

I hope this helps someone along in their gRPC adventure.

Functions - Arguments vs Parameters

sdcaulley — Wed, 30 Sep 2020 12:54:56 +0000

I have been doing some deep dive reading in my pursuit of learning JavaScript better. Today's learning is about functions and specifically the difference between arguments and parameters.

Arguments are the values you pass in, and parameters are the named variables inside the function that receive those passed-in values.

The example:

function foo(x,y) {
// ..
}

var a = 3;

foo(a, a * 2);

a and a * 2 (actually, the result of a * 2, which is 6) are the arguments to the foo(..) call. x and y are the parameters that receive the argument values (3 and 6, respectively).

from Functional-Light JavaScript by Kyle Simpson

Authentication with swagger-client in the Browser

sdcaulley — Tue, 07 Apr 2020 15:32:20 +0000

Use of the OpenAPI specification has been growing. My place of work is very interested in leveraging it as much as possible - in both the backend and the frontend. Being on the frontend team, I was tasked with creating an API call using an OpenAPI spec, and swagger-client - a JavaScript client for using the OpenAPI spec. In this post, I want to share with you what I learned about authentication and swagger-client. It is a fairly simple process, but is not laid out clearly in the documentation for swagger-client. The code below is based on the example I found in the migration documentation.

This implementation is through the browser and called for an independent module. Following the instruction in the README, I pulled the CDN into the application through our index.html:

<script src='//unpkg.com/swagger-client' type='text/javascript'></script>

This made the SwaggerClient class globally available in the application.

In the module, I started by creating a function that creates a new client, specific to the spec I want to use.

// function takes the spec url I want to use and the authentication code needed.
async function createClient(specUrl, auth) {
  const client =  await new SwaggerClient(
    specUrl,
    {
      authorizations: {
        // This key will be defined by the OpenApi spec
        APIKeyQueryParam: auth,
      }
    }
  )
  .catch(err => console.error('client error: ', err));

  return client;
}

I then created the function to make the API call.

export default async function apiCall(specUrl, auth) {
  const client = await createClient(specUrl, auth);

  const data = await client.apis['Search Parameter'].titleSearch({ s: 'potter' })
    .catch(err => console.error('data fetch error: ', err));

  const result = data.body.Search;

  return result;
}

Setting up my functions this way allows a neat decoupling of the creation of the client and the consumption of the client. It will also allow me to later refactor both functions to more generic uses as our code base matures.

I hope this helps anyone else trying to do a browser implementation of the swagger-client.