David Metcalf

Posted on Apr 6, 2021 • Updated on May 17, 2021 • Originally published at blog.metcalf.dev

Running other languages with Node and JavaScript

#node #javascript #python #webdev

As a web developer, JavaScript is my primary coding language. I can also grind and Google my way through some basic Python. I have done some tutorials and built a very simple text-based game, but that is about it. Python is such a powerful and readable language that I do plan on learning it some day. So many side projects, so little time.

I am doing some contract work for a machine learning startup, mostly writing frontend components (Next + TypeScript) plus a bit of database querying. Recently, I was assigned a GitHub ticket by the lead developer that gave me an unique opportunity to work with a little Python (of a much higher quality than I can write). The task involved establishing a connection to a third party API so clients can push and pull data between cloud platforms via a simple form in our UI. The docs for this outside company gave detailed cURL commands, and they were kind enough to provide us with some Python scripts we could use as a starting point. 🐍

Reinventing the wheel 🚲

With these scripts in hand, I reviewed their code. Calling this API successfully involved running three commands, with each successive command requiring user credentials and the data returned by the previous commands to be passed in as sys.argv arguments. For my fellow Node / JS people, these are equivalent to process.argv arguments.

$ python first.py <id> <secret>
12345
$ python second.py <id> <secret> 12345
67890
$ python third.py <id> <secret> 12345 67890
{
  "response": "lots-of-data",
}

How could I run all these commands with our TypeScript application? Do I need to go through this Python code, line by line, converting it to TS in our backend? Should I use these scripts and the cURL commands as inspiration to just write it all from scratch? That means axios, OAuth2 tokens, GraphQL queries, large JSON datasets; a lot of new code!

Put the children to work 👩‍👩‍👧‍👦

After considering the tools we already had in our stack, I realized that I could lean on Node to spin up these Python scripts exactly as they were, using the child_process module. A Child process makes use of Node's ability to spawn subprocesses. These child processes can be asynchronous, running in the background, or they can be synchronous by blocking the event loop until exited or terminated.

Spin up a spawn 🕸️

Let's look at a simple way we can utilize the spawn() method. For this exercise, we are going to use Python to perform some math for us.

First, our Python:

# exponents.py
# access to command line arguments
import sys

# args are counted like lists or arrays
# argv[0] is the file name
base = sys.argv[1]
exponent = sys.argv[2]

# pow() raises first param to power of the second
# args are str, so convert to int
result = pow(int(base), int(exponent))

print(result)

# to execute:
# python exponents.py <number> <another_number>

We can easily do that math with JavaScript. Just pretend we're actually crunching serious numbers with NumPy or something.

Now on the JavaScript side of our app, we can run that Python script with child_process.spawn():

/* index.js */
/* bring in node's spawn method */
import { spawn } from 'child_process';

/* args for python */
let fileName = './exponents.py';
let base = 5;
let exponent = 3;

/**
 * @desc      spawn async background process
 * @param     {string} command 'python', 'node' etc.
 * @param     {Object[]} array of args
 */
const exponentProcess = spawn('python', [fileName, base, exponent]);

/**
 * @desc      get and log value returned by python
 * @listens   'data' in pipeline: stdout.on()
 * @returns   {string} data from python math
 */
const getProduct = () => {
  exponentProcess.stdout.on('data', (data) => {
    let result = data.toString();
    console.log(result);
  });
};

/* expected return is '125' */
getProduct();

Now, run this code in your terminal:

user@Machine ~/file/location
$ node index.js
125

Pretty cool! We'll examine this JS code below.

(Remember, you need a package.json to run Node, so npm init -y)

Awaiting your return ⏳

In this contrived example, things are running smooth. There is not a lot of room for error in such a simple program. But what if you are running a more demanding and complex set of instructions in your child process? What if you are making multiple API calls, one after another? The Python (or other language) code may or may not be built to handle it's own duties. Our good friend JavaScript, however might need a little help being patient. We can add some guard rails and error handling in our function:

const getProduct = () => {
  let result = '';
  exponentProcess.stdout.on('data', async (data) => {
    try {
      result = await data.toString();
    } catch (err) {
      console.error('Something went wrong.', err);
    }
    console.log(result);
    return result;
  });
};

What just happened?

Time for a quick break down. The second line of our function invokes the spawn method that was assigned to the constant exponentProcess. Think of Node child processes as being connected by pipelines, with the output of one process being connected to the input of the next. We then add a listener to the stdout (standard output) of the Python process. This .on() takes in an event type ('data', but there are many others) and a callback function. We can simply make this callback async, and then await the value for result inside of a try...catch block. Now we can pass this value to another part of our program:

    try {
      result = await data.toString();
    } catch (err) {
      console.error('Something went wrong.', err);
    }
    anotherFunction(result);

Again, this is overkill in this example. I still think it is good practice to account for potential lag or unwanted responses!

Wrap up 🌯

This is far from a thorough guide. I myself have only scratched the surface of child_process super powers. What other cool tricks can you come up with? What other languages can you leverage in your JavaScript web app? Would love to hear your thoughts in the comments! 👇

Thanks for checking out my second post on DEV!

Enjoy this post? How about sharing a Tweet to spread the love!

Top comments (8)

David Metcalf • Apr 6 '21 • Edited

If you want to explore using something like this in production, there are resources to make this more traceable & scalable, or to simply add a sugary layer of abstraction. Checkout npm python-shell

Here is a repo with starter files: Spawn-Python

Cameron Thompson • Apr 6 '21

I was thinking of something like this the other day! Thanks!

David Metcalf • Apr 6 '21

That's awesome! It's an area of Node that I'm still exploring and learning about, but so much potential.

The use-case I introduced it with is still in development. That ticket may not actually be solved with spawn()! But we learn from trying new approaches, right?