DEV Community: Jorge Barrachina Gutiérrez

Getting started with FX: Powerful and handy JSON manipulation from the command line

Jorge Barrachina Gutiérrez — Thu, 03 Dec 2020 09:46:33 +0000

Why this post?

If you spend a lot of your time-consuming APIs and having to build data pipelines, you will enjoy this post (if you don't know fx yet).

According to several sources, data scientists spend between 70-80% of their time normalizing data before they begin to play with it. That's a lot of time, so it's a good time investment to have powerful tools at your disposal without a steep learning curve.

When we talk about data processing, one of my favorite anecdote is the following one: Command-line Tools can be 235x Faster than your Hadoop Cluster. Sometimes people spend thousands of dollars on new software without realizing they can do the same task faster and cheaper with some alternatives. You've got me, I'm an old-school command-line guy ;-)

The point I'm trying to make here is that there is a ton of tools out there, but my advice to you is: "be conservative with your toolset". Spend more time with the tools you use daily. If you detect a recurrent issue in several scenarios, then it's time to look for alternatives.

I used jq for a couple of years but every time I struggled with some data transformation, the pain came from the same place: learning the concrete syntax for that tool. In that sense, fx freed me from that inconvenience.

Don't get me wrong. jq it's a very useful tool, but it takes time to control it. In my case, I don't want to spend more time learning a concrete syntax that is only useful with jq.

In this post, I'm not going to cover how to use jq, but if you're interested in it, here are some useful references:

fx

How to install

Prerequisite: You have to install nodejs on your computer.

npm i -g fx

fx can do a lot of things (see What can I do with fx below), but let me first explain the two modes on which fx operates:

fx modes

Interactive: When you are not familiar with the data (JSON) you're playing with, this mode is quite useful because it lets you explore the data structure, find values, filter them, apply some transformation... Think interactive mode as a playground. Here you can see a sneak peak of fx in action:

As you can see, it's pretty intuitive! You can explore any JSON data in the same way you do when you are accessing an object in Javascript. Bonus point: it supports auto-completion.

Commandline (CLI): Once you know the data, it's time to apply some transformations. This mode can be used in scripts, it's pipe-friendly, so you can concatenate several fx commands in a one-liner. Think cli mode as a grep,sed or awk command, but fx reads JSONs instead of lines. Let's see another visual example:

TIP: When you want to select text in interactive mode you need to press Alt / Fn key depending on your terminal

Once we saw the available modes of fx, let's practice with some examples.

What can I do with fx

If you want to follow the examples, just type in your terminal:

curl -sS "https://jsonplaceholder.typicode.com/users" -o users.json

Quick JSON exploration

When you are in interactive mode you can search for strings or use regular expressions. If you are familiar with vim editor you'll feel at home. If you're not familiar with regular expressions, you can start just typing "/" followed by the string you are looking for.

By default, the search is case-insensitive, so you don't have to worry about that.

To navigate across search results, just press Enter to go to the next match.

If you don't feel comfortable with regular expressions but you want to practice, I recommend you to take a look at RegEx101. It's a playground where you can start to master regular expressions.

Transform

From each user, I want to keep only the website and the "geo" keys.

Easy peasy! Because in fx we can use plain Javascript, let's translate this scenario to a Javascript code, and later apply this directly in fx:

For each user (object) we can make use of destructuring technique, to get the keys we want (website,geo) from the object and discard the rest:

var user = {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret",
  "email": "Sincere@april.biz",
  "address": {
    "street": "Kulas Light",
    "suite": "Apt. 556",
    "city": "Gwenborough",
    "zipcode": "92998-3874",
    "geo": {
      "lat": "-37.3159",
      "lng": "81.1496"
    }
  },
  "phone": "1-770-736-8031 x56442",
  "website": "hildegard.org",
  "company": {
    "name": "Romaguera-Crona",
    "catchPhrase": "Multi-layered client-server neural-net",
    "bs": "harness real-time e-markets"
};

var {website,geo,...rest} = user; // user is our object
// website = "hildegard.org"
// geo = {"lat": "-37.3159", "lng": "81.1496"}

Now we want to apply this operation on each user, so let's do it with .map

users.map(({website,geo,...rest}) => ({website,geo}))

In fx, we will do it like this:

cat users.json | fx '.map(({website,geo,...rest}) => ({website,geo}))'

Isn't it beautiful?

You may think: "this is very basic stuff Jorge." Yep, indeed. But put yourself in the shoes of someone who has to do lots of different data transformations every day, or someone who is just getting insights from different data sources each time... Do you think that person is going to write a script every time?

That's the beauty of fx for me, it lets you do things very quickly without the need to learn anything more!

Filter

From our JSON file, I want to filter the company names of those which email has the domain .biz

cat users.json | fx '.filter(({email,...rest}) => /\.biz$/.test(email))' '.map(user => user.company.name)'

Got it!

Extra ball: Could I convert the results above in a CSV format (company and mail by line)?

cat users.json | fx '.filter(({email,...rest}) => /\.biz$/.test(email))' '.map(user => `${user.company.name};${user.email}`)' '.join("\n")'

Use your favorite npm module along fx

fx offers a way to include npm modules in the execution context.

When you are dealing with data structures in Javascript, lodash is a very handy option. Also, dayjs let us play with dates and time data easily.

Let's see how to use it along with fx:

Create .fxrc file in $HOME directory, and require any packages or define global functions.
Install lodash and dayjs globally in your computer:

npm i -g lodash dayjs

Set NODE_PATH env variable. This step is IMPORTANT to allow fx to make use of globally installed packages.

export NODE_PATH=`npm root -g`

Put in your .fxrc file:

Object.assign(global, require('lodash/fp'))
global.dayjs = require("dayjs");

Now, let's play this scenario: I want to have a list of the 5 most recent github repositories I have that includes the day of the week I created each of them.

curl -sS "https://api.github.com/users/ntkog/repos" |  \
fx '.map(({name,created_at,clone_url,...rest}) => ({name,created_at,clone_url}))' \
'sortBy("created_at")' \
'reverse' \
'take(5)' \
'map(repo => ({...repo, weekDay : dayjs(repo.created_at).format("dddd")}))'

Let's look at each step (we can chain several transformations in fx as you can see)

"\" at the end of the line it's just for separating one command into several lines

Get all info of my github repos and pipe it to fx

curl -sS "https://api.github.com/users/ntkog/repos" |

Keep only name,created_at,clone_url from each object

fx '.map(({name,created_at,clone_url,...rest}) => ({name,created_at,clone_url}))' \

Sort array by created_at key

'sortBy("created_at")' \

Invert the order of the results

'reverse' \

Take 5 objects

'take(5)' \

Add weekDay key to each object

'map(repo => ({...repo, weekDay : dayjs(repo.created_at).format("dddd")}))'

It's a very expressive way to transform the data step by step, don't you think?

Explore more possibilities

I didn't find a lot of articles talking about fx, but this talk from Антон Медведев gave me a lot of ideas on how to get the most out of fx.

Give fx a try, you won't regret it! Let me know if you find other tricks!

Happy hacking :-)

Photo by Samuel Sianipar on Unsplash

Postman quick tricks

Jorge Barrachina Gutiérrez — Sun, 22 Nov 2020 10:21:23 +0000

Postman is an awesome tool. It lets you automate a lot of the work when you are playing with API's. But are you really getting the most of out it?

I'm going to show some little tricks that can help you save valuable minutes in your day-to-day workflow.

If you want to reproduce these tricks, you will need to have installed nodejs and Postman Collection SDK in your computer.

Scenario #1: Rename all the items of a collection adding a prefix sequence index

Sometimes we are working on a large postman collection and we want to be explicit on what order of execution the user should follow.

Adding a prefix to each item of the collection seems like a good idea, but if we have several items in our collection, doing this manually is pretty boring..., there has to be a way to do it quickly...

Indeed! There is an easy way! Here is the code for the impatient:

Create a file called rename_items_collection.js and paste the following code:

// Import Postman Collection SDK
const fs = require('fs');
const {Collection} = require('postman-collection');
const FILENAME = './sample-collection.json';
const SEP = '-';

// Read our postman collection file
const myCollection = new Collection(require(FILENAME));

// Update list of items renaming each of them with a sequence prefix
myCollection.items.members = myCollection.item.members.map((item,idx) => item = {...item, name: `${idx+1} ${SEP} ${item.name}`});

// Output collection content
console.log(JSON.stringify(myCollection.toJSON()));

Open a terminal and type:

node rename_items_collection.js

You will see in your screen the contents of the collection. If you want to save it, run this one:

node rename_items_collection.js > renamed_collection.json

Now, you can import renamed_collection.json in your Postman App and you will see each item name prefixed with an index.

Scenario #2: Make requests with fake data

You need to test your API with some random and fake data, but you don't want to implement some function to randomize each data type.

Did you know that Postman has dynamic variables based on faker.js mocking data library?

The best part: There is some "Finance" data you can mock. Here are some examples:

Random IBAN account number ? : use {{$randomBankAccountIban}}
Random ISO-4217 currency code (3-letter) ? : use {{$randomCurrencyCode}}
Random Bitcoin address : use {{$randomBitcoin}}

Take a look of the complete variable list.

If you want to use these variables in a Pre-request section, you should use it as in the following example:

// Generate a random UUID

// This works
var uuid = pm.variables.replaceIn('{{$guid}}');

//This won't work
var uuid = {{$guid}}

Scenario #3: Check JWT claims with Javascript within Postman

I don't know you, but when I work I have several applications opened, sometimes too many.

When I have to test or debug an API that makes use of OAuth 2.0 with JWT, sometimes I need to check if a request has proper data in the JWT. It's useful to remember Occam's Razor :

" of two competing theories, the simpler explanation of an entity is to be preferred"

What does it have to do with this scenario?

When you are troubleshooting some requests, we tend to look for complex assumptions. It's better to start with the easiest ones, which are the most frequent. So, let's do it.

Imagine we have the following JWT:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c

As we know, every JWT consists of 3 parts (the '.' "splits" each part). I've just given you a clue... .

If you want to know the claim content (Ignoring verifing the JWT signature), can you do it?

Yes! with 2 lines of Javascript!

Put the following lines in the Pre-request tab on the request you want to check

var [jose_header,payload,] = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c".split('.');
console.log(JSON.parse(atob(payload)))

with the atob javascript native function we can decode Base64

If you have the JWT content in a variable called assertion you can substitute the string with the following example

var [jose_header,payload,] = pm.variables.get("assertion").split('.');
console.log(JSON.parse(atob(payload)))

Here you have a reminder diagram on Postman supported variables and their scopes:

When you run this code, you will see in the Postman console:

{
  "sub": "1234567890",
  "name": "John Doe",
  "iat": 1516239022
}

Scenario #4: Signing JWT tokens directly within Postman

Maybe you know this amazing cryptography tool called jsrsasign : It supports a lot of the common tasks you have to do when working with secure APIs:

RSA/RSAPSS/ECDSA/DSA signing/validation
ASN.1
PKCS#1/5/8 private/public key
X.509 certificate
CRL
OCSP
CMS SignedData
TimeStamp
CAdES JSON Web Signature/Token/Key (JWS/JWT/JWK)

There are multiple ways to use this library within Postman. We, as developers, should evaluate which way is better for our use case. Here you have two ways of using jsrsasign:

Load jsrsasign from external URL

This is the simplest way to use it: postman-util-lib. Kudos to joolfe.

If you want to try this way, on the postman-util-lib website there is a good documentation on how to use it

But here are two corner cases you can think about:

Should we trust a site we cannot control?
What if you work in a restricted environment where every url needs to "be validated" beforehand in the firewall of your organization?

Thinking about those scenarios, I want to share with you a way of using this awesome library locally.

Load jsrsasign locally

So, let's do it!

Trial #1: Read the library from a local file

Unfortunately, this is not possible yet in Postman :-( . Take a look this issue in Postman App Support.

Trial #2: Serve the library from localhost

Let's grab the file from https://raw.githubusercontent.com/kjur/jsrsasign/master/jsrsasign-all-min.js.
Let's serve this file from localhost. We can use http-server nodejs package to do it. If you prefer to serve the file with another method, there are a ton of them.

mkdir jsrsasign_library
cd jsrsasign_library
wget https://raw.githubusercontent.com/kjur/jsrsasign/master/jsrsasign-all-min.js
npm i -g http-server
http-server . -p 8080

From your browser you can reach the file at http://localhost:8080/jsrsasign-all-min.js

Next, assume you have a variable in the Postman environment called sign_secret . If you just want to try it, you can substitute in the following code with a string (Although it's a bad practice)

Now go to Pre-Request tab , and copy the following

var URL_local_jsrsasign = "http://localhost:8080/jsrsasign-all-min.js"
if(!pm.globals.get("jsrsasign")) {
    pm.sendRequest(URL_local_jsrsasign ,(err, res) => {
      if(!err){
       pm.globals.set("jsrsasign", res.text());
      }    
    });
}
// Load jsrsasign library in global context
eval(pm.globals.get("jsrsasign"));

const jose_header = {
  "typ": "JWT",
  "alg": "RS256"
};
const payload = {
  "sub": "1234567890",
  "name": "John Doe",
  "iat": 1516239022
}

// Sign JWT
var jwt = KJUR.jws.JWS.sign("HS256", jose_header, payload, pm.environment.get("sign_secret"));
console.log(jwt);

// verify JWT
isValid = KJUR.jws.JWS.verify(jwt, pm.environment.get("sign_secret"), ["HS256"]);

I hope you find these little tricks useful. Happy hacking!

Cover Photo Credit: Photo by Barn Images on Unsplash