DEV Community: simo

Purest

simo — Mon, 08 Feb 2021 16:03:42 +0000

The last REST API client library that you will ever need

Countless services on the Internet are being exposed over REST API. Most if not all REST API service providers have client libraries for various programming languages to interface with their API.

While all of that is nice, that also means that for every REST API service provider we have to learn a new API interface of that particular client library.

And if that's not the worst, then what if we have to interface with multiple REST API service providers using multiple REST API client libraries in a single code base?

It becomes a mess

The reason why is because we are dealing with client libraries that were never designed to interoperate between each other, even though they are doing roughly the same operations under the hood. The solution to this is to go one layer below and create the client library ourselves.

But how?

Purest is a generic REST API client library for building REST API client libraries. It's a tool for abstracting out REST APIs.

Introduction

Lets take a look at some basic configuration for Google:

{
  "google": {
    "default": {
      "origin": "https://www.googleapis.com",
      "path": "{path}",
      "headers": {
        "authorization": "Bearer {auth}"
      }
    }
  }
}

With it we can instantiate that provider:

var google = purest({provider: 'google', config})

Then we can request some data from YouTube:

var {res, body} = await google
  .get('youtube/v3/channels')
  .qs({forUsername: 'GitHub'})
  .auth(token)
  .request()

The above example demonstrates how a REST API provider can be configured and used in Purest by accessing its default endpoint.

Lets have a look at another example:

{
  "google": {
    "default": {
      "origin": "https://www.googleapis.com",
      "path": "{path}",
      "headers": {
        "authorization": "Bearer {auth}"
      }
    },
    "youtube": {
      "origin": "https://www.googleapis.com",
      "path": "youtube/{version}/{path}",
      "version": "v3",
      "headers": {
        "authorization": "Bearer {auth}"
      }
    }
  }
}

This time around we have an explicit endpoint called youtube for accessing YouTube only:

var {res, body} = await google('youtube')
  .get('channels')
  .qs({forUsername: 'GitHub'})
  .auth(token)
  .request()

The provider configuration is just a convenience for extracting out the request options that we don't want to specify for every request. The auth method is used for replacing the {auth} token found in your configuration, get is the HTTP method being used and its value is the substitute for the {path} token. The qs method is sort of a convention for naming a querystring object that is then being encoded and appended to the request URL.

The above request results in:

GET https://www.googleapis.com/youtube/v3/channels?forUsername=GitHub
authorization: Bearer access_token

What else?

So far we have used Purest like this:

var google = purest({provider: 'google', config})

This allows us to have a configuration and a provider instance for it. Any other dynamic option that is needed for the request have to be passed for every request.

Sometimes, however, we may want to configure certain dynamic values per instance:

var google = purest({provider: 'google', config,
  defaults: {auth: token}
})

Then we no longer need to set the access token for every request:

var {res, body} = await google('youtube')
  .get('channels')
  .qs({forUsername: 'GitHub'})
  .request()

Cool, but what if we want to make our API more expressive?

What if we want to make it our own?

var google = purest({provider: 'google', config,
  defaults: {auth: token},
  methods: {get: ['select'], qs: ['where']}
})

Yes we can:

var {res, body} = await google('youtube')
  .select('channels')
  .where({forUsername: 'GitHub'})
  .request()

Every method in Purest can have multiple user defined aliases for it.

Lastly, accessing an endpoint defined in your configuration can be done by using the explicit endpoint method or its default alias called query:

var {res, body} = await google
  .query('youtube')
  .select('channels')
  .where({forUsername: 'GitHub'})
  .request()

Congratulations!

Now you know the basics.

But the possibilities are endless ...

Lets have a look at another example.

Refresh Token

One very common thing to do when working with REST API providers is to refresh your access token from time to time:

{
  "twitch": {
    "oauth": {
      "origin": "https://api.twitch.tv",
      "path": "kraken/oauth2/{path}"
    }
  }
}

Using the above configuration and the default aliases defined in Purest we can refresh the access token like this:

var {res, body} = await twitch
  .query('oauth')
  .update('token')
  .form({
    grant_type: 'refresh_token',
    client_id: '...',
    client_secret: '...',
    refresh_token: '...'
  })
  .request()

Again query is just an alias for the endpoint method used to access the oauth endpoint in your configuration. The update method is an alias for post and 'token' replaces the {path} in the path configuration. The form method is sort of a convention for naming application/x-www-form-urlencoded request body object that then is being encoded as request body string.

The above request results in:

POST https://api.twitch.tv/kraken/oauth2/token
content-type: application/x-www-form-urlencoded

grant_type=refresh_token&client_id=...&client_secret=...&refresh_token=...

Kinda nice!

Alright, but lets take a look at something more practical:

{
  "twitch": {
    "refresh": {
      "origin": "https://api.twitch.tv",
      "path": "kraken/oauth2/token",
      "method": "POST",
      "form": {
        "grant_type": "refresh_token",
        "refresh_token": "{auth}"
      }
    }
  }
}

Then we can set the application credentials for the entire instance:

var twitch = purest({provider: 'twitch', config, defaults: {
  form: {
    client_id: '...',
    client_secret: '...'
  }
}})

And refresh the access token like this:

var {res, body} = await twitch('refresh')
  .auth('the-refresh-token')
  .request()

Each one of your users will have their own refresh_token, but most likely all of them will be authenticated using a single OAuth application. So it makes sense to configure the provider to use your app credentials by default and only supply the refresh token on every request.

OpenID Connect

OpenID Connect is a popular framework for user authentication and user identity.

One very common theme about it is verifying your JSON Web Token (JWT) that can be either access_token or id_token:

{
  "auth0": {
    "discovery": {
      "origin": "https://{subdomain}.auth0.com",
      "path": ".well-known/openid-configuration"
    }
  }
}

The above configuration is about the discovery endpoint of Auth0 that contains a JSON document outlining certain settings being set for that tenant. The {subdomain} is your tenant name or tenant.region where region applies:

var auth0 = purest({provider: 'auth0', config,
  defaults: {subdomain: tenant}
})

var {body:doc} = await auth0('discovery').request()
var {body:jwk} = await auth0.get(doc.jwks_uri).request()

We request the discovery endpoint and store that document as the doc variable. Then we request the absolute jwks_uri returned in that JSON document and store it as jwk variable. The jwks_uri endpoint returns yet another JSON document containing a list of public keys that can be used to verify a token issued from that tenant:

var jws = require('jws')
var pem = require('jwk-to-pem')

var jwt = jws.decode('id_token or access_token')
var key = jwk.keys.find(({kid}) => kid === jwt.header.kid)

var valid = jws.verify(
  'id_token or access_token', jwt.header.alg, pem(key)
)

We use two additional third-party modules to decode the JSON Web Token, find the corresponding key id (kid), and then verify that token by converting the public key to a PEM format.

OAuth 1.0a

Last but not least

Some providers are still using OAuth 1.0a for authorization. One popular provider that comes to mind is Twitter:

{
  "twitter": {
    "default": {
      "origin": "https://api.twitter.com",
      "path": "{version}/{path}.{type}",
      "version": "1.1",
      "type": "json",
      "oauth": {
        "token": "$auth",
        "token_secret": "$auth"
      }
    }
  }
}

For convenience we set the application credentials for the entire instance:

var twitter = purest({provider: 'twitter', config, defaults: {
  oauth: {
    consumer_key: '...',
    consumer_secret: '...'
  }
}})

And then we pass the user's token and secret with every request:

var {res, body} = await twitter
  .get('users/show')
  .qs({screen_name: 'github'})
  .auth('...', '...')
  .request()

That works, but having to remember all those weird configuration key names every time is hard. Why not put all of them in the default endpoint configuration once and forget about them:

{
  "twitter": {
    "default": {
      "origin": "https://api.twitter.com",
      "path": "{version}/{path}.{type}",
      "version": "1.1",
      "type": "json",
      "oauth": {
        "consumer_key": "{auth}",
        "consumer_secret": "{auth}",
        "token": "{auth}",
        "token_secret": "{auth}"
      }
    }
  }
}

Then all we need to do is pass them as array of strings:

var twitter = purest({provider: 'twitter', config, defaults: {
  auth: ['...', '...', '...', '...']
}})

And focus only on what's important:

var {res, body} = await twitter
  .get('users/show')
  .qs({screen_name: 'github'})
  .request()

Streaming and Multipart

Lets upload some files:

{
  "box": {
    "upload": {
      "method": "POST",
      "url": "https://upload.box.com/api/2.0/files/content",
      "headers": {
        "authorization": "Bearer {auth}"
      }
    }
  },
  "drive": {
    "upload": {
      "method": "POST",
      "url": "https://www.googleapis.com/upload/drive/v3/files",
      "headers": {
        "authorization": "Bearer {auth}"
      }
    }
  },
  "dropbox": {
    "upload": {
      "method": "POST",
      "url": "https://content.dropboxapi.com/2/files/upload",
      "headers": {
        "authorization": "Bearer {auth}",
        "content-type": "application/octet-stream"
      }
    }
  }
}

As usual we have to instantiate our providers:

var box = purest({provider: 'box', config, defaults: {auth: token}})
var drive = purest({provider: 'drive', config, defaults: {auth: token}})
var dropbox = purest({provider: 'dropbox', config, defaults: {auth: token}})

The file upload endpoint for Box expects a multipart/form-data encoded request body:

var {res, body} = await box('upload')
  .multipart({
    attributes: JSON.stringify({
      name: 'cat.png',
      parent: {id: 0},
    }),
    file: fs.createReadStream('cat.png')
  })
  .request()

This is a common way for transferring binary files over the Internet. Every time you submit a Web form that allows you to pick a file from your local file system, the browser is then encoding that data as multipart/form-data, which is what the multipart method does when an object is passed to it.

We are also using the default fs module found in Node.js to stream that cat photo. Imagine that being a really big and fluffy cat that also happens to weight a lot of megabytes.

This is how we upload our cat photos to Google Drive instead:

var {res, body} = await drive('upload')
  .multipart([
    {
      'Content-Type': 'application/json',
      body: JSON.stringify({name: 'cat.png'})
    },
    {
      'Content-Type': 'image/png',
      body: fs.createReadStream('cat.png')
    }
  ])
  .request()

Note that we are still using the multipart method, but this time around we are passing an array instead. In that case the request body will be encoded as multipart/related, which is yet another way to encode multipart request bodies. You can read more about that endpoint here.

Lastly to upload our cat photo to DropBox we stream it as raw request body:

var {res, body} = await dropbox('upload')
  .headers({
    'Dropbox-API-Arg': JSON.stringify({path: '/cat.png'}),
  })
  .body(fs.createReadStream('cat.png'))
  .request()

No additional encoding is expected for the upload endpoint in DropBox.

Great!

But lets do something a bit more dynamic:

{
  "box": {
    "upload": {
      "method": "POST",
      "url": "https://upload.box.com/api/2.0/files/content",
      "headers": {
        "authorization": "Bearer {auth}"
      }
    }
  },
  "dropbox": {
    "download": {
      "url": "https://content.dropboxapi.com/2/files/download",
      "headers": {
        "authorization": "Bearer {auth}"
      }
    }
  }
}

var {res:download} = await dropbox('download')
  .headers({
    'Dropbox-API-Arg': JSON.stringify({path: '/cat.png'}),
  })
  .stream()

await box('upload')
  .multipart({
    attributes: JSON.stringify({
      name: 'cat.png',
      parent: {id: 0},
    }),
    file: {
      body: download,
      options: {name: 'cat.png', type: 'image/png'}
    }
  })
  .request()

We are making the download request using the .stream() method. This instructs Purest to return the raw response stream.

Then we are piping the response stream from DropBox to the request stream for Box by passing it to the multipart file key. This time, however, we need to pass a few additional options because Purest cannot reliably determine the file name and the mime type to embed into the multipart body.

Conclusion

Simple problems need simple solutions

Purest allows us to go one layer below and elegantly compose our own REST API client.

Purest is a tool for creating abstractions without having to create one.

Purest is a primitive for writing HTTP clients.

Happy Coding!

Serverless OAuth Proxy

simo — Tue, 01 Sep 2020 13:00:02 +0000

Let's talk about OAuth! And more specifically let's talk about Grant:

A completely transparent OAuth Proxy that supports 200+ Social Login providers.

Being able to have a fully functional OAuth Client with just a few lines of code is great. However, up until recently Grant was assuming that you have an HTTP Server up and running with either Express, Koa, Hapi or Fastify on top.

And while you can get a Virtual Machine on the cheap, or spin up your own server instances, that still implies a few things about your architecture:

You are either hosting an HTTP server already, and so attaching Grant to certain routes is not an issue
Or you are willing to host just Grant as a standalone OAuth Client / Proxy Server

But what if your actual API consists of Serverless Functions only? Do you still have to host Grant as a separate HTTP server?

NO!

Grant now comes with 4 Serverless Function handlers for:

In this article we'll go over 4 different examples, covering 4 different topics in Grant. Each example will be covered in a different Serverless Function handler, either aws, azure, gcloud or vercel, but all topics and examples apply to any other Grant handler. Including the conventional HTTP Framework based ones such as express, koa, hapi and fastify.

So I encourage you to read through the entire article even if you are interested in only one Serverless Function provider. Each section will also cover the minimum amount of details that you want to know about that specific provider.

At any given point you can jump right into the examples by following the links above. All examples are using Terraform for reproducible deployments with a Makefile on top for orchestration, but you can use any other method to deploy your infrastructure.

AWS Lambda

var grant = require('grant').aws({
  config: {/*Grant configuration*/}, session: {secret: 'grant'}
})

exports.handler = async (event) => {
  var {redirect, response} = await grant(event)
  return redirect || {
    statusCode: 200,
    headers: {'content-type': 'application/json'},
    body: JSON.stringify(response)
  }
}

The first example is about using the State Transport in Grant:

{
  "defaults": {
    "transport": "state"
  }
}

With State Transport the response data will be available in the response object returned by the Grant handler. In all other cases when Grant needs to perform an HTTP redirect, you have to return the redirect object instead.

One specific thing about AWS Lambda sitting behind AWS API Gateway is that it is required to specify the full path prefix that includes the stage name of your AWS API Gateway:

{
  "defaults": {
    "origin": "https://[id].execute-api.[region].amazonaws.com",
    "prefix": "/[stage]/connect"
  },
  "google": {}
}

Then we login by navigating to:

https://[id].execute-api.[region].amazonaws.com/[stage]/connect/google

And the redirect URL of your OAuth App have to be set to:

https://[id].execute-api.[region].amazonaws.com/[stage]/connect/google/callback

Azure Function

var grant = require('grant').azure({
  config: {/*Grant configuration*/}, session: {secret: 'grant'}
})

module.exports = async (context, req) => {
  var {redirect} = await grant(req)
  return redirect
}

With Azure Functions we are going to take a look at the Querystring Transport instead:

{
  "defaults": {
    "transport": "querystring"
  }
}

When Querystring Transport is being used, specifying a callback route or absolute URL is required:

{
  "google": {
    "callback": "/hello"
  },
  "twitter": {
    "callback": "/hi"
  }
}

Grant will redirect to that callback path or absolute URL, and encode the response data as querystring.

This transport is useful when using Grant as OAuth Proxy, or in cases when you want to handle the OAuth response data in another lambda function. Note that this transport may leak private data encoded as querysrting in your callback route.

Similar to AWS Lambda, when the redirect object is being returned by the Grant handler you need to return that object or do something else. Since Grant is going to redirect always when the Querystring Transport is being used, you only need to return the redirect object, because the response data will be handled somewhere else.

One specific thing about Azure is that it is required to set the following requestOverrides for the Grant handler:

{
  "$schema": "http://json.schemastore.org/proxies",
  "proxies": {
    "oauth": {
      "matchCondition": {
        "route": "{*proxy}"
      },
      "requestOverrides": {
        "backend.request.querystring.oauth_code": "{backend.request.querystring.code}",
        "backend.request.querystring.code": ""
      },
      "backendUri": "http://localhost/{proxy}"
    }
  }
}

Azure uses the code querystring parameter to authenticate users. That same code querystring parameter, however, is also being used by the OAuth2.0 framework. That's why it is required to map the code parameter to oauth_code instead, and unset the code parameter, so that we can pass through the Azure's authentication layer. Then the Azure handler for Grant will map the oauth_code back to code so that it can be processed correctly.

Google Cloud Function

var grant = require('grant').gcloud({
  config: {/*Grant configuration*/},
  session: {secret: 'grant', store: require('./store')}
})

exports.handler = async (req, res) => {
  await grant(req, res)
}

With Google Cloud Functions we're going to take a look at the Session Transport:

{
  "defaults": {
    "transport": "session"
  }
}

Up until now all of our examples were using the built-in Cookie Store:

{session: {secret: 'grant'}}

Note that we're now specifying a store key as well:

{session: {secret: 'grant', store: require('./store')}}

This will instruct Grant to use an external Session Store implementation in place of the built-in Cookie Store one:

var request = require('request-compose').client

var path = process.env.FIREBASE_PATH
var auth = process.env.FIREBASE_AUTH

module.exports = {
  get: async (sid) => {
    var {body} = await request({
      method: 'GET', url: `${path}/${sid}.json`, qs: {auth},
    })
    return body
  },
  set: async (sid, json) => {
    await request({
      method: 'PATCH', url: `${path}/${sid}.json`, qs: {auth}, json,
    })
  },
  remove: async (sid) => {
    await request({
      method: 'DELETE', url: `${path}/${sid}.json`, qs: {auth},
    })
  },
}

This is an example implementation of using Firebase as external Session Store. The required methods to implement are get and set. All methods receive a Session ID, and the set method additionally receives an object that needs to be stored in the Session Store.

When Cookie Store is being used all of the data that Grant needs to persist in order to execute the OAuth flow will be stored in the Browser Cookie as Base64 encoded string. Although being signed those values can be read and potentially may leak private data.

The Cookie Store is still a valid option for certain cases, but you have to understand what values are going to be encoded there depending on the transport being used in Grant, and any potential Dynamic Overrides that you might be using.

With Session Transport specifying a callback route is optional. In case you have one, it will be used to redirect the user to another lambda to handle the response:

var Session = require('grant/lib/session')({
  secret: 'grant', store: require('./store')
})

exports.handler = async (req, res) => {
  var session = Session(req)

  var {response} = (await session.get()).grant
  await session.remove()

  res.statusCode = 200
  res.setHeader('content-type', 'application/json')
  res.end(JSON.stringify(response))
}

Unlike Querystring Transport the response data won't be encoded in the URL as querystring, and therefore it cannot leak private data.

Note that in this case we're accessing the internal session module directly:

require('grant/lib/session')

The reason why is because Grant have nothing to do in the callback route. That's also the place where the session can be destroyed if needed.

Unlike AWS Lambda and Azure Function, the Grant handler for Google Cloud Function handles the HTTP redirects internally. The redirect variable is still being returned by the handler when Grant is going to perform an HTTP redirect, but its value is set to a boolean true instead.

You have to specify the redirect_uri explicitly because the actual request URL contains the lambda name in the path, but that is never sent to your lambda handler:

{
  "defaults": {
    "origin": "https://[region]-[project].cloudfunctions.net"
  },
  "google": {
    "redirect_uri": "https://[region]-[project].cloudfunctions.net/[lambda]/connect/google/callback"
  }
}

Then we login by navigating to:

https://[region]-[project].cloudfunctions.net/[lambda]/connect/google

And the redirect URL of your OAuth App have to be set to:

https://[region]-[project].cloudfunctions.net/[lambda]/connect/google/callback

Vercel

Lastly, we're going to take a look at the Dynamic State Overrides using Vercel:

var grant = require('grant').vercel({
  config: require('./config.json'),
  session: {secret: 'grant', store: require('../store')}
})

module.exports = async (req, res) => {
  if ('/connect/google' === req.url) {
    var state = {dynamic: {scope: ['openid']}}
  }
  else if ('/connect/twitter' === req.url) {
    var state = {dynamic: {key: 'CONSUMER_KEY', secret: 'CONSUMER_SECRET'}}
  }

  var {response, session} = await grant(req, res, state)

  if (response) {
    await session.remove()
    res.statusCode = 200
    res.setHeader('content-type', 'application/json')
    res.end(JSON.stringify(response))
  }
}

Configuration can be loaded dynamically and passed to the Grant handler as state object.

Dynamic State Overrides are useful alternative to the Dynamic HTTP Overrides, in cases when you want to configure Grant dynamically with potentially sensitive data that you don't want to send over HTTP.

Note, however, that any Dynamic Override configuration set on login is being stored in the session, so using external Session Store when overriding sensitive configuration is highly recommended.

Similar to the Google Cloud Function handler, the Grant handler for Vercel handles the HTTP redirects internally. The redirect variable is still being returned by the handler when Grant is going to perform an HTTP redirect, but its value is set to a boolean true instead.

Lastly, all Serverless Grant handlers also return the session instance, used to manage the state that Grant stores in the Cookie or Session Store.

Conclusion

With all of the Serverless Compute offerings out there we can have our API running in no time and pay for what we use only.

Using Grant on top of any of those Cloud Providers can give us Social Login to any OAuth Provider basically for free.

All Serverless handler examples can be found here: aws, azure, gcloud, vercel

Happy Coding!

OAuth Like a BOSS

simo — Mon, 07 Jan 2019 12:16:05 +0000

In my previous article I talked about Grant:

An OAuth middleware for NodeJS that also happens to be a completely transparent OAuth proxy.

This time around we're going to explore a few real world examples:

The Usual Stuff

Imagine that we have a web app hosted on awesome.com Imagine also that our app leverages the GitHub API to manage our user's repositories. We also have a web form on our website to allow our users to pick only the permissions they are willing to grant to our app:

<form action="/connect/github" method="POST">
  <p>Grant read/write access to:</p>
  <label>
    <input type="radio" group="scope" name="scope" value="repo" />
    public and private repositories</label>
  <label>
    <input type="radio" group="scope" name="scope" value="public_repo" />
    public repositories only</label>
  <button>Login</button>
</form>

This form is going to POST the chosen OAuth scope to the /connect/github route, that Grant operates on.

Next we need a Grant server to handle the OAuth flow for us:

var express = require('express')
var session = require('express-session')
var parser = require('body-parser')
var grant = require('grant-express')

express()
  .use(session({secret: 'dev.to'}))
  .use(parser.urlencoded()) // only needed for POST requests
  .use(grant(require('./config.json')))
  .use('/login', (req, res) => res.end(`the above HTML form`))
  .use('/hello', (req, res) => {
    var {access_token} = req.session.grant.response
    console.log(access_token)
    res.end('nice!')
  })
  .listen(3000)

With the following configuration:

{
  "defaults": {
    "origin": "https://awesome.com", "state": true, "transport": "session"
  },
  "github": {
    "key": "...", "secret": "...", "dynamic": ["scope"], "callback": "/hello"
  }
}

We are allowing the OAuth scope to be set dynamically for GitHub. We are also going to use the session as transport to deliver the result of the OAuth flow in our final callback route.

Lastly, we have to create an actual OAuth app on GitHub and copy/paste its key and secret to the configuration above. We also have to set its Authorization callback URL to https://awesome.com/connect/github/callback, that's the second route reserved by Grant.

This will allow us to pick a scope and login with GitHub by navigating to https://awesome.com/login

Login from Another Host

Now imagine that we have another app hosted on GitHub Pages at https://simov.github.io/stars/, that allows users to explore stats and history about the Stars received by a given repository hosted on GitHub.

Our app is going to access public data only. Unfortunately the default rate limit imposed by GitHub on their API is 60 HTTP request per hour. If the request is sent along with an access token, however, the rate limit is lifted to up to 5000 HTTP requests per hour.

So, we need to login again, but we already have a Grant server up and running on awesome.com, so why not reuse it:

{
  "defaults": {
    "origin": "https://awesome.com", "state": true, "transport": "session"
  },
  "github": {
    "key": "...", "secret": "...", "dynamic": ["scope"], "callback": "/hello",
    "overrides": {
      "stars": {
        "key": "...", "secret": "...", "dynamic": ["callback"], "transport": "querystring"
      }
    }
  }
}

We want to have a sub configuration for GitHub, called stars. It will be a different OAuth app, notice the key and the secret.

We also want to set the final callback URL dynamically, but not the scope allowed above it. We're going to login without any explicit scopes set, which in the case of GitHub means getting read access to public data only.

Lastly, we're overriding the transport inherited from the defaults. We need the response data encoded as querystring in a fully qualified absolute callback URL, pointing back to our browser app hosted on GitHub Pages.

Then we have to navigate to the connect route to login:

// store the current URL
localStorage.setItem('redirect', location.href)
// set callback URL dynamically - https://simov.github.io/stars/
var callback = encodeURIComponent(location.origin + location.pathname)
// navigate to the connect route
location.href = `https://awesome.com/connect/github/stars?callback=${callback}`

It's handy to have our final callback URL set dynamically in case we want to host our app on a different domain in future.

After logging in, the user will be redirected back to our browser app, hosted on GitHub:

https://simov.github.io/stars/?access_token=...

Now is the time to extract the access_token from the querystring and store it for future use:

var qs = new URLSearchParams(location.search)
localStorage.setItem('token', qs.get('access_token'))

It's a nice final touch to redirect our users back to where they were before embarking on the quest to login with our OAuth app:

location.href = localStorage.getItem('redirect') // go back to the last URL
localStorage.removeItem('redirect')

Here is the app I was talking about.

Login from Browser Extension

Next we have a Browser Extension that augments the GitHub's UI by adding a cool little button to it. When clicked, it aggregates some useful information about the repository we are currently browsing:

{
  "manifest_version": 2,
  "name": "Cool Little Button",
  "version": "1.0.0",
  "background" : {"scripts": ["background.js"]},
  "content_scripts": [{"matches": ["https://github.com/*"], "js": ["content.js"]}],
  "permissions": ["storage"]
}

This extension, however, relies on data fetched from the GitHub API, that by default will, again, limit us to 60 HTTP requests per hour. So it will be nice to let our users login quickly and easily, directly from within our extension, and therefore lift the rate limit to up to 5000 HTTP requests per hour:

{
  "defaults": {
    "origin": "https://awesome.com", "state": true, "transport": "session"
  },
  "github": {
    "key": "...", "secret": "...", "dynamic": ["scope"], "callback": "/hello",
    "overrides": {
      "stars": {
        "key": "...", "secret": "...", "dynamic": ["callback"], "transport": "querystring"
      },
      "extension": {
        "dynamic": false, "transport": "querystring",
        "callback": "https://github.com/extension/callback"
      }
    }
  }
}

This time we are going to reuse the OAuth app inherited from the root level of the GitHub configuration (the key and the secret). We also don't want the user to set any of the configuration options dynamically.

Then we can open up a new tab from within the background.js script and let our users login:

chrome.tabs.create({url: 'https://awesome.com/connect/github/extension')})

We will redirect our users back to a non existing page on GitHub. In this case GitHub will respond with generic HTML page for the 404 Not Found status code, but our access token will be encoded in the querystring:

https://github.com/extension/callback?access_token=...

So again, all we need to do is extract it and store it, this time by placing the following code into the content.js script:

var qs = new URLSearchParams(location.search)
chrome.storage.sync.set({token: qs.get('access_token')})

Login from Another Language

Grant is not limiting us to JavaScript and NodeJS on the server. Being able to be configured dynamically over HTTP, and sent back the result wherever we want to, Grant gives us access to 180+ login providers from any other programming language.

This time around we'll name our sub configuration proxy, allowing dynamic configuration of every option available in Grant:

{
  "defaults": {
    "origin": "https://awesome.com", "state": true, "transport": "session"
  },
  "github": {
    "key": "...", "secret": "...", "dynamic": ["scope"], "callback": "/hello",
    "overrides": {
      "stars": {
        "key": "...", "secret": "...", "dynamic": ["callback"], "transport": "querystring"
      },
      "extension": {
        "dynamic": false, "transport": "querystring",
        "callback": "https://github.com/extension/callback"
      },
      "proxy": {
        "dynamic": true
      }
    }
  }
}

For the purpose of this example I'm going to use Go, I just had to pick one, but the following applies to any other language:

package main
import (
  "fmt"
  "net/url"
  "net/http"
)
func main() {
  http.HandleFunc("/login", func (w http.ResponseWriter, r *http.Request) {
    qs := url.Values{}
    qs.Add("key", "...") // yes
    qs.Add("secret", "...") // we're passing an OAuth app dynamically!
    qs.Add("scope", "repo user gist")
    qs.Add("transport", "querystring")
    qs.Add("callback", "http://localhost:3000/hello")
    http.Redirect(w, r, "https://awesome.com/connect/github/proxy?" + qs.Encode(), 301)
  })
  http.HandleFunc("/hello", func (w http.ResponseWriter, r *http.Request) {
    qs, _ := url.ParseQuery(r.URL.RawQuery)
    fmt.Println(qs["access_token"][0])
    w.Write([]byte("nice!"))
  })
  http.ListenAndServe(":3000", nil)
}

Now all we need to do is navigate to http://localhost:3000/login

Conclusion

Well, I can summarize this entire article in 5 lines of JSON configuration.

The point of it, however, was to demonstrate how the server side login can be connected with various types of applications, and how Grant fits in the broader picture.

Hopefully this will serve as a handy guide for frontend developers wanting the leverage the servers side login, and also backend developers wanting to quickly get access to a lot of providers.

Here is the source code of all examples.

Happy Coding!

OAuth Simplified

simo — Mon, 10 Dec 2018 12:01:03 +0000

We often need to implement some sort of login functionality in our web apps, or authorize our app to access the user's data programmatically.

Historically this was done by using the user's password directly for authorization. Unfortunately this leads to alot of problems, so a delegated authorization framework was invented, called OAuth.

Two versions of OAuth are available: OAuth 1.0a and OAuth 2.0, the latter being the most widely used today. Both OAuth 1.0a and the so called server side flow (Authorization Code Grant) of OAuth 2.0 require a server app, and involve multiple steps to improve security.

This makes these two types of authorization flows a bit more complicated to implement, so naturally, we look for third-party modules ready to drop in and get going.

Login

A few years ago I was using Passport to implement login functionality using OAuth in my NodeJS app. The problem was, however, that Passport was never built around the idea of having multiple login providers in a single app. While totally possible it required separate module for every provider, developed by a different person.

On top of that Passport makes one additional request after the OAuth flow to get a more detailed profile of the user. Unfortunately that's not part of the specification, and over time it become an obstacle, because I always needed something slightly different.

I ended up with fair amount of glue code in my app, just for a handful of login providers. And to make matters worse - now I had to write tests for it.

So I asked myself:

Is it possible to create something dramatically simpler?

You see, it doesn't matter if something already exists:

You can innovate always!

You can always take already existing solution and create a better one, a simpler one, unique in its own way.

Meet Grant!

Goals

Grant was built around very narrow and extreme use case:

Support for unlimited number of OAuth providers, without the need of any glue code.

As we all know good configuration data structure is the backbone of every great app. At its core Grant is exactly that:

A simple, yet powerful configuration data structure.

Having all aspects of the module defined as JSON, opens up the doors for all kinds of interesting use cases:

static configuration per environment
nested static sub configurations per provider
dynamic configuration per authorization attempt

In fact, the configuration is not even required, it can be passed dynamically via GET or POST request:

Making Grant a completely transparent OAuth proxy.

Meaning that you can deploy it somewhere on your stack and access it from any other server and programming language.

How

First we need to register an OAuth app for every provider that we want to login with. For example, we have to register OAuth 2.0 client app for Google, and OAuth 1.0a client app for Twitter. The redirect URL of our OAuth apps should always end with /connect/[provider]/callback:

http://localhost:3000/connect/google/callback
http://localhost:3000/connect/twitter/callback

Then we need a configuration file where we can put our OAuth app credentials along with a few other options:

{
  "defaults": {"origin": "http://localhost:3000", "callback": "/hello", "state": true},
  "google": {"key": "...", "secret": "...", "scope": ["openid"], "nonce": true},
  "twitter": {"key": "...", "secret": "..."}
}

Lastly, our server may look like this:

var express = require('express')
var session = require('express-session')
var grant = require('grant-express')

express()
  .use(session({secret: 'dev.to'}))
  .use(grant(require('./config.json')))
  .use('/hello', (req, res) => res.end(JSON.stringify(req.query, null, 2)))
  .listen(3000)

This will allow us to login with Google and Twitter by navigating to the following URLs in our browser:

http://localhost:3000/connect/google
http://localhost:3000/connect/twitter

Conclusion

OAuth has never been easier!

And to prove that Grant comes with support for 180+ login providers, but any other provider conforming to the OAuth spec should work out of the box.

Grant was also developed along with an example app showcasing the design goals behind it, and its real potential. As you can imagine this app consists of almost no code on the server, and only a JSON configuration.

Lastly, the official documentation of the module is a great source of information where I try to cover diverse set of features and use cases.

Happy Coding!

Composable HTTP Client for NodeJS

simo — Mon, 24 Sep 2018 13:03:44 +0000

So I made this HTTP client for NodeJS:

var compose = require('request-compose')

BTW if you want to skip ahead and immerse yourself in the FP goodness right away - look no further

And how is it used?

var {res, body} = await compose.client({
  url: 'https://api.github.com/users/simov',
  headers: {
    'user-agent': 'request-compose'
  }
})

Oh WOW! REALLY!?
Yet another HTTP Client for NodeJS!
A M A Z I N G ! ! !

.. APIs, APIs .. Everywhere

As an end user, what if I want something fixed, changed or added in someone else's module? What are my options?

Open up an issue on GitHub and ask for it
Implement it myself and submit a pull request
Search for alternative module that have what I need
Repeat

The reason why is because the module authors present you with an API about what you can do, and what you cannot. You are essentially locked in. The authors also fiercely guard the scope of their project from something unrelated creeping in.

But what if we had more powerful primitives allowing us to step one layer below, and elegantly compose our own thing. Just for ourselves, completely bypassing the API and scope bottleneck presented in the other's solution.

Composition

Luckily there is such primitive called Functional Composition:

In computer science, function composition (not to be confused with object composition) is an act or mechanism to combine simple functions to build more complicated ones. Like the usual composition of functions in mathematics, the result of each function is passed as the argument of the next, and the result of the last one is the result of the whole.

In fact, what request-compose exposes is exactly that:

var compose = (...fns) => (args) =>
  fns.reduce((p, f) => p.then(f), Promise.resolve(args))

At its core request-compose is not even a client, it's a Functional Programming pattern, an idea, a simple one-liner to help you compose your own thing.

Whit it you can compose any function, asynchronous or not:

var sum = compose(
  (x) => x + 1,
  (x) => new Promise((resolve) => setTimeout(() => resolve(x + 2), 1000)),
  (x) => x + 3,
  async (x) => (await x) + 4
)
await sum(5) // 15 (after one second)

Or being slightly more on the topic - compose your own HTTP client:

var compose = require('request-compose')
var https = require('https')

var request = compose(
  (options) => {
    options.headers = options.headers || {}
    options.headers['user-agent'] = 'request-compose'
    return options
  },
  (options) => new Promise((resolve, reject) => {
    https.request(options)
      .on('response', resolve)
      .on('error', reject)
      .end()
  }),
  (res) => new Promise((resolve, reject) => {
    var body = ''
    res
      .on('data', (chunk) => body += chunk)
      .on('end', () => resolve({res, body}))
      .on('error', reject)
  }),
  ({res, body}) => ({res, body: JSON.parse(body)}),
)

var {res, body} = await request({
  protocol: 'https:',
  hostname: 'api.github.com',
  path: '/users/simov',
})

Can you spot the API?
There is none.
It's all yours, your own Promise based HTTP Client.
Congratulations!

Practicality

That's cool and all but not very practical. After all we usually try to extract code into modules, not coding everything up in one place.

And why you'll even bother using request-compose if you have to do all the work by yourself.

The answer is simple:

You can choose what you want to use, extend it however you want to, or don't use it all - compose your own thing from scratch.

There are a bunch of functions, however, cleverly named middlewares that encapsulate pieces of HTTP client logic that you may find useful:

var compose = require('request-compose')
var Request = compose.Request
var Response = compose.Response

var request = compose(
  Request.defaults({headers: {'user-agent': 'request-compose'}}),
  Request.url('https://api.github.com/users/simov'),
  Request.send(),
  Response.buffer(),
  Response.string(),
  Response.parse(),
)

var {res, body} = await request()

It's important to note that these middlewares are just an example of a possible implementation. My own implementation. But you are not locked into it, because it's not hidden behind API walls.

You are free to compose your own thing:

var compose = require('request-compose')
var Request = compose.Request
var Response = compose.Response

var request = (options) => compose(
  Request.defaults(),
  // my own stuff here - yay!
  ({options}) => {
    options.headers['user-agent'] = 'request-compose'
    options.headers['accept'] = 'application/vnd.github.v3+json'
    return {options}
  },
  // base URL? - no problem!
  Request.url(`https://api.github.com/${options.url}`),
  Request.send(),
  Response.buffer(),
  Response.string(),
  Response.parse(),
)(options)

var {res, body} = await request({url: 'users/simov'})

Put that in a module on NPM and call it a day.

Full Circle

Having separate middlewares that we can arrange and extend however we want to is great, but can our code be even more expressive and less verbose?

Well, that's the sole purpose of the compose.client interface to exist:

var {res, body} = await compose.client({
  url: 'https://api.github.com/users/simov',
  headers: {
    'user-agent': 'request-compose'
  }
})

And as you may have guessed the options passed to compose.client are merely composing the HTTP client under the hood using the exact same built-in middlewares.

Going BIG

Lets take a look at the other side of the coin - instead of laser focusing on the HTTP internals - we can ask ourselves:

How can we use the Functional Composition to build something bigger?

How about composing a Higher-Order HTTP Client:

var compose = require('request-compose')

var search = ((
  github = compose(
    ({query}) => compose.client({
      url: 'https://api.github.com/search/repositories',
      qs: {q: query},
      headers: {'user-agent': 'request-compose'},
    }),
    ({body}) => body.items.slice(0, 3)
      .map(({full_name, html_url}) => ({name: full_name, url: html_url})),
  ),
  gitlab = compose(
    ({query, token}) => compose.client({
      url: 'https://gitlab.com/api/v4/search',
      qs: {scope: 'projects', search: query},
      headers: {'authorization': `Bearer ${token}`},
    }),
    ({body}) => body.slice(0, 3)
      .map(({path_with_namespace, web_url}) =>
        ({name: path_with_namespace, url: web_url})),
  ),
  bitbucket = compose(
    ({query}) => compose.client({
      url: 'https://bitbucket.org/repo/all',
      qs: {name: query},
    }),
    ({body}) => body.match(/repo-link" href="[^"]+"/gi).slice(0, 3)
      .map((match) => match.replace(/repo-link" href="\/([^"]+)"/i, '$1'))
      .map((path) => ({name: path, url: `https://bitbucket.org/${path}`})),
  ),
  search = compose(
    ({query, cred}) => Promise.all([
      github({query}),
      gitlab({query, token: cred.gitlab}),
      bitbucket({query}),
    ]),
    (results) => results.reduce((all, results) => all.concat(results)),
  )) =>
    Object.assign(search, {github, gitlab, bitbucket})
)()

var results = await search({query: 'request', {gitlab: '[TOKEN]'}})

Now you have an HTTP Client that simultaneously searches for repositories in GitHub, GitLab and BitBucket. It also returns the results neatly packed into Array, ready to be consumed by your frontend app.

Wrap it up in a Lambda and deploy it on the Cloud.
That's your Serverless!

Conclusion

What if we had modules that doesn't lock us in? What if there is no API, or one that's completely optional and extendable. What if we had tools that empower us to be the author ourselves, and built our own thing that's best for us.

The idea behind request-compose is exactly that, plus it is a fully featured, and functional functional (get it?) HTTP Client for NodeJS. Or rather should I say: it contains an opinionated HTTP client bundled in it. It covers most of the use cases that you may encounter, and it's far from a toy project, nor is my first HTTP Client.

Not saying it's the best one, but just so you know :)

Happy Coding!

Anonymous Recursion in JavaScript

simo — Sun, 06 Aug 2017 10:46:09 +0000

(
  (
    (f) => f(f)
  )
  (
    (f) =>
      (l) => {
        console.log(l)
        if (l.length) f(f)(l.slice(1))
        console.log(l)
      }
  )
)
(
  [1, 2, 3]
)

Yes, there is such thing, and I thought it would be an interesting example to share. It features: closures, self-executing functions, arrow functions, functional programming, and anonymous recursion.

You can copy/paste the above example in your browser's console. The output is as follows:

[ 1, 2, 3 ]
[ 2, 3 ]
[ 3 ]
[]
[]
[ 3 ]
[ 2, 3 ]
[ 1, 2, 3 ]

Speaking of functional programming, here is how similar example look like in Scheme (one of the languages JavaScript was influenced by):

(
  (
    (lambda (f) (f f))
    (lambda (f)
      (lambda (l)
        (print l)
        (if (not (null? l)) ((f f) (cdr l)))
        (print l)
      )
    )
  )
  '(1 2 3)
)

Unwind

Like in many other programming languages calling a function is done by appending parentheses () after its name:

function foo () { return 'hey' }
foo()

In JavaScript we can wrap any number of expressions in parentheses:

('hey', 2+5, 'dev.to')

The result of the above snippet is 'dev.to'. The reason why is because JavaScript returns the last expression as result.

Wrapping a single anonymous (lambda) function in parentheses () means that the result will be the anonymous function itself:

(function () { return 'hey' })

That in itself is not very useful because the anonymous function does not have a name, and we won't be able to reference it unless we call it immediately during initialization.

Like a regular function, we can append parentheses () after it to call it:

(function () { return 'hey' })()

The same goes with the arrow functions:

(() => 'hey')()

Again, appending parentheses () after the anonymous function means that we are executing it, also known as self-executing function.

Closures

A closure is the combination of a function and the lexical environment within which that function was declared. Combined with arrow functions we can define it like this:

var foo = (hi) => (dev) => hi + ' ' + dev

Calling the above function in the browser's console will print 'hey dev.to':

foo('hey')('dev.to')

Note that we have access to the hi argument from the outer scope of the enclosing function inside the enclosed inner one.

The above code is identical to:

function foo (hi) {
  return function (dev) { return hi + ' ' + dev }
}

And the self-executing version would be:

(
  (hi) =>
    (
      (dev) => `${hi} ${dev}`
    )
    ('dev.to')
)
('hey')

First the hey parameter is passed to the outermost scope to the above function as hi argument. Then that function returns yet another self-executing function that needs to be evaluated first. The dev.to parameter is then passed as the dev argument to the innermost function, and that function return the final result: 'hey dev.to'.

Going Deep

Here is a slightly modified version of the above self-executing function:

(
  (
    (dev) =>
      (hi) => `${hi} ${dev}`
  )
  ('dev.to')
)
('hey')

First the hey parameter is passed as argument to the outermost scope, but instead of a function there we have yet another expression that needs to be evaluated first. So the dev.to parameter is then passed to the inner self-executing function as the dev argument and returns another function. That last function is what satisfies the outermost scope and therefore receives the hey parameter.

It's important to note that the self-executing functions and closures are used to initialize and encapsulate state, and this is what we're going to use in our next example.

Anonymous Recursion

Going back to our initial example, this time annotated:

(
  (
    (f) => f(f) // 3.
  )
  (
    (f) => // 2.
      (l) => { // 4.
        console.log(l)
        if (l.length) f(f)(l.slice(1))
        console.log(l)
      }
  )
)
(
  [1, 2, 3] // 1.
)

The input array [1, 2, 3] is passed to the outermost scope
This entire function is passed as argument to the function above
This function receives the bottom one as argument f and calls it with itself
2. being called in 3. results in returning the 4. function which is the one that satisfies the outermost scope and therefore receives the input array as the l argument

The reason for all of this is to have a reference to the f function inside the recursive one that receives the input array l. That way we can call it:

f(f)(l.slice(1))

Note that f is a closure so we need to call it with itself just to get access to the innermost function that operates on the input array.

For explanatory purposes the first console.log(l) statement represents the recursive top-down, and the second one the recursive bottom-up.

Conclusion

I hope you enjoyed this article and learned something new out of it. Closures, self-executing functions and functional programming patterns are not black magic. They follow simple principles that are easy to understand and fun to play with.

That being said you have to develop a sense of your own when to use them, or not. If your code becomes harder to maintain, then it's probably a good idea to refactor it a little bit.

Nevertheless understanding these fundamental techniques is pivotal to creating clean and elegant solutions, as well as leveling up.

Happy Coding!