DEV Community: Pierre Carion

HTTPS for your local endpoints for less than $1 a month

Pierre Carion — Tue, 04 Feb 2020 00:13:13 +0000

Originally published at https://pcarion.com/article/ssh-tunnel

There are quite a few instances when you want a public URL to hit your development machine:

you want to expose a webserver running on your local machine to the internet so that a colleague or a customer may have a look at it.
you are using a service, like twillio, which allows you to setup webhooks URL: the service will call those URLs to notify you when something happens.
you want to test an OAuth integration, with Facebook or Twitter, and you want to provide public https callback URLs.

An alternative option is to install your code on a public server and have those requests hit this server. During development, it is much more convenient to have those requests hit your local machine so that you can debug and see the logs in real time.

You can also use a service like ngrok: this is a tool, developed by Alan Shreve, very easy to use and perfect for webhooks.

We'll describe here how you can setup your own solution on ec2.

Such a home made solution has several advantages:

you can have persistent URLs - ngrok gives a different domain each time you use it unless you become a paying customer.
you can automate the process (see the end of this post) meaning that you can quickly tear down your EC2 instance and restart it quickly when you need the URLs again.

At $0.0069 per hour for a nano instance, the service will cost you $1 per month if you use it for about 144 hours. This is more than enough for a "night and weekend" project.

Description of the solution

The solution looks like this:

A machine (C) on the public internet will be able to access multiple URLs, on the same public URL, to access one or more services on your local machine (A).

For instance:

the URL https://api.mydomain.com will access http://localhost:3000 on your dev machine
the URL https://api.mydomain.com will access http://localhost:3000 on your dev machine

Warning: a custom made solution is definitely more complex than using ngrok, so you should think twice before taking that route. You need to be comfortable with bash scripting and DNS setup to implement that solution.

This solution requires:

a domain name: this domain, and subdomains, will be used to configure public URL(s) to access your local machine
a public DNS: as you want to have public URL to access your server, a DNS is required.
a machine on the public internet: this machine will act as a bridge between the public internet and your local machine. You can use Digital ocean or AWS EC2, with full root access
a sshd daemon running on that server: SSH is the swiss army knife for that kind of setup, and you need to be able to fully configure the SSH server, especially to setup a reverse tunnel
a SSL certificate. To protect your connection, you need to setup SSL certificates so that the public URL can be available only through https. We will use Let's encrypt for that.

The Gateway server (B) is the machine on the public internet and you need to configure a set of services on that server:

configure the SSD service to allow TCP port forwarding
install a reverse proxy - HAProxy - to forward different subdomain to different ssh tunnels
configure let's encrypt to allow SSL traffic over https

In the rest of this document, we will use EC2 to install this gateway server.

EC2 instance configuration

The setup is not intended for a production service and the traffic on that machine should be very low. For that reason, you can use a very small instance to run that server : I am using a t2.nano instance with an ubuntu OS.

security group

The only setup to pay attention to is the network/security group definition.

By default, the setup would allow only the port 22 for your ssh access:

You need more inbound ports for your server:

22 for your SSH access
80 for your incoming http access
443 for your incoming https access
different ssh tunnels ports, like 8080, 8085, 8090, etc... The number of ports depend on the number of services you want to expose through the ssh tunnel

You can create a security group with those inbound ports:

private key file

To use the SSH connection, you need to create a key pair . Download the associated .pem file, copy it in a safe place and do a chmod 400 on that file to avoid the error:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for 'ssh_tunnel.pem' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "ssh_tunnel.pem": bad permissions

server configuration

At the end of this document, we will show how to automate entirely the setup process.

The following description is useful if you want to have a better understanding of what the automated script will be doing.

Using your pem file, you can ssh to the server and proceed with the configuration.

First, make sure that all the packages are up to date

sudo apt-get update

setup the sshd server for the reverse ssh tunnel

You need to make sure that the sshd server is running and has the proper configuration.

This command will check the sshd configuration:

$ sshd -T | grep -E 'gatewayports|allowtcpforwarding'
gatewayports no
allowtcpforwarding yes

For the ssh tunnels to work, you need both parameters to be set to yes.

Open the file /etc/ssh/sshd_config and add or set those 2 lines:

AllowTcpForwarding yes
GatewayPorts yes

Then, you need to restart the ssh daemon for those parameters to be taken into account:

service sshd restart

And then you can check that all is in order:

$ sshd -T | grep -E 'gatewayports|allowtcpforwarding'
gatewayports yes
allowtcpforwarding yes

Test of the reverse ssh tunnel setup

Before moving forward with the server configuration, you can already check that the ssh setup is working properly.

In order to test that a local service can be reached from the public internet, you need to start some sort of local server on your development machine.

A simple HTTP server will do the trick: from a directory containing no sensitive data, you can start a simple python server:

python -m SimpleHTTPServer 8000

You can check from your local browser from http:127.0.0.1:8000 that the server is working, and ... not exposing sensitive data.

Now is the time to invoke the SSH port forwarding voodoo incantation, from your local machine:

ssh -i "tunnel.pem" ubuntu@ec2-a-b-c-d.us-west-1.compute.amazonaws.com -N -R 8080:localhost:8000

This SSH command is much simpler than it looks like:

ubuntu@ec2-a-b-c-d.us-west-1.compute.amazonaws.com this is the public DNS Address of your server and you can find the value in your aws/ec2 console
-N: by default, ssh will create a shell on the remote machine. We don't need that here
-R: with this option you are asking ssh to answer on the remote side (your gateway)
8080:localhost:8000: any connection on port 8080 on the gateway will be tunneled to the the port 8000 on your local machine (where the webserver we started previously is listening on).

If all work as expected, you can open your browser at: http://ec2-a-b-c-d.us-west-1.compute.amazonaws.com:8080

and you should see the file served by your local server!

Your reverse ssh tunnel is working.

More information about reverse ssh tunneling can be found here.

DNS setup

The next step is that you want a nicer URL to access your service right?

You need to configure the DNS for your domain yourdomain.com and create a A record for api.yourdomain.com , with a TTL of 600, with a value of a.b.c.d which is the IP address of your gateway.

I may take time for the DNS configuration to propagate, but once it is done, you can then access your local web server through the URL: http://api.yourdomain.com:8080.

Better, but can still be improved: you may want to setup multiple subdomains which would allow you to host multiple local services, or have multiple machines using this tunnel (each using a specific subdomain).

To do that, you need a reverse proxy on your gateway.

HAProxy setup

Let's install HAProxy:

$ install haproxy
$ sudo apt-get install -y haproxy
$ haproxy -v
HA-Proxy version 1.8.8-1ubuntu0.9 2019/12/02
Copyright 2000-2018 Willy Tarreau <willy@haproxy.org>

You then need to configure its main configuration file: /etc/haproxy/haproxy.cfg

The setup is pretty basic: base on the domain being accessed (like api.yourdomain.com), you serve data from a local server (127.0.0.1:8080), which, through the ssh reverse tunnel will connect back to your local machine

Example of subdomain setting in haproxy.cfg

frontend account
    bind *:80
    mode http
    acl host_api hdr(host) -i api.yourdomain.com
    use_backend account if host_api

backend api
    mode http
    server node1 127.0.0.1:8080

With that configuration, you can then verify that the URL http://api.yourdomain.com serves also your local HTTP server data.

We won't go any further yet as the configuration of HAProxy is very dependent on the next step.

SSL - letsencrypt

Nowadays, https is almost always required to access a server.

You can easily get a free SSL certificate using Let's encrypt.

The installation of the required tool is easy:

sudo add-apt-repository -y ppa:certbot/certbot
sudo apt-get update
sudo apt-get install -y certbot

There is a rate limiting with let's encrypt services so... you are limited in the number of trial and errors to configure your certificates.

There are 2 conditions to ensure before starting the let's encrypt setup:

you need to make sure that the DNS setup is done and propagated for http://yourdomain.com domain name - the let's encrypt server relies on that to ensure that you are the rightful owner of the domain
you must stop haproxy or any service using the port 80 as this port will be used by let's encrypt, using their own cerbot server to retrieve the certificate

Once those conditions are met, you can start the certificate retrieval process:

sudo certbot certonly --standalone -d yourdomain.com -d api.yourdomain.com -d www.yourdomain.com --non-interactive --agree-tos --email you@email.com

You need to change the command line above with:

the list of subdomains you want to have a certificate for in the -d arguments
your email address as the -email argument

If all goes well, you should see this kind of output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for yourdomain.com
http-01 challenge for api.yourdomain.com
http-01 challenge for www.yourdomain.com
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/yourdomain.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/yourdomain.com/privkey.pem
   Your cert will expire on 2020-04-25. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

import certificates to HAProxy

The certificates generated above will be used by HAProxy and they need some massaging to be usable there:

sudo mkdir /etc/haproxy/certs
DOMAIN='yourdomain.com' sudo -E bash -c 'cat /etc/letsencrypt/live/$DOMAIN/fullchain.pem /etc/letsencrypt/live/$DOMAIN/privkey.pem > /etc/haproxy/certs/$DOMAIN.pem'

The last step is to configure HAProxy, with all your domain and the https setup.

We will cover that in the next section

Automation

As you can see, there are a lot of sets involved to set up a server but there is a way to automate the entire process.

ec2 template

You can speed up the creation of the ec2 instance by using a template.

You can create a template from the instance you just created and make sure that:

you are using the right security group with all the inbound ports you want to use
you use the key-pair previously created

With that template, the creation of the instance becomes very easy with basically one click.

automation steps

create an instance using the ec2 instance template
once the instance is booted, connect to it using your pem file to make sure your ssh setup is correct
retrieve the IP address of this new instance and update your DNS setup. Confirm with a DNS lookup that the IP address has been updated
you can confirm that the DNS is correct ifyou can connect to your instance with your domain name:

ssh -i "tunnel.pem" ubuntu@yourdomain.com
update the following script (see below) and run it with:

ssh -i "tunnel.pem" ubuntu@yourdomain.com 'bash -s' < setup_tunnel_host.sh

automation script - setup_tunnel_host.sh

Here is the script that you can run to automate all the steps describe above...

You need to set a couple of variables at the top of that script

this script will setup a SSH tunnel for you domain: BASE_DOMAIN_NAME
your email address for let's encrypt: LETE_EMAIL
the script setup 2 subdomains with 2 ports (it's easy to update the script to use more ports/subdomains):
- SUBDOMAIN1/ REMOTE_REDIRECT_PORT1
- SUBDOMAIN2/ REMOTE_REDIRECT_PORT2

On your local machine, you can then initiates both tunnels with:

ssh -i "tunnel.pem" [ubuntu@yourdomain.com](mailto:ubuntu@chirloute.com) -N -R 8080:localhost:8000 
ssh -i "tunnel.pem" [ubuntu@yourdomain.com](mailto:ubuntu@chirloute.com) -N -R 8090:localhost:3000

You must update the local port to match the server running on your local machine.

Enjoy!

How to create a resume with Gatsbyjs

Pierre Carion — Sun, 26 Jan 2020 19:36:45 +0000

Originally published at https://pcarion.com/article/gatsby-resume

Online resume

There are a lot of reasons to use a static site generator to generate your resume online:

it is always nice to have a fast loading time
you don't want to take any chance with the site being down when a prospective employer accesses your resume
it is cheap to deploy: the load on the page should be low, so a free hosting solution like Netlify makes sense

For all those reasons, Gatsby makes a good choice as a platform to publish your resume online.

This article will do exactly that:

the resume we will build is deployed with Netlify at https://resume-with-gatsby.netlify.com/
you can find the source code on GitHub at https://github.com/pcarion/resume-with-gatsby

Resume definition format

You will update the content of your resume regularly, so you really don't want to sprinkle this content inside a static HTML template.

The JSON Resume is an open-source project to create a JSON-based standard for resumes.

Personally, I find that a JSON file is more difficult to update than a yaml file, so we'll use the format defined by the JSON resume project, but in a YAML format.

The project comes with a sample resume, from the fictional character Richard Hendriks from the excellent series "Silicon Valley" on HBO:

https://github.com/jsonresume/resume-schema/blob/v1.0.0/examples/valid/complete.json

Using a YAML format, the resume looks like this:

---
basics:
  name: Richard Hendriks
  label: Programmer
  image: ''
  email: richard.hendriks@mail.com
  phone: "(912) 555-4321"
  url: http://richardhendricks.example.com
  summary: Richard hails from Tulsa. He has earned degrees from the University of
    Oklahoma and Stanford. (Go Sooners and Cardinal!) Before starting Pied Piper,
    he worked for Hooli as a part time software developer. While his work focuses
    on applied information theory, mostly optimizing lossless compression schema of
    both the length-limited and adaptive variants, his non-work interests range widely,
    everything from quantum computing to chaos theory. He could tell you about it,
    but THAT would NOT be a “length-limited” conversation!
  location:
    address: 2712 Broadway St
    postalCode: CA 94115
    city: San Francisco
    countryCode: US
    region: California

Gatsby setup

To install Gatsby, you start with installing the 3 required packages

**yarn init -y
yarn add gatsby react react-dom**

You can then add the usual scripts in your package.json file to start the Gatsby server:

"scripts": {
  "build": "gatsby build",
  "clean": "gatsby clean",
  "develop": "gatsby develop"
},

At this stage, you can already start Gatsby with yarn develop, but there is nothing available for rendering.

Loading the resume in Gatsby

The gatsby-source-filesystem plugin allows you to load any file in Gatsby.

yarn add gatsby-source-filesystem

To configure this plugin, you create a gatsby-config.js file where you tell the plugin which directory to parse to read files.

module.exports = {
  plugins: [
    {
      resolve: `gatsby-source-filesystem`,
      options: {
        path: `./resume`,
      },
    },
  ],
}

You can then start Gatsby (yarn run develop) and load the Graphiql server, usually at that URL: http://localhost:8000/___graphql

The gatsby-source-filesystemwill populate the allFile root with the files it found, as shown by this query:

query MyQuery {
  allFile {
    edges {
      node {
        absolutePath
      }
    }
  }
}

the results will show your resume file:

{
  "data": {
    "allFile": {
      "edges": [
        {
          "node": {
            "absolutePath": "<path to>/resume/resume.yaml"
          }
        }
      ]
    }
  }
}

Exploration

The step here is not required, but it shows how you could write your own plugin to parse the resume.

In the gatsby-node.js file, you could write a very basic plugin to read the content of the resume, using the onCreateNode method to later create new nodes to expose your resume in the GraphQL API:

async function onCreateNode({ node, loadNodeContent }) {
  if (node.internal.mediaType !== `text/yaml`) {
    return;
  }

  const content = await loadNodeContent(node);
  console.log(`File: ${node.absolutePath}:`);
  console.log(content);
}

exports.onCreateNode = onCreateNode;

When you restart the server, you can see the content of the resume file on screen:

success onPreBootstrap - 0.009s
success createSchemaCustomization - 0.002s
File: <path to>/resume/resume.yaml:
---
basics:
  name: Richard Hendriks
  label: Programmer

Interesting but there are existing plugins who already do that ... and much more.

Transformer plugin: gatsby-transformer-yaml

This plugin is a good candidate for us:

it reads the yaml file
parses it
makes it available if the GraphQL API

You install that plugin with:

yarn add gatsby-transformer-yaml

and the magic happens when you restart the server: 2 new root types appear in the Graphiql interface, allResumeYaml and resumeYaml.

If we suppose that the site will contain only one resume, we can get the content of the resume using that query:

query MyQuery {
  resumeYaml {
    basics {
      email
      name
      phone
    }
  }
}

and, we get back:

{
  "data": {
    "resumeYaml": {
      "basics": {
        "email": "richard.hendriks@mail.com",
        "name": "Richard Hendriks",
        "phone": "(912) 555-4321"
      }
    }
  }
}

Time to render some data

Let's create our first page:

you can delete the gatsby-node.js file containing our exploration plugin.
create a src/pages/index.js

We'll start with the most basic data extraction (name and email) to make sure that all is in order.

In src/pages/index.js :

import React from 'react';

const Resume = ({ data }) => {
  const resume = data.resumeYaml;
  const { basics } = resume;
  return (
    <React.Fragment>
      <h1>{basics.name}</h1>
      <h2>{basics.email}</h2>
    </React.Fragment>
  );
};

export default Resume;

export const query = graphql`
  query MyQuery {
    resumeYaml {
      basics {
        email
        name
      }
    }
  }
`;

This will give us our first page:

Not quite enough yet to get you a job at Hooli, but it's a promising start.

Resume rendering

The JSON resume project contains also a library of themes to render resume following the specification.

For the purpose of this presentation, I have chosen the flat theme by Mattias Erming, because

it is pretty well designed
it is open source (MIT) - https://github.com/erming/jsonresume-theme-flat
the styling is easy to extract: one style.css file and one template file

Process to build the resume

I won't describe all the steps to port that template to Gatsby, just the major steps

extra package: gatsby-react-helmet

Helmet is nice addition because it makes it very easy to:

to set page title
add external css dependencies

You install that package with:

yarn add gatsby-plugin-react-helmet react-helmet

and add it to your gatsby-config.js file.

issue with CSS ordering

The way you add a specif css document (the one coming with the flat theme example), is by importing if in the gatsby-browser.js document:

import "./src/styles/style.css"

The style requires 2 other CSS loaded from a CDN, and you can use Helmet to do that:

<Helmet>
  <title>{basics.name}</title>
  <link
    rel="stylesheet"
    href="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.2.0/css/bootstrap.min.css"
  />
  <link
    rel="stylesheet"
    href="https://cdnjs.cloudflare.com/ajax/libs/octicons/2.0.2/octicons.min.css"
  />

</Helmet>

The problem is that the local style loaded by gatsby-browser.js is loaded before the styles loaded by Helmet and that broke the rendering: for example the H1..4 styles were the ones defined by bootstrap, and not the local styles file.

The only solution I found, was to load bootstrap as a dependency and load it from gatsby-browser.js, before the local style:

import 'bootstrap/dist/css/bootstrap.css'
import "./src/styles/style.css"

porting process

The process involved updating the GraphQL query to import more and more data (the minimal theme only omits the projects section of the resume).

The final query asks for all the data:

query MyQuery {
    resumeYaml {
      basics {
        email
        name
        label
        phone
        url
        summary
        profiles {
          network
          url
          username
        }
      }
      work {
        description
        endDate(formatString: "MMM, YYYY")
        highlights
        location
        name
        startDate(formatString: "MMM, YYYY")
        position
        summary
        url
      }
      volunteer {
        endDate(formatString: "MMM, YYYY")
        highlights
        organization
        position
        startDate(formatString: "MMM, YYYY")
        summary
        url
      }
      education {
        area
        courses
        endDate(formatString: "MMM, YYYY")
        gpa
        institution
        startDate(formatString: "MMM, YYYY")
        studyType
      }
      awards {
        awarder
        date(formatString: "MMM, YYYY")
        summary
        title
      }
      publications {
        name
        publisher
        releaseDate(formatString: "MMM, YYYY")
        summary
        url
      }
      skills {
        keywords
        level
        name
      }
      languages {
        fluency
        language
      }
      interests {
        keywords
        name
      }
      references {
        name
        reference
      }
    }
  }

and each section of the resume has its own component:

const Resume = ({ resume }) => {
  const { basics, work, volunteer, education, awards, publications, skills, languages, interests, references } = resume;
  return (
<React.Fragment>
  <Helmet>
    <title>{basics.name}</title>
    <link
      rel="stylesheet"
      href="https://cdnjs.cloudflare.com/ajax/libs/octicons/2.0.2/octicons.min.css"
    />
  </Helmet>
  <Header basics={basics} />
  <div id="content" className="container">
    <Contact basics={basics} />
    <About basics={basics} />
    <Profiles profiles={basics.profiles||[]} />
    <Work works={work} />
    <Volunteer volunteers={volunteer} />
    <Education educations={education} />
    <Awards awards={awards} />
    <Publications publications={publications} />
    <Skills skills={skills} />
    <Languages languages={languages} />
    <Interests interests={interests} />
    <References references={references} />
  </div>
</React.Fragment>
);

Pretty easy to read and update.

Deployment

We won't describe here the actual deployment as there are a lot of articles out there describing how to deploy a Gatsby site.

For instance, the site https://resume-with-gatsby.netlify.com/ was deployed on Netlify with a couple of clicks.