How to develop and test a local microservice with remote microservices

In this article, we'll discuss how to develop, debug, and test a microservice locally on your laptop, without having to run all the other microservices of an app also on your laptop.

The solution presented in this article could also be used to debug, fix, and test a problem reported by your customer in your live production cluster by routing accesses to your microservice (that has the bug) to the one running in your laptop with the fix.

Problem Statement:

Let's say your team is focussed on developing and testing a front-end ReactJS microservice for your company's application that has like 25 other microservices such as Spring Boot, Elastic Search, Redis, Postgres, Grafana, and so on, in it.

As a dev, you would usually try to run all of these microservices in your laptop so that you could develop,build and test changes to your front-end ReactJS App natively on your laptop. But running all these 25 microservices on your laptop will make it into a toaster, because you are pushing your laptop cpu, memory and power consumption to their maximum capacity throughout the day.

Workarounds Used Today

Many try to workaround this problem by upgrade their RAM or disk or fan module or even the laptop itself.

This adds more costs to your organization. Sometimes it may make you wonder, "Was it really a smart decision to convert your monolithic applications into microservices, in the first place?"

Don't worry, you are not left alone. Many companies that have begin their journey into the microservices world, for the benefits it offers, have faced the exact same problem. And, fortunately, SocketXP has a solution to address this problem.

Wall Power Sockets Analogy

Let me ask you this question. Would you run a huge nuclear power reactor in your house just because you needed some electricity to run your TV, Air Conditioner and Washing machine? The answer would be "no". You'd just install a couple of sockets on your wall, run a power cable to the energy company in your city and hook up your electrical equipments to those wall sockets. Neat and simple isn't it?

Then why do we try to run all our microservices(both 800 pound gorillas and tiny little mice ones) on that poor little laptop on your lap, all day long, just because you need to focus on developing and testing one front-end microservice? You could very well install software "wall sockets" on your laptop and have your ReactJS front-end microservice talk to 25 other microservices running remotely in a cloud (or some on-prem cluster perhaps?) via the software "wall sockets" And vice-versa.

Our SocketXP Microservice Remote Access solution would help create those software "wall-sockets" in your laptop, for all your remote microservices.

How SocketXP Microservice Remote Access Solution Works?

You need to download and run a SocketXP proxy agent (also available as Docker Container) on your laptop and on your remote cluster where other N-1 microservices of your application reside.

SocketXP proxy agent will create "tcp or http" sockets on your laptop (one for each of the microservice running in a remote cluster) and run a bidirectional TLS tunnel between your laptop and your remote cluster. We'd also adjust DNS records (for those remotely running microservices) in your laptop to route to the localhost IP address. So that, any requests from your front-end microservices to these 25 other microservices, would be routed to these local sockets.

Your front-end microservice would continue to connect to other microservices of the app as usual, without requiring any modifications to your code or configurations.*

Docker Container Demo

For this exercise, we are going to run Postgres DB as a Docker Container in a remote server and make a python based backend microservice (under development in my laptop) access the remote DB via the SocketXP proxy.

Postgres DB is accessible via the hostname postgres and port 5432

Here is the simple backend microservice written in python.

$cat backend.py 
import psycopg2

con = psycopg2.connect(database="postgres", user="gvelrajan", password="pa$$word123", host="postgres", port="5432")

print("Database opened successfully")

cur = con.cursor()
cur.execute('''CREATE TABLE STUDENT
      (ADMISSION INT PRIMARY KEY     NOT NULL,
      NAME           TEXT    NOT NULL,
      AGE            INT     NOT NULL,
      COURSE        CHAR(50),
      DEPARTMENT        CHAR(50));''')

print("Table created successfully")

cur.execute("INSERT INTO STUDENT (ADMISSION,NAME,AGE,COURSE,DEPARTMENT) VALUES (34231, 'Dave', 19, 'Information Technology', 'ICT')");
cur.execute("INSERT INTO STUDENT (ADMISSION,NAME,AGE,COURSE,DEPARTMENT) VALUES (34232, 'Gary', 18, 'Electrical Engineering', 'ICT')");
cur.execute("INSERT INTO STUDENT (ADMISSION,NAME,AGE,COURSE,DEPARTMENT) VALUES (34233, 'Abel', 17, 'Computer Science', 'ICT')");

print("Student records added to the table successfully")

con.commit()
con.close()

The python backend microservice connects to the DB using the hostname 'postgres' and port '5432'

Run the Postgres DB Docker Container

First create a docker volume

$ docker volume create postgres

Next run the Postgres DB container and mount the postgres volume in the host machine to the directory /var/lib/postgresql/data inside the container.

$ docker run --name postgres --restart unless-stopped --net appnet --mount source=postgres,target=/var/lib/postgresql/data -e POSTGRES_USER=gvelrajan -e POSTGRES_PASSWORD=pa$$word123 -e POSTGRES_DB=postgres -d postgres:latest

Run SocketXP Proxy Agent container in a Cluster/Server

First go to SocketXP Portal. Signup for a free account and get your authtoken there.

Use your authtoken in the below config file.

$ cat config.json
{
    "authtoken": "<your-auth-token-goes-here>"
    "tunnel_enabled": true, 
    "tunnels" : [{ 
        "destination": "tcp://postgres:5432", 
        "iot_device_id": "postgres-mservice-cluster1", 
    }] 
}

Store the config.json file in a local directory(/home/gvelrajan/config/config.json) and map the directory into the SocketXP docker container at /data as shown in the command below to start the docker container.

docker run --name socketxp --restart unless-stopped --net appnet -d  -v /home/gvelrajan/config:/data expresssocket/socketxp:latest

Execute the following command to check the logs for success status or any errors:

docker logs socketxp

It would have something like this:

Connected.
TCP tunnel [gvelrajan-5bz4y6hi] created.

Now go to the SocketXP Portal's Tunnel Page. Check the name of the tunnel created and its status.

Run SocketXP Proxy Agent on your Laptop

First download the SocketXP Proxy Agent in your laptop from the SocketXP download page. SocketXP Proxy Agent is available for all OS versions - Windows/Mac/Linux.

Use the same SocketXP authtoken you used in the previous section to authenticate the proxy agent with SocketXP Cloud Gateway.

$ cat config.json
{
    "authtoken": "<your-auth-token-goes-here>"
    "tunnel_enabled": true, 
    "tunnels" : [{ 
        "destination": "tcp://postgres:5432", 
        "iot_device_id": "postgres-mservice-cluster1",
        "iot_slave": true
    }] 
}

$ socketxp --config config.json

The above config requests socketxp agent to run in slave mode and create a local proxy TCP socket in your laptop at 127.0.0.1:5432. Also it requests socketxp agent to create a secure TLS connection to the postgres-mservice-cluster1 proxy device (running in your remote cluster or server) we created in the previous section.

Note:
Although it is called TCP socket, we run TLS on top of it. So your data is securely transmitted over the internet end-to-end.

Next, update the DNS record in your laptop with the following command:

$ echo "127.0.0.1    postgres" | sudo tee -a /etc/hosts > /dev/null

We need to add the above hostname mapping for our postgres microservice so that any local requests to the microservice will be routed to our SocketXP proxy agent listening at IP 127.0.0.1.

Run the python backend microserver

We are all set to run the python backend microservice under development in our laptop.

$ python backend.py
Database opened successfully
Table created successfully
Student records added to the table successfully

Let's validate this success by running an SQL query in our remote postgres database.

$ docker exec -it postgres psql -U gvelrajan postgres
postgres=# \c postgres
postgres=# \d
          List of relations
 Schema |  Name   | Type  |   Owner   
--------+---------+-------+-----------
 public | student | table | gannygans
(1 row)
postgres=# select * from student;
 admission | name | age |                       course                       |                     department                     
-----------+------+-----+----------------------------------------------------+----------------------------------------------------
     34231 | Dave |  19 | Information Technology                             | ICT                                               
     34232 | Gary |  18 | Electrical Engineering                             | ICT                                               
     34233 | Abel |  17 | Computer Science                                   | ICT                                               
(3 rows)

(END)

Awesome! We are able to develop/test our local python backend service and make it to seamlessly connect with the remote postgres DB microservice.

Make a remote NGINX microservice locally accessible

Now suppose, in addition to the postgres db microservice, we also need to make an NGINX microserice locally accessible from our laptop, use the following config.json file to setup the SocketXP Proxy Agent in the remote server/cluster.

$ cat config.json
{
    "authtoken": "<your-auth-token-goes-here>"
    "tunnel_enabled": true, 
    "tunnels" : [
      { 
        "destination": "tcp://postgres:5432", 
        "iot_device_id": "postgres-mservice-cluster1", 
      },
      { 
        "destination": "tcp://nginx:80", 
        "iot_device_id": "nginx-mservice-cluster1", 
      },
    ] 
}

Next, go to your laptop, and update the config.json file to create a local Proxy for the nginx microservice, as shown below.

$ cat config.json
{
    "authtoken": "<your-auth-token-goes-here>"
    "tunnel_enabled": true, 
    "tunnels" : [
      { 
        "destination": "tcp://postgres:5432", 
        "iot_device_id": "postgres-mservice-cluster1",
        "iot_slave": true 
      },
      { 
        "destination": "tcp://nginx:80", 
        "iot_device_id": "nginx-mservice-cluster1", 
        "iot_slave": true 
      },
    ] 
}

$ socketxp --config config.json

The above config will create a TCP listening socket at port 80 in your laptop. Make sure port 80 is free in your laptop and not used by any other application.

Next, add the follow DNS entry into your /etc/hosts file, so that any local DNS resolution requests to access your ngnix service will be routed to the SocketXP Proxy Agent listening on IP 127.0.0.1.

$ echo "127.0.0.1    nginx" | sudo tee -a /etc/hosts > /dev/null

Now open up a browser in your laptop and point it to: http://nginx:80 or simply http://nginx. It will open up the NNGINX home page as shown below.

We are looking for beta customers to evaluate our SocketXP Microservice Remote Access Solution. Please write to us at: [e-mail]

This article was originally published at: https://www.socketxp.com/blog