DEV Community: Swapnil Abhimanyu Wagh

How To Access S3 From Private Subnet

Swapnil Abhimanyu Wagh — Tue, 18 Jan 2022 19:13:08 +0000

Hello Dev's, in this article, I will show you how can you use the S3 Gateway endpoint to connect to S3 from a private subnet.

You might be thinking what is the difference between accessing S3 from a public subnet vs accessing it from a private subnet.

If you are using S3 you might already know that S3 is a public service and you can access it using the public endpoint, so the URL is publicly resolvable. What I exactly mean here, if you are in VPC which is a private environment and you want to connect to S3 you have to go out to the internet. So there will be an internet gateway your request will go through that internet gateway to access the S3.

Hope you might get the answer to the above question, in the public subnet you can access the internet via the internet gateway, and in the private subnet, you don’t have internet so you cant reach out to the internet and request time out.

Imagine due to some security policies you don’t want to go over the public internet to access the S3. Then how can you access the S3, here comes the “S3 Gateway Endpoint” to rescue.

What is “VPC Endpoint”?

As per AWS official documentation:

A VPC endpoint enables connections between a virtual private cloud (VPC) and supported services, without requiring that you use an internet gateway, NAT device, VPN connection, or AWS Direct Connect connection. Therefore, you control the specific API endpoints, sites, and services that are reachable from your VPC.

AWS provides the following types of VPC Endpoint

Interface endpoints
Gateway load balancer endpoints
Gateway endpoints

S3 Gateway endpoint provides a way for the request to be routed through AWS Network to S3, without going out to the internet.

Let's understand what problem exactly the S3 gateway solves for us by doing a small Lab.

Prerequisites

AWS Account - Click here to create one
Basic understanding of EC2, VPC & S3

Let's Setup a Lab

To set up our lab we will create the below resources in our AWS account

Public & Private Subnet
Public & Private route table
EC2 instances in both subnets
IAM Role to access S3 from EC2
S3 Gateway Endpoint

Public & Private Subnets

A public subnet is a subnet that has a route to the internet gateway, due to that the EC2 instances launched in a public subnet can access the internet. On the other hand, a private subnet is a subnet that does not have an internet gateway attached to it.

To create a subnet go to VPC and from the left side menu select Subnets and click on the Create Subnet button, fill in the form and click on Create Subnet.

Follow the same process to create a private subnet, once you created two subnets that will look like

Public & Private route table

Before creating a route table we will create an Internet Gateway to attach to our public route table

Once our internet gateway is created we will attach it to our VPC

To create a route table click on the Route Tables menu from the left sidebar and click Create route table

Once our route table is created we will add a route to an internet gateway

and associate the public subnet to our public route table

Launch EC2 instances

We will launch two EC2 instances each in a public and private subnet. Navigate to Services -> EC2 and click Launch Instances, it will open the launch EC2 instance wizard,

Once AMI is selected, on the next screen select the instance type, and then click Next

On the Configure Instance step select the VPC & Subnet, as we want to put this EC2 instance in the public subnet we will select the public subnet we created from the subnet dropdown

As we will stick with the default setting for other steps click Review and Launch and then Launch. The popup will show up, select the existing key pair or create a new key pair for your EC2 instance, this key will be used for SSH access

Follow the same steps again to launch our second instance, just make sure you select the private subnet this time.

Now you should have the two instances, as we launch one instance in the public subnet hence that instance got the public IP.

Create IAM Role

To access the S3 service securely from our EC2 instances, we will create one IAM role which we will then assign to EC2 instances.

To create an IAM role go to IAM -> Roles and click the Create Role button

Select the type of trusted entity and select the service to which this role will be assigned, select AWS services and then EC2 from the use case

Next, we will select which permissions we want to assign to this new role, as we want to access the S3 service we will search for S3 and select AmazonS3FulllAccess

Go to the review step and give a name to the newly created role and click Create Role

Assign Role to EC2

Now we will assign this role to our EC2 instances, (you can create this role before launching the EC2 and then select the role from IAM Role dropdown in EC2 instance launch wizard)

To assign the role go to EC2 and select one of the EC2 instances then click on
Action -> Security -> Modify IAM role

It will open a new screen, select the role that we created in the above step, and click save,

Demo 1: Fall Down - Access S3 from the private subnet

In this demonstration, we will test the scenario of what exactly happens if we don't have the S3 Gateway assigned to the private subnet.

Let's SSH into our public instance, then from the public instance, we will SSH into our private instance using the ssh-agent forwarding.

Use below command to add SSH key in ssh-agent,

❯ ssh-add -K SW-Demo-Key.pem

Once we added the key, use the below command to ssh into the public EC2 instance,

❯ ssh -A ec2-user@13.127.220.246

Now we will ssh to our private instance using private IP

❯ ssh ec2-user@10.10.1.245

Next, we will try to access the S3 from a private EC2 instance using the S3 command-line tool,

[ec2-user@ip-10-10-1-245 ~]$ aws s3 ls --region ap-south-1

This will not work because, the EC2 instance in the private subnet doesn't have access to the internet, therefore S3 CLI cant reach the S3 service.

Create S3 Gateway Endpoint

To create an S3 gateway endpoint, navigate to services ->VPC->Endpoints and click the Create Endpoint button,

As we want to create an endpoint for S3 search for S3 in the search box and select the service of type Gateway

Once we choose which type of endpoint we want to create now we need to apply that endpoint to our private subnet, to do that select the VPC and then select the private route table.

Next, click the Create Endpoint button to create the endpoint,

AWS will automatically add a route in the selected route table in the background.

Demo 2:

Now let's try to access the S3 one more time from our private EC2 instance, SSH into the private EC2 instance as shown above using the SSH agent forwarding, and re-run the S3 CLI command

[ec2-user@ip-10-10-1-245 ~]$ aws s3 ls --region ap-south-1
2022-01-18 18:31:44 sw-s3-endpoint-demo

Hurry !! this time we can successfully connect to the S3 using the S3 Gateway endpoint.

And that’s it, guys, don't forget to delete all the resources created during this lab.

Hope you find this tutorial helpful. Don't forget to share and leave a comment if it helps you.

How To Generate & Install SSL Certificate In Apache Web Server On Ubuntu

Swapnil Abhimanyu Wagh — Sat, 05 Dec 2020 14:44:31 +0000

SSL, Secure Socket Layer is a protocol created in order to place normal traffic between server and client in encrypted and protected wrapper, without any possibility of traffic being intercepted in between of transmission. SSL certificates encrypt a site information and create more secure and trusted connection. Certificates can show server's identification information to site visitors. Certificates Authorities like DigiCert , GoDaddy can issue the self-signed SSL certificates that verify the server information. In this guide I will cover how to create a self-signed SSL certificates for Apache on an Ubuntu Server which will encrypt traffic to your server. As we are generating SSL certificates are for testing purpose and not issued by any Certificates Authorities it will not provide third party validation of your server identity, but still it will help you to transfer information securely.

Prerequisites:

You need a Ubuntu server and a user with root / sudo privileges. You also need a Apache web server installed if its not there you can use following command to install Apache.

$ sudo apt-get update 
$ sudo apt-get install apache2

Step 1: Enable SSL module in Apache:

In order to setup SSL certificates in Apache web server we need to enable the pre-installed Apache module. To enable SSL module in Apache use below command.

$ sudo a2enmod ssl

Once you enable the module we need to restart the Apache server for changes to be applied. Use below command to restart Apache:

$ sudo service apache2 restart

Now our server is ready to setup SSL.

Step 2: Generate a Self-Signed Certificates:

We will create a new directory where we will store server key and certificates.

$ sudo mkdir /etc/apache2/ssl

First you generate the keys for the Certificate Signing Request (CSR)

$ cd /etc/apache2/ssl 
$ sudo openssl genrsa -des3 -out yourdomain.pkey 2048

It will prompt you to enter passphrase. Its up to you to enter a passphrase or not. If you do, every time you restart the Apache service you will ask for this passphrase. We will enter a passphrase and then will create a 'insecure' key from 'secure' one. Create a insure key without a passphrase form a secure key yourdomain.pkey.

$ sudo openssl rsa -in yourdomain.pkey -out yourdomain.key

And now we will create a CSR from the key file. With the CSR and key file a self-signed certificate can be generated.

$ sudo openssl req -new -key yourdomain.key -out yourdomain.csr

Once enter will ask you number of questions one most important is Common Name use your domain or IP of you server as a value.

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:New York
Locality Name (eg, city) []:NYC
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Organization Pvt. Ltd
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:yourdomain.com
Email Address []:webmaster@yourdomain.com

Once we have CSR and key file now we can generate a self-signed certificate using following command

$ sudo openssl x509 -req -days 365 -in yourdomain.csr -signkey yourdomain.key -out yourdomain.crt

Step 3 : Configure Apache to use the Signed SSL Certificate :

Now that we have all our necessary certificates and key file available, now we can configure Apache to use these in VirtualHost. Next, add an entry to /etc/apache2/ports.conf for the domain you'll be using to host your SSL-enabled site.

NameVirtualHost yourdomain.com:443

Now create a Apache configuration file with VirtualHost entry for your site. Create a file yourdomain-ssl with below contain in /etc/apache2/site-available/folder of Apache.

<VirtualHost yourdomain.com:443>
    SSLEngine On
    SSLCertificateFile /etc/apache2/ssl/yourdomain.crt
    SSLCertificateKeyFile /etc/apache2/ssl/yourdomain.key

    ServerName http://www.yourdomain.com
    ServerAlias yourdomain.com
    DocumentRoot /var/www/yourdomain

    <Directory /var/www/yourdomain>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride All
        Order allow,deny
        allow from all
    </Directory>
</VirtualHost>

Step 4 : Permanent Redirection to https :

To redirect your site permanently to https we will edit the VirtualHost entry for for 80 port which is in a file yourdominin /etc/apache2/site-available folder of Apache, or else create one with below contain.

<VirtualHost *:80>
    ServerName http://www.yourdomain.com
    ServerAlias yourdomain.com
    DocumentRoot /var/www/yourdomain
    Redirect / https://www.yourdomain.com
</VirtualHost>

Or else you can create .htaccess file in your site root folder for permanent redirection to https with below contain

RewriteEngine On
RewriteCond %{HTTPS} off [OR]
RewriteCond %{HTTP_HOST} ^yourdomain\.com$ [NC]
RewriteRule ^ https://www.yourdomain.com%{REQUEST_URI} [L,R=301]

Step 5 : Test your Apache config and activate the VirtualHost :

It is always a good practice to check your Apache configuration for any error before we activate the VirtualHost and restart the Apache service, cause Apache will not start if your configuration file has some errors. Run below command to test configuration

$ apachectl configtest

If your configuration is right then activate the VirtualHost using below command

$ sudo a2ensite yourdomain      // Simple http configuration file 
$ sudo a2ensite yourdomain-ssl     // SSL configuration file

Now, restart the Apache service for changes to take effect

$ sudo service apache2 restart

Step 5 : Test your Setup :

Now, that we have everything prepared and done you can try visiting your server domain name or public IP using the https protocol like this https://yourdomin.com. You will get a security warning from browser regarding your server identity because we are using self-signed certificates and it has been not signed by any certificate authority that browser trust.

Since this is expected you can move ahead, hit 'Proceed Anyway' button. Once Added Security Exception for your browser you will taken to the your site this time with encrypted traffic, check the SSL configuration by clicking on lock icon in address bar.

Hope you find this tutorial helpful. Feel free to ask questions! Don't forget to like or to leave a comment if its really help you.

How to Password Protect a Website or Web SubDirectory With .htaccess & .htpasswd

Swapnil Abhimanyu Wagh — Sat, 05 Dec 2020 14:40:43 +0000

Working on a website that you need others to see, but not the whole world? Password protecting a website (or a sub directory within a website).

Protecting files on your website from unauthorized users can be very important. You can use PHP or any language to listen for login authorization information on each page, but that doesn’t protect your images, documents, and other media and it is not proper way to do so.

That’s why I’ve found the new method of protecting files and directories the most reliable and is actually a pretty easy thing to do.

To password protect we will use .htaccess and .htpasswd method.

Step1: Basic Coniguration

To make .htaccess files work as expected, you need to have below line in your site Apache configuration,

AllowOverride All

So your file will look like

ServerName password-protected.com DocumentRoot /var/www/PasswordProtected # This relaxes Apache security settings. AllowOverride all

This tells Apache that it’s okay to allow .htaccess files to over-ride previous directives. You must reload Apache before this change will have an effect

sudo service apache2 reload

Step2: Create .htaccess and .htpasswd files

Create a file called .htaccess in the directory that you want to password-protect (in my case I am using /var/www/PasswordProtected directory) with the following content

AuthUserFile /var/www/PasswordProtected/.htpasswd AuthName “Authorization Required” AuthType Basic require valid-user

Then create the file /var/www/PasswordProtected/.htpasswd which contains the users that are allowed to login and their passwords.

We do that with the htpasswd command, to use this command make sure apache2-utils package is install.

htpasswd -c /var/www/PasswordProtected/.htpasswd USER1

The -c flag is used only when you are creating a new file. After the first time, you will omit the -c flag, when you are adding new users to an already-existing password file. Otherwise you will overwrite the file!!

And that’s it you are done !! Your website is password protected now.

How To Replace a lost Key Pair with new one on an EC2 instance

Swapnil Abhimanyu Wagh — Sat, 05 Dec 2020 05:58:34 +0000

Access to Amazon EC2 instances specially the Linux instances are always protected by a private key pair. Private key is only the way to get access to the instance, what if you lost the key? Is your servers lost in black hole? Thankfully nothing is lost you still can access your server, let me show you how can you solve this problem.

Before we proceed please not that to do this we need to stop our running instance that means what ever app or website running on the serer will be offline for that period. In order to do this process we need the EC2 root volume to be an EBS.

Quick Start

Launch New Instance

First thing we need is to start and one instance with a key pair that we can access we will use this instance for recovering the old EBS. New instance must be created in same Availability Zone of the original instance. Firstly we will note down the original instance AZ.

Now we will launch the new instance using the Launch Instance wizard, make sure that we are launching the instance same AZ in which our original instance is. You can chose AZ from **subnet **as shown in below image.

Before we actually launch the instance we will see the key pair section from there we will create and download a new key pair and then we will launch the instance.

Stop Original Instance

When the new instance is ready, now we will detach the volume of original instance and attach it to new instance, in order to do that we need to stop the original instance. To stop the instance right click on the instance go to Instance State and chose Stop option.

Detach Volume

Now our instance is stop then we go to the volume section in EC2 console. Here we will look for the original instance root volume, using the Attachment Information.This column shows to which instance the volume is attach and its device path usually the root drive path is /dev/sad1.Now we will Detach the volume from original instance, in order to do that right click on volume and chose Detach Volume option.

Attach Volume

Once we detach the volume, now we need to attach the volume to the Recovery instance, right click on the volume and chose Attach Volume option in dialog we will select the Recovery instance and also add a device path as /dev/xvdf (or /dev/sdf)

Once we attach the volume to recovery instance we are now ready to mount that in same instance. We can access the instance using the key pair generated during the instance launch. To access instance use below command

Setting the rigth permissions for the .pem file:

 $ chmod 600 NewKeyPair.pem

Connecting to the instance using the certificate:

 $ ssh -i NewKeyPair.pem ubuntu@public_ip

Copy Key

Once you are connected to the instance now we have to mount the attach volume and copy the authorized_keys file from new instance to the mounted drive at same location. Use below command to mount and copy the file.

Becoming root:

$ sudo -i

Creating the mount point:

$ mkdir /mnt/original

Mounting the volume:

$ mount /dev/xvdf /mnt/original

Replace the authorized_keys file with the new one. Now access to the Old instance will be possible only using the new pem. Path and name of file depends on the Linux Flavor you are using.

cat /home/ubuntu/.ssh/authorized_keys > /mnt/original/home/ubuntu/.ssh/authorized_keys

Unmunt the volume:

$ umount /mnt/original/

Once we are done with copying the keys, now we will detach the volume from recovery instance by right clicking on volume and choosing the detach volume option as we do it earlier. After that we will re-attach it to the Original instance, make sure when you are re-attaching the volume that should on on root path so enter /dev/sda1 in device path option.

Access Using New Key

Now we can start the original instance back, and now you can access it using the new key pair.

You can use same steps in case your machine is not accessible and you want access to your data in EBS.

Hope you find this tutorial helpful. Feel free to ask questions! Don’t forget to like or to leave a comment if its really help you.

How To Deploy NodeJS Apps With PM2 & NGINX In Cluster Mode On Production Environment

Swapnil Abhimanyu Wagh — Wed, 20 May 2020 20:36:23 +0000

What is Process Manager?

Process Manager is a special program design to effectively manage server process and take benefit of server resources. It's useful to keep application online and restart on failure.

Process Manager is also useful for clustering, logging and monitoring the application. Process Managers make it possible to demonize the application so it will be running in background as a service.

Prerequisites:

In this tutorial we assume that you have following setup:

Ubuntu server and a user with root / sudo privileges
All the necessary package installed to run simple NodeJS App

Which Process Manager?

There are multiple Process Manager available, as listed below but in this tutorial we will be focusing on PM2.

Why PM2?

Following are the complete features set of PM2

Install Process Manager:

Use npm to install the pm2 globally so it will available system-wide for use

$ sudo npm i pm2 -g

Now let start our NodeJS app using pm2 start command

First change the directory to our node application directory

$ cd /opt/hello-pm2/
$ pm2 start app.js --name Hello -i 2 --watch

It will also register our app in the process list of PM2, which you can see in the output of the above command

PM2 as a service:

PM2 will take care of all the application running under it and will restart automatically if the application killed or crash, but what if the system boot or reboot? PM2 has answer for this, PM2 provide an easy way to start PM2 as a system service in systemd.

The startup command generate and configure a PM2 startup script.

$ pm2 startup

Now to setup the startup script copy/paste the last line from the output or earlier command,

[PM2] Init System found: systemd
meswapnilwagh
[PM2] To setup the Startup Script, copy/paste the following command:
sudo env PATH=$PATH:/usr/bin /usr/lib/node_modules/pm2/bin/pm2 startup systemd -u meswapnilwagh --hp /home/meswapnilwagh

Run the command to setup PM2 to start on boot/reboot

sudo env PATH=$PATH:/usr/bin /usr/lib/node_modules/pm2/bin/pm2 startup systemd -u meswapnilwagh --hp /home/meswapnilwagh

Basic PM2 Commands:

Just like all other command line utility, PM2 also comes with bundle of subcommands which are helpful to manage application running under PM2

Start Application in cluster

To start application in cluster mode you can use -i flag and specify the number of instances you want to run you can also use --name flag to name your process.

sudo pm2 start /opt/hello-pm2/app.js --name Hello -i 4

Stop Application

sudo pm2 stop Hello

Restart Application

sudo pm2 restart Hello

List Applications

sudo pm2 list

Monitor Application Process

sudo pm2 monit

For more usage of PM2 please refer PM2 quick start.

NGINX As Reverse Proxy :

Till now we configure PM2 and running our node app in cluster seems all good, but are you still ready for production? How can you get rid of that port in your URL? Answer to all your question is Nginx (Engine-X).

What is NGINX?

Officially, Nginx is a web server which can also be used as a reverse proxy, load balancer, mail proxy and HTTP cache.

Best practice to deploy NodeJS application in production, is by using Nginx as reverse proxy to route the web client's requests to appropriate node process.

Install NGINX

Use following command to install Nginx on Ubuntu

$ sudo apt-get update
$ sudo apt-get install nginx

Configure NGINX

Open the nginx default site config file:

$ sudo nano /etc/nginx/sites-available/default

Now add below configuration in the file (You can take backup of original file for safer side)

server {
  listen       80;
  server_name  mycooldomain.com;

  location / {
    proxy_pass http://localhost:4000;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $host;
    proxy_cache_bypass $http_upgrade;
  }
}

As you can see the nginx listening on http://mycooldomain.com and the location / block take care of the incoming request and forwarding to NodeJS application.

Save the file and restart nginx,

$ sudo service nginx restart

Now open your browser and navigate to http://mycooldomain.com, you can see how node app is being served without using any port in URL.

Congratulations!! You had successfully deployed NodeJS app on production using PM2 and Ngnix.

Hope you find this tutorial helpful. Don't forget to share if its really help you. For any query please DM at Swapnil Wagh