AWS site-to-site VPN using StrongSwan

It's a Sunday and I get a call from one of my ex-colleagues and he said "site-to-site VPN does not work when I want to reach VM's on your laptop because we are behind our ISP's NAT". And immediately I was like "Devops engineers and their Networking pitfalls". I popped open my beer and started setting this up.

The setup is:

I have an EC2 instance (10.100.11.0/24)
I have 2 VM's on VMware Workstation.
One of them is a strongswan VM which will act as the on-premise router.(192.168.1.9)
Second One is a mariadb VM that the EC2 instance should be able to ping. (192.168.1.10)

Long-story-short version of system admins A.K.A - The architecture diagram

The easiest part of the setup - The AWS side of configuration

Create a Virtual Private Gateway and associate it with your VPC.
Identify your Public IP at the On-premise end and create the customer gateway in AWS. Enter your public ip here.
Create a site-to-site VPN, associate it with the Virtual Private Gateway and the customer gateway we created in step 1 and 2.
Download the configuration from the top-end of the page. This has the instructions to set up the tunnel interface on StrongSwan, along with other configurations.
Launch a Ec2 instance in your VPC .

The trickiest part of the setup - The on-premise configuration

Open the configuration we downloaded and make changes . We are actually making way for the IKE1 and IKE2 setups. I will be covering what are these in a separate document.
Make changes to the ipsec.conf and ipsec.secrets on the strongswan VM. These settings are in the configuration we downloaded from AWS when we created the VPN. The settings should be something like:

Create tunnel interface and assign IP's as mentioned in the downloaded file.

Add sysctl settings mentioned in the file.It should have something like:
net.ipv4.conf.Tunnel1.rp_filter=2 #This value allows the Linux kernel to handle asymmetric routing net.ipv4.conf.Tunnel1.disable_policy=1 #This value disables IPsec policy (SPD) for the interface net.ipv4.conf.<PHYSICAL INTERFACE>.disable_xfrm=1 #This value disables crypto transformations on the physical interface net.ipv4.conf.<PHYSICAL INTERFACE>.disable_policy=1 #This value disables IPsec policy (SPD) for the interface net.ipv4.ip_forward = 1
Flush routes:
ip route flush table 220 ip route flush cache
Add a route. Now, this is the CIDR where the EC2 instance lives. We are making this route so the packet destined for the ec2 instance goes via the tunnel interface.
ip route add 192.168.0.0/16 via 169.254.51.57 dev vti1
Add a route to the Ec2 instance's route table. This route tells the packet to travel via the Virtual Private Gateway if the destination network is our on-premise network.

And If we have everything in place our Ec2 should be able to ping our on-premise strongswan VM:

Now, this strongswan is the gateway at the on-premise end. So in order for the EC2 to be able to reach the mariadb instance we have to add a route. A return route to be specific.