DEV Community: Frederic

Full Stack App Automated deployment on AWS EKS

Frederic — Fri, 02 Jul 2021 16:37:27 +0000

Full Stack Application on AWS EKS

This example automatically deploys a full-stack application with Kubernetes on AWS using their managed control plane called Elastic Kubernetes Service

Automation is performed with GruCloud, which is an infrastructure as code tool written in Javascript.

High-level description

This infrastructure combines 2 providers: AWS and Kubernetes.

A few modules for each of these providers are being used:

In this example, the load balancer, target groups, listeners, and rules are managed by GruCloud instead of the AWS Load Balancer Controller.
The benefit of not using the LBC is to free a lot of resources, the LBC depends on the Cert Manager which brings more resources,
By not using the AWS LBC and the Cert Manager, we can save 4 pods, and numerous other resources such as ServiceAcount, ClusterRole, ClusterRoleBinding, we can even get rid of the CRD stuff.
Fewer pods mean we can choose a cheaper worker node.

Modules for AWS resources

Modules for K8s resources

The local module defining the app on the k8s side is located at base.

There is no trace of manifests in YAML, instead, Kubernetes manifests are described in Javascript, allowing the use of variables, conditionals, loops, importing code, and even the use of a debugger. Therefore, GruCloud is an alternative to helm.

Resources

From the infrastructure code, 2 kinds of visual representation can be generated with the GruCloud CLI gc:

a mind map indicating the type of the resource: gc tree
a diagram showing the relationship between the resources: gc graph

Mindmap

Diagram

The following diagram shows the AWS and K8s resources and their relashionship:

Workflow

The next flowchart tells the actions to perform to configure, deploy, update and destroy this infrastructure:

Requirements

AWS CLI

GruCloud CLI:

Getting the source code

This example is located at examples/starhackit/eks-lean

Configuration

Amazon EKS

The first part of this deployment is to create an EKS control plane, a node group for the workers, and all their numerous dependencies.

Configuration for the AWS resources is located at configAws.js

Set the rootDomainName and domainName according to your use case

For end-to-end automation, the rootDomainName should be registered or transferred to the AWS Route53 service.

K8s

The second part is the Kubernetes deployment of the full-stack application composed of a React front end, a Node backend, Postgres as the SQL database, and finally Redis for the cache and published/subscriber models.

Configuration for the K8s resources is located at configK8s.js

When the backend container is changed to another backend, do not forget to change the target group health check in configAWS.js

GruCloud workflow

This chart depicts the workflow with the main gc commands:

gc info
gc graph
gc apply
gc list
gc destroy

Listing

Let's find out if everything is set up properly by listing the live resources from AWS:

gc list

Listing resources on 2 providers: aws, k8s
✓ aws
  ✓ Initialising
  ✓ Listing 29/29
k8s
  Initialising
  Listing 0/8
List Summary:
Provider: aws
┌───────────────────────────────────────────────────────────────────────────────────────────────┐
│ aws                                                                                           │
├────────────────────┬──────────────────────────────────────────────────────────────────────────┤
│ IamPolicy          │ AmazonEKSClusterPolicy                                                   │
│                    │ AmazonEKSVPCResourceController                                           │
│                    │ AmazonEKSWorkerNodePolicy                                                │
│                    │ AmazonEC2ContainerRegistryReadOnly                                       │
│                    │ AmazonEKS_CNI_Policy                                                     │
├────────────────────┼──────────────────────────────────────────────────────────────────────────┤
│ Route53Domain      │ grucloud.org                                                             │
├────────────────────┼──────────────────────────────────────────────────────────────────────────┤
│ Vpc                │ default                                                                  │
├────────────────────┼──────────────────────────────────────────────────────────────────────────┤
│ InternetGateway    │ igw-9c2f1ae7                                                             │
├────────────────────┼──────────────────────────────────────────────────────────────────────────┤
│ SecurityGroup      │ default                                                                  │
├────────────────────┼──────────────────────────────────────────────────────────────────────────┤
│ Subnet             │ default                                                                  │
│                    │ default                                                                  │
│                    │ default                                                                  │
│                    │ default                                                                  │
│                    │ default                                                                  │
│                    │ default                                                                  │
├────────────────────┼──────────────────────────────────────────────────────────────────────────┤
│ RouteTable         │ rtb-19753867                                                             │
└────────────────────┴──────────────────────────────────────────────────────────────────────────┘
19 resources, 8 types, 1 provider
Command "gc l" executed in 5s

Note the presence of a default VPC, subnets, security group, and internet gateway.

Verify that the domain name is registered with Route53Domain, in this case, grucloud.org

The Kubernetes control created by EKS is not up yet, as a consequence, listing the k8s resources cannot be retrieved at this stage.

Deploying

It is show time for deploying all the AWS and Kubernetes resources in one command:

gc apply

The app should be now running with Kubernetes on AWS after 15 minutes.

When all the resources are created, custom code can be invoked in hook.js.
In this example, we verify access to the webserver and the API server securely.

The kubeconfig has been updated with the endpoint from the EKS cluster.

kubectl config current-context

arn:aws:eks:eu-west-2:999541460000:cluster/cluster

Let's list and produce a diagram of the AWS resources freshly created:

gc list -p aws --graph -a --default-exclude --types-exclude Certificate --types-exclude Route53Domain --types-exclude NetworkInterface

Notice that the NodeGroup has created an AutoScaling Group, which in turn creates EC2 instances, instance profiles, and volumes.

Updating

Let's update the cluster with another version of the front end.
Edit configK8s.js and change the frontend version.

The gc apply command will find out the difference between the expected version and the deployed version.

Querying resources on 2 providers: aws, k8s
✓ aws
  ✓ Initialising
  ✓ Listing 30/30
  ✓ Querying
    ✓ HostedZone 1/1
    ✓ Certificate 1/1
    ✓ Route53Record 2/2
    ✓ Vpc 1/1
    ✓ InternetGateway 1/1
    ✓ Subnet 4/4
    ✓ RouteTable 3/3
    ✓ Route 3/3
    ✓ ElasticIpAddress 1/1
    ✓ NatGateway 1/1
    ✓ IamRole 2/2
    SecurityGroup 3/4
    ✓ SecurityGroupRuleIngress 6/6
    ✓ SecurityGroupRuleEgress 1/1
    ✓ KmsKey 1/1
    ✓ EKSCluster 1/1
    ✓ EKSNodeGroup 1/1
    ✓ LoadBalancer 1/1
    ✓ TargetGroup 2/2
    ✓ Listener 2/2
    ✓ Rule 3/3
✓ k8s
  ✓ Initialising
  ✓ Listing 8/8
  ✓ Querying
    ✓ Namespace 1/1
    ✓ ConfigMap 2/2
    ✓ StatefulSet 2/2
    ✓ Service 4/4
    ✓ Deployment 2/2
┌────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ 1 Deployment from k8s                                                                              │
├────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ┌───────────────────────────────────────────────────────────────────────────────────────────────┐  │
│ │ UPDATE: name: default::web, id: web                                                           │  │
│ ├───────────────────────────────────────────────────────────────────────────────────────────────┤  │
│ │ Key: spec                                                                                     │  │
│ ├───────────────────────────────────────────────┬───────────────────────────────────────────────┤  │
│ │ - template:                                   │ + template:                                   │  │
│ │   spec:                                       │   spec:                                       │  │
│ │     containers:                               │     containers:                               │  │
│ │       0:                                      │       0:                                      │  │
│ │         image: fredericheem/ui:v10.14.0       │         image: fredericheem/ui:v10.15.0       │  │
│ │                                               │                                               │  │
│ └───────────────────────────────────────────────┴───────────────────────────────────────────────┘  │
└────────────────────────────────────────────────────────────────────────────────────────────────────┘


┌─────────────────────────────────────────────────────────────────────────────────────────────┐
│ Plan summary for provider aws                                                               │
└─────────────────────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────────────────────┐
│ Plan summary for provider k8s                                                               │
├─────────────────────────────────────────────────────────────────────────────────────────────┤
│ DEPLOY RESOURCES                                                                            │
├────────────────────┬────────────────────────────────────────────────────────────────────────┤
│ Deployment         │ default::web                                                           │
└────────────────────┴────────────────────────────────────────────────────────────────────────┘
? Are you sure to deploy 1 resource, 1 type on 1 provider? › (y/N)

It is the equivalent of kubectl apply, except that kubectl is "fire and forget", but gc apply the changes and waits for the resources to be ready before returning.

Destroying

Running Kubernetes cluster on AWS or other cloud providers is not cheap. Stop paying when the cluster is not used with one command:

gc destroy

Resources will be destroyed in the right order and automatically.

Deploy an HTTPS Static Website on GCP with GruCloud

Frederic — Wed, 30 Jun 2021 18:55:51 +0000

Hosting a secure static website on GCP, domain managed by AWS Route53

The goal of this example is to automate with GruCloud, the deployment of a static website served with HTTPS on GCP, the Google Cloud Platform.

The domain and DNS settings are handled by AWS Route53.

See the manual way at hosting static website

Google Domain does not provide API, and the other google DNS service Cloud Domain does not support transferring a domain.

For this reason, this example uses AWS Route53.

Resources

This deployment is composed of the resources depicted in the following diagram:

The command gc graph generates this diagram from the code iac.js.

Below are the links to the resource documentation:

Hooks

The file hook.js contains actions to perform after the infrastructure is deployed and after it is destroyed.

The file route53Utils.js contains functions to add and remove a DNS record of type A to map the domain name to the load balancer's IP address by calling the AWS CLI.

aws route53 list-hosted-zones-by-name --dns-name ${domainName}
aws route53 list-resource-record-sets --hosted-zone-id ${hostedZoneId}
aws route53 change-resource-record-sets --hosted-zone-id ${hostedZoneId} --change-batch '${changeBatch}'

Requirements

GruCloud CLI

Ensure gc, the GruCloud CLI is installed:

GCP

Follow this flowchart to install and configure the gcloud CLI.

AWS

Ensure the AWS CLI is installed as it will be invoked in the hook.js to add and remove a DNS record to map the domain name to the load balancer's IP address.

Route53 Domain

Let's verify that the AWS account has a domain name registered:

aws route53domains list-domains --region us-east-1

{
    "Domains": [
        {
            "DomainName": "grucloud.org",
            "AutoRenew": true,
            "TransferLock": false,
            "Expiry": "2021-11-16T13:56:10+01:00"
        }
    ]
}

Alternatively, see the Route53 Domain Listing

Route53 Hosted Zone

Create a Hosted Zone, later on, a DNS record of type A will be created in this zone:

aws route53 create-hosted-zone --name grucloud.org --caller-reference 2021-06-30-2

You could also visit the Route53 Dashboard

Getting the source code

This example is located at examples/google/storage/website-https

Config

Edit config.js and set the following variables:

projectId: the project Id, must be unique, see restrictions here
bucketName: the bucket name which is also the domain name, i.e. mywebsite.com or subdomain.mywebsite.com, see the bucket naming guideline
hostedZoneName: the hosted zone name in Route53, i.e. mywebsite.com
websiteDir: the directory containing the static website.

Initialise

The init command will create the project, set up the billing, enable the API services, create the service account and its credentials file, and bind the IAM roles to this service account

gc init

Add the service account as the domain owner

The service account operating by grucloud needs to be added as the domain owner.

This service account created previously with the init command is in the form of grucloud@YourProjectId.iam.gserviceaccount.com

This service account is also stored in the generated credential file as the client_email property.
To find out where the credential file is located:

gc info

  - provider:
      name: google
      type: google
    stage: dev
    projectId: grucloud-test
    projectName: grucloud-test
    applicationCredentialsFile: /Users/yourusername/.config/gcloud/your-project-id.json
    serviceAccountName: grucloud
    hasGCloud: true
    config:
      bucketName: grucloud.org
      websiteDir: ./websites/simple
      managedByTag: -managed-by-gru
      managedByKey: managed-by
      managedByValue: grucloud
      region: europe-west4
      zone: europe-west4-a

Follow the manual steps at the domain name verification documentation.

GruCloud CLI Workflow

This chart shows the GruCloud CLI main commands:

Deploy

Deploy this infrastructure with the apply command

gc apply

Querying resources on 1 provider: google
✓ google
  ✓ Initialising
  ✓ Listing 7/7
  ✓ Querying
    ✓ Bucket 1/1
    ✓ Object 1/1
    ✓ SslCertificate 1/1
    ✓ BackendBucket 1/1
    ✓ UrlMap 1/1
    ✓ HttpsTargetProxy 1/1
    ✓ GlobalForwardingRule 1/1
┌────────────────────────────────────────────────────────────────────┐
│ Plan summary for provider google                                   │
├────────────────────────────────────────────────────────────────────┤
│ DEPLOY RESOURCES                                                   │
├────────────────────┬───────────────────────────────────────────────┤
│ Bucket             │ grucloud.org                                  │
├────────────────────┼───────────────────────────────────────────────┤
│ Object             │ index.html                                    │
├────────────────────┼───────────────────────────────────────────────┤
│ SslCertificate     │ ssl-certificate                               │
├────────────────────┼───────────────────────────────────────────────┤
│ BackendBucket      │ backend-bucket                                │
├────────────────────┼───────────────────────────────────────────────┤
│ UrlMap             │ url-map                                       │
├────────────────────┼───────────────────────────────────────────────┤
│ HttpsTargetProxy   │ https-target-proxy                            │
├────────────────────┼───────────────────────────────────────────────┤
│ GlobalForwardingR… │ global-forwarding-rule                        │
└────────────────────┴───────────────────────────────────────────────┘
Deploying resources on 1 provider: google
✓ google
  ✓ Initialising
  ✓ Deploying
    ✓ Bucket 1/1
    ✓ Object 1/1
    ✓ SslCertificate 1/1
    ✓ BackendBucket 1/1
    ✓ UrlMap 1/1
    ✓ HttpsTargetProxy 1/1
    ✓ GlobalForwardingRule 1/1
  ✓ default::onDeployed
    ✓ add dns record for the load balancer
    ✓ get https://storage.googleapis.com/grucloud.org/index.html
    ✓ get https://storage.googleapis.com/grucloud.org
    ✓ ssl certificate ready
    ✓ get https://grucloud.org
7 resources deployed of 7 types and 1 provider
Running OnDeployedGlobal resources on 1 provider: google
Command "gc a -f" executed in 11m 48s
fredericheem@pc7 website-https %

At the end of the deployment of the GCP resources, the onDeployed hook is invoked.

First, a DNS type A record is added to map the domain name to the load balancer's IPv4 address, i.e the GlobalForwardingRule resource.

The SSL certificate is waiting for this record to verify the ownership of the domain. It may take a few minutes for the SSL certificate to be ready.

The last stage is to get the webpage with HTTPS.
It ensures the deployment is completed successfully.

List the resources.

Verity the state of the resource with the gc list command, the --graph generates a graph of the live resources.

gc list --graph

Listing resources on 1 provider: google
✓ google
  ✓ Initialising
  ✓ Listing 7/7
┌─────────────────────────────────────────────────────────────────────┐
│ 1 SslCertificate from google                                        │
├─────────────────┬────────────────────────────────────────────┬──────┤
│ Name            │ Data                                       │ Our  │
├─────────────────┼────────────────────────────────────────────┼──────┤
│ ssl-certificate │ id: 6441474765553268206                    │ Yes  │
│                 │ creationTimestamp: 2021-06-30T03:44:17.99… │      │
│                 │ name: ssl-certificate                      │      │
│                 │ description: Managed By GruCloud           │      │
│                 │ selfLink: https://www.googleapis.com/comp… │      │
│                 │ certificate: -----BEGIN CERTIFICATE-----   │      │
│                 │ MIIFUTCCBDmgAwIBAgIRAPkAAOJC7+5VCgAAAADtD… │      │
│                 │ RjELMAkGA1UEBhMCVVMxIjAgBgNVBAoTGUdvb2dsZ… │      │
│                 │ TEMxEzARBgNVBAMTCkdUUyBDQSAxRDQwHhcNMjEwN… │      │
│                 │ MTA0ODQyWjAXMRUwEwYDVQQDEwxncnVjbG91ZC5vc… │      │
│                 │ AQUAA4IBDwAwggEKAoIBAQCwg05xlwu+1ypsJgQ5W… │      │
│                 │ I6TrqTwTpTq8lIwpEBvksGz2w99nJ5lLiCBhH7MRW… │      │
│                 │ 4z4l7D+s/u8xCe7XB2D/B6suXPdVpPjl21kHLQUQC… │      │
│                 │ dwrlJS4M3KfRYXhesJxZEeO3+PoXKw3wBkIpSRfvk… │      │
│                 │ jqdpROyJRaJ/gH1/1Ixe9wxQ0xRf9jAZClyfL8M/D… │      │
│                 │ aQiL+jB/x1AJYwuud04PPZuBNE3ouln8UnD5MD65C… │      │
│                 │ BgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBB… │      │
│                 │ ADAdBgNVHQ4EFgQUn0Lwt6ba79AHuJDIGAlW2310Y… │      │
│                 │ DrJXkZQq5dRdhpCD3lOzuJIwagYIKwYBBQUHAQEEX… │      │
│                 │ dHRwOi8vb2NzcC5wa2kuZ29vZy9ndHMxZDQwMQYIK… │      │
│                 │ a2kuZ29vZy9yZXBvL2NlcnRzL2d0czFkNC5kZXIwF… │      │
│                 │ dWQub3JnMCEGA1UdIAQaMBgwCAYGZ4EMAQIBMAwGC… │      │
│                 │ BDUwMzAxoC+gLYYraHR0cDovL2NybHMucGtpLmdvb… │      │
│                 │ V0NrLmNybDCCAQYGCisGAQQB1nkCBAIEgfcEgfQA8… │      │
│                 │ Vo7jTRMZM7/fDC8gC8xO8WTjAAABelyN0mYAAAQDA… │      │
│                 │ 64j1+EfMUP7z2a1r49DvMLeIKdM1AiEA/JSBRq8oP… │      │
│                 │ POCDVgsI/BQAdwDuwJXujXJkD5Ljw7kbxxKjaWoJe… │      │
│                 │ jdKCAAAEAwBIMEYCIQDoxrQxCcEH2vVeg2AtstxI1… │      │
│                 │ AOsmu4oulyYswA7MTs/Dy29W+nXk0D0mGAOzAHkCz… │      │
│                 │ A4IBAQAoD2cok3xHy87yVKdF7NUtjVAS6/jB6/KUd… │      │
│                 │ ZXvfhKYWZuc9WCaEUNZo45y582uuf/Wv1dBXSiH6v… │      │
│                 │ fsiQzTeOA5+5EmnQLkvaSpfs1VgMwwc0w+bd3mhHa… │      │
│                 │ NlSHP9oWk4Y4sk9ngaDu6p+nilPNXpKOXeVWchB5f… │      │
│                 │ fSjfQ5lMpeZMQi68FPuiNkcaJ5CY0ntHcwenhUpew… │      │
│                 │ 7hT0UnMmPf21WKP4EdMmmw+4dHwx               │      │
│                 │ -----END CERTIFICATE-----                  │      │
│                 │ -----BEGIN CERTIFICATE-----                │      │
│                 │ MIIFjDCCA3SgAwIBAgINAgCOsgIzNmWLZM3bmzANB… │      │
│                 │ CQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRyd… │      │
│                 │ MBIGA1UEAxMLR1RTIFJvb3QgUjEwHhcNMjAwODEzM… │      │
│                 │ MDQyWjBGMQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR… │      │
│                 │ Y2VzIExMQzETMBEGA1UEAxMKR1RTIENBIDFENDCCA… │      │
│                 │ ggEPADCCAQoCggEBAKvAqqPCE27l0w9zC8dTPIE89… │      │
│                 │ UebUQpK0yv2r678RJExK0HWDjeq+nLIHN1Em5j6rA… │      │
│                 │ saztIIJ7O0g/82qj/vGDl//3t4tTqxiRhLQnTLXJd… │      │
│                 │ H3Rcsejcqj8p5Sj19vBm6i1FhqLGymhMFroWVUGO3… │      │
│                 │ 2190Q0Lm/SiKmLbRJ5Au4y1euFJm2JM9eB84Fkqa3… │      │
│                 │ zvxtxvusLJzLWYHk55zcRAacDA2SeEtBbQfD1qsCA… │      │
│                 │ DwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDA… │      │
│                 │ AQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQUJeIYDrJXk… │      │
│                 │ VR0jBBgwFoAU5K8rJnEaK0gnhS9SZizv8IkTcT4wa… │      │
│                 │ CCsGAQUFBzABhhpodHRwOi8vb2NzcC5wa2kuZ29vZ… │      │
│                 │ AoYkaHR0cDovL3BraS5nb29nL3JlcG8vY2VydHMvZ… │      │
│                 │ MCswKaAnoCWGI2h0dHA6Ly9jcmwucGtpLmdvb2cvZ… │      │
│                 │ A1UdIARGMEQwCAYGZ4EMAQIBMDgGCisGAQQB1nkCB… │      │
│                 │ aHR0cHM6Ly9wa2kuZ29vZy9yZXBvc2l0b3J5LzANB… │      │
│                 │ IVToy24jwXUr0rAPc924vuSVbKQuYw3nLflLfLh5A… │      │
│                 │ FZbhXkBH0PNcw97thaf2BeoDYY9Ck/b+UGluhx06z… │      │
│                 │ K+Xh3I0tqJy2rgOqNDflr5IMQ8ZTWA3yltakzSBKZ… │      │
│                 │ uPCJvscp1/m2pVTtyBjYPRQ+QuCQGAJKjtN7R5DFr… │      │
│                 │ 3cTIfzE7cmALskMKNLuDz+RzCcsYTsVaU7Vp3xL60… │      │
│                 │ YgPmOT4X3+7L51fXJyRH9KfLRP6nT31D5nmsGAOgZ… │      │
│                 │ VS5H0HyIBMEKyGMIPhFWrlt/hFS28N1zaKI0ZBGD3… │      │
│                 │ lVlWPzXe81vdoEnFbr5M272HdgJWo+WhT9BYM0Ji+… │      │
│                 │ 1dFpgJu8fF3LG0gl2ibSYiCi9a6hvU0TppjJyIWXh… │      │
│                 │ JDwRjW/656r0KVB02xHRKvm2ZKI03TglLIpmVCK3k… │      │
│                 │ x/9tpNFlQTl7B39rJlJWkR17QnZqVptFePFORoZmF… │      │
│                 │ -----END CERTIFICATE-----                  │      │
│                 │ -----BEGIN CERTIFICATE-----                │      │
│                 │ MIIFYjCCBEqgAwIBAgIQd70NbNs2+RrqIQ/E8FjTD… │      │
│                 │ MQswCQYDVQQGEwJCRTEZMBcGA1UEChMQR2xvYmFsU… │      │
│                 │ CxMHUm9vdCBDQTEbMBkGA1UEAxMSR2xvYmFsU2lnb… │      │
│                 │ OTAwMDA0MloXDTI4MDEyODAwMDA0MlowRzELMAkGA… │      │
│                 │ GUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBMTEMxFDASB… │      │
│                 │ MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCA… │      │
│                 │ ladAPKH9gvl9MgaCcfb2jH/76Nu8ai6Xl6OMS/kr9… │      │
│                 │ iV6nqlKr+CMny6SxnGPb15l+8Ape62im9MZaRw1NE… │      │
│                 │ KSUjB6G00j0uYODP0gmHu81I8E3CwnqIiru6z1kZ1… │      │
│                 │ DrXYfiYaRQM9sHmklCitD38m5agI/pboPGiUU+6DO… │      │
│                 │ j5ZPaK49l8KEj8C8QMALXL32h7M1bKwYUH+E4EzNk… │      │
│                 │ cuHKZPfmghCN6J3Cioj6OGaK/GP5Afl4/Xtcd/p2h… │      │
│                 │ CruOC7XFxYpVq9Os6pFLKcwZpDIlTirxZUTQAs6qz… │      │
│                 │ iYH6TKX/1Y7DzkvgtdizjkXPdsDtQCv9Uw+wp9U7D… │      │
│                 │ Eua++tgy/BBjFFFy3l3WFpO9KWgz7zpm7AeKJt8T1… │      │
│                 │ sZWwpbkNFhHax2xIPEDgfg1azVY80ZcFuctL7TlLn… │      │
│                 │ 9f6BQdgAmD06yK56mDcYBZUCAwEAAaOCATgwggE0M… │      │
│                 │ BgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTkrysmc… │      │
│                 │ BgNVHSMEGDAWgBRge2YaRQ2XyolQL30EzTSo//z9S… │      │
│                 │ JQYIKwYBBQUHMAGGGWh0dHA6Ly9vY3NwLnBraS5nb… │      │
│                 │ MAKGHWh0dHA6Ly9wa2kuZ29vZy9nc3IxL2dzcjEuY… │      │
│                 │ oCOGIWh0dHA6Ly9jcmwucGtpLmdvb2cvZ3NyMS9nc… │      │
│                 │ MAgGBmeBDAECATAIBgZngQwBAgIwDQYLKwYBBAHWe… │      │
│                 │ AwMwDQYJKoZIhvcNAQELBQADggEBADSkHrEoo9C0d… │      │
│                 │ NR3t5P+T4Vxfq7vqfM/b5A3Ri1fyJm9bvhdGaJQ3b… │      │
│                 │ WprKASOshIArAoyZl+tJaox118fessmXn1hIVw41o… │      │
│                 │ 9U5pCZEt4Wi4wStz6dTZ/CLANx8LZh1J7QJVj2fhM… │      │
│                 │ +qduBmpvvYuR7hZL6Dupszfnw0Skfths18dG9ZKb5… │      │
│                 │ d0lIKO2d1xozclOzgjXPYovJJIultzkMu34qQb9Sz… │      │
│                 │ -----END CERTIFICATE-----                  │      │
│                 │                                            │      │
│                 │ managed:                                   │      │
│                 │   domains:                                 │      │
│                 │     - "grucloud.org"                       │      │
│                 │   status: ACTIVE                           │      │
│                 │   domainStatus:                            │      │
│                 │     grucloud.org: ACTIVE                   │      │
│                 │ type: MANAGED                              │      │
│                 │ subjectAlternativeNames:                   │      │
│                 │   - "grucloud.org"                         │      │
│                 │ expireTime: 2021-09-28T03:48:42.000-07:00  │      │
│                 │ kind: compute#sslCertificate               │      │
│                 │                                            │      │
└─────────────────┴────────────────────────────────────────────┴──────┘


┌─────────────────────────────────────────────────────────────────────┐
│ 1 Bucket from google                                                │
├──────────────┬───────────────────────────────────────────────┬──────┤
│ Name         │ Data                                          │ Our  │
├──────────────┼───────────────────────────────────────────────┼──────┤
│ grucloud.org │ kind: storage#bucket                          │ Yes  │
│              │ selfLink: https://www.googleapis.com/storage… │      │
│              │ id: grucloud.org                              │      │
│              │ name: grucloud.org                            │      │
│              │ projectNumber: 91170824493                    │      │
│              │ metageneration: 2                             │      │
│              │ location: EUROPE-WEST4                        │      │
│              │ storageClass: STANDARD                        │      │
│              │ etag: CAI=                                    │      │
│              │ timeCreated: 2021-06-30T10:44:18.829Z         │      │
│              │ updated: 2021-06-30T10:44:19.640Z             │      │
│              │ website:                                      │      │
│              │   mainPageSuffix: index.html                  │      │
│              │   notFoundPage: 404.html                      │      │
│              │ labels:                                       │      │
│              │   managed-by: grucloud                        │      │
│              │   stage: dev                                  │      │
│              │ iamConfiguration:                             │      │
│              │   bucketPolicyOnly:                           │      │
│              │     enabled: true                             │      │
│              │     lockedTime: 2021-09-28T10:44:18.829Z      │      │
│              │   uniformBucketLevelAccess:                   │      │
│              │     enabled: true                             │      │
│              │     lockedTime: 2021-09-28T10:44:18.829Z      │      │
│              │   publicAccessPrevention: unspecified         │      │
│              │ locationType: region                          │      │
│              │ iam:                                          │      │
│              │   kind: storage#policy                        │      │
│              │   resourceId: projects/_/buckets/grucloud.org │      │
│              │   version: 1                                  │      │
│              │   etag: CAI=                                  │      │
│              │   bindings:                                   │      │
│              │     - role: roles/storage.legacyBucketOwner   │      │
│              │       members:                                │      │
│              │         - "projectEditor:grucloud-test"       │      │
│              │         - "projectOwner:grucloud-test"        │      │
│              │     - role: roles/storage.legacyBucketReader  │      │
│              │       members:                                │      │
│              │         - "projectViewer:grucloud-test"       │      │
│              │     - role: roles/storage.legacyObjectOwner   │      │
│              │       members:                                │      │
│              │         - "projectEditor:grucloud-test"       │      │
│              │         - "projectOwner:grucloud-test"        │      │
│              │     - role: roles/storage.legacyObjectReader  │      │
│              │       members:                                │      │
│              │         - "projectViewer:grucloud-test"       │      │
│              │     - role: roles/storage.objectViewer        │      │
│              │       members:                                │      │
│              │         - "allUsers"                          │      │
│              │                                               │      │
└──────────────┴───────────────────────────────────────────────┴──────┘


┌─────────────────────────────────────────────────────────────────────┐
│ 1 Object from google                                                │
├────────────┬─────────────────────────────────────────────────┬──────┤
│ Name       │ Data                                            │ Our  │
├────────────┼─────────────────────────────────────────────────┼──────┤
│ index.html │ kind: storage#object                            │ NO   │
│            │ id: grucloud.org/index.html/1625049860598468    │      │
│            │ selfLink: https://www.googleapis.com/storage/v… │      │
│            │ mediaLink: https://storage.googleapis.com/down… │      │
│            │ name: index.html                                │      │
│            │ bucket: grucloud.org                            │      │
│            │ generation: 1625049860598468                    │      │
│            │ metageneration: 1                               │      │
│            │ contentType: text/html                          │      │
│            │ storageClass: STANDARD                          │      │
│            │ size: 333                                       │      │
│            │ md5Hash: tVxlJ6kUMyBVNUg4XlUdJA==               │      │
│            │ crc32c: TISzGQ==                                │      │
│            │ etag: CMTVu72Wv/ECEAE=                          │      │
│            │ timeCreated: 2021-06-30T10:44:20.599Z           │      │
│            │ updated: 2021-06-30T10:44:20.599Z               │      │
│            │ timeStorageClassUpdated: 2021-06-30T10:44:20.5… │      │
│            │ metadata:                                       │      │
│            │   managed-by: grucloud                          │      │
│            │   stage: dev                                    │      │
│            │                                                 │      │
└────────────┴─────────────────────────────────────────────────┴──────┘


┌─────────────────────────────────────────────────────────────────────┐
│ 1 BackendBucket from google                                         │
├────────────────┬─────────────────────────────────────────────┬──────┤
│ Name           │ Data                                        │ Our  │
├────────────────┼─────────────────────────────────────────────┼──────┤
│ backend-bucket │ id: 8720064536531564011                     │ Yes  │
│                │ creationTimestamp: 2021-06-30T03:44:20.676… │      │
│                │ name: backend-bucket                        │      │
│                │ description: Managed By GruCloud            │      │
│                │ selfLink: https://www.googleapis.com/compu… │      │
│                │ bucketName: grucloud.org                    │      │
│                │ enableCdn: false                            │      │
│                │ kind: compute#backendBucket                 │      │
│                │                                             │      │
└────────────────┴─────────────────────────────────────────────┴──────┘


┌─────────────────────────────────────────────────────────────────────┐
│ 1 UrlMap from google                                                │
├──────────┬───────────────────────────────────────────────────┬──────┤
│ Name     │ Data                                              │ Our  │
├──────────┼───────────────────────────────────────────────────┼──────┤
│ url-map  │ id: 1187115894224057828                           │ Yes  │
│          │ creationTimestamp: 2021-06-30T03:44:27.793-07:00  │      │
│          │ name: url-map                                     │      │
│          │ description: Managed By GruCloud                  │      │
│          │ selfLink: https://www.googleapis.com/compute/v1/… │      │
│          │ defaultService: https://www.googleapis.com/compu… │      │
│          │ fingerprint: AFWyLkZ6QVA=                         │      │
│          │ kind: compute#urlMap                              │      │
│          │                                                   │      │
└──────────┴───────────────────────────────────────────────────┴──────┘


┌─────────────────────────────────────────────────────────────────────┐
│ 1 HttpsTargetProxy from google                                      │
├────────────────────┬─────────────────────────────────────────┬──────┤
│ Name               │ Data                                    │ Our  │
├────────────────────┼─────────────────────────────────────────┼──────┤
│ https-target-proxy │ id: 6271089959767242210                 │ Yes  │
│                    │ creationTimestamp: 2021-06-30T03:44:29… │      │
│                    │ name: https-target-proxy                │      │
│                    │ description: Managed By GruCloud        │      │
│                    │ selfLink: https://www.googleapis.com/c… │      │
│                    │ urlMap: https://www.googleapis.com/com… │      │
│                    │ c:                        │      │
│                    │   - "https://www.googleapis.com/comput… │      │
│                    │ fingerprint: _7WjZisemFI=               │      │
│                    │ kind: compute#targetHttpsProxy          │      │
│                    │                                         │      │
└────────────────────┴─────────────────────────────────────────┴──────┘


┌─────────────────────────────────────────────────────────────────────┐
│ 1 GlobalForwardingRule from google                                  │
├────────────────────────┬─────────────────────────────────────┬──────┤
│ Name                   │ Data                                │ Our  │
├────────────────────────┼─────────────────────────────────────┼──────┤
│ global-forwarding-rule │ id: 953550543260874208              │ Yes  │
│                        │ creationTimestamp: 2021-06-30T03:4… │      │
│                        │ name: global-forwarding-rule        │      │
│                        │ description: Managed By GruCloud    │      │
│                        │ IPAddress: 34.149.122.91            │      │
│                        │ IPProtocol: TCP                     │      │
│                        │ portRange: 443-443                  │      │
│                        │ target: https://www.googleapis.com… │      │
│                        │ selfLink: https://www.googleapis.c… │      │
│                        │ loadBalancingScheme: EXTERNAL       │      │
│                        │ networkTier: PREMIUM                │      │
│                        │ labelFingerprint: 42WmSpB8rSM=      │      │
│                        │ fingerprint: 3GGsVKhNuO4=           │      │
│                        │ kind: compute#forwardingRule        │      │
│                        │                                     │      │
└────────────────────────┴─────────────────────────────────────┴──────┘


List Summary:
Provider: google
┌────────────────────────────────────────────────────────────────────┐
│ google                                                             │
├────────────────────┬───────────────────────────────────────────────┤
│ SslCertificate     │ ssl-certificate                               │
├────────────────────┼───────────────────────────────────────────────┤
│ Bucket             │ grucloud.org                                  │
├────────────────────┼───────────────────────────────────────────────┤
│ Object             │ index.html                                    │
├────────────────────┼───────────────────────────────────────────────┤
│ BackendBucket      │ backend-bucket                                │
├────────────────────┼───────────────────────────────────────────────┤
│ UrlMap             │ url-map                                       │
├────────────────────┼───────────────────────────────────────────────┤
│ HttpsTargetProxy   │ https-target-proxy                            │
├────────────────────┼───────────────────────────────────────────────┤
│ GlobalForwardingR… │ global-forwarding-rule                        │
└────────────────────┴───────────────────────────────────────────────┘
7 resources, 7 types, 1 provider
Command "gc list" executed in 4s

Update

Let's deploy a new version of the website, in this very simple example, edit website/simple/index.html, change something, and save the file.

The plan command helps to find out what is going to be deployed:

gc plan

Querying resources on 1 provider: google
✓ google
  ✓ Initialising
  ✓ Listing 7/7
  ✓ Querying
    ✓ Bucket 1/1
    ✓ Object 1/1
    ✓ SslCertificate 1/1
    ✓ BackendBucket 1/1
    ✓ UrlMap 1/1
    ✓ HttpsTargetProxy 1/1
    ✓ GlobalForwardingRule 1/1
┌───────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ 1 Object from google                                                                                  │
├───────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ┌─────────────────────────────────────────────────────────────────────────────────────────────────┐   │
│ │ UPDATE: name: index.html, id: grucloud.org/index.html/1625068113228566                          │   │
│ ├─────────────────────────────────────────────────────────────────────────────────────────────────┤   │
│ │ Key: md5                                                                                        │   │
│ ├────────────────────────────────────────────────┬────────────────────────────────────────────────┤   │
│ │ - nnLSNGP9RyK8uy7mni91nA==                     │ + tVxlJ6kUMyBVNUg4XlUdJA==                     │   │
│ └────────────────────────────────────────────────┴────────────────────────────────────────────────┘   │
└───────────────────────────────────────────────────────────────────────────────────────────────────────┘


┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Plan summary for provider google                                                               │
├────────────────────────────────────────────────────────────────────────────────────────────────┤
│ DEPLOY RESOURCES                                                                               │
├────────────────────┬───────────────────────────────────────────────────────────────────────────┤
│ Object             │ index.html                                                                │
└────────────────────┴───────────────────────────────────────────────────────────────────────────┘
1 resource to deploy on 1 provider
Command "gc p" executed in 4s

In this case, gc computes the MD5 hash of the file and compares it with the live version.
Next, use the apply command to effectively deploy.

Destroy

Dispose of the infrastructure in the right order with:

gc destroy

┌─────────────────────────────────────────────────────────────────────────────┐
│ Destroy summary for provider google                                         │
├────────────────────┬────────────────────────────────────────────────────────┤
│ SslCertificate     │ ssl-certificate                                        │
├────────────────────┼────────────────────────────────────────────────────────┤
│ Bucket             │ grucloud.org                                           │
├────────────────────┼────────────────────────────────────────────────────────┤
│ BackendBucket      │ backend-bucket                                         │
├────────────────────┼────────────────────────────────────────────────────────┤
│ UrlMap             │ url-map                                                │
├────────────────────┼────────────────────────────────────────────────────────┤
│ HttpsTargetProxy   │ https-target-proxy                                     │
├────────────────────┼────────────────────────────────────────────────────────┤
│ GlobalForwardingR… │ global-forwarding-rule                                 │
└────────────────────┴────────────────────────────────────────────────────────┘
Destroying resources on 1 provider: google
✓ google
  ✓ Initialising
  ✓ Destroying
    ✓ SslCertificate 1/1
    ✓ Bucket 1/1
    ✓ BackendBucket 1/1
    ✓ UrlMap 1/1
    ✓ HttpsTargetProxy 1/1
    ✓ GlobalForwardingRule 1/1
  ✓ default::onDestroyed
    ✓ remove the load balancer A DNS record
6 resources destroyed, 6 types on 1 provider
Running OnDestroyedGlobal resources on 1 provider: google
Command "gc d -f" executed in 56s

Deploy, update, and destroy AWS EC2 Instance automatically

Frederic — Fri, 30 Apr 2021 02:29:19 +0000

This tutorial explains the deployment automation of a simple AWS EC2 instance with the GruCloud AWS provider.

Instead of manually creating, updating, and destroying EC2 instances, the infrastructure will be described as Javascript code. The GruCloud CLI then reads this code, retrieves the lives resources through the AWS API, and decides what needs to be created, updated, or destroyed.

Requirements

AWS Account

Ensure access to the Amazon Console and create an account if necessary.

AWS CLI

Ensure the AWS CLI is installed and configured:

aws --version

If not, visit https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html

Access and Secret Key

Visit the security credentials

Click on Access key (access key ID and secret access key).
Click on the button Create New Access Key.

Write down the AWSAccessKeyId and AWSSecretKey

In a further episode, the access and secret key will be obtained from a dedicated IAM user with the correct role and policy.

Configure AWS CLI

Configure the account with the previously obtained AWSAccessKeyId and AWSSecretKey, as well as the region, for instance us-east-1

aws configure

Getting the GruCloud CLI

GruCloud is written in Javascript running on Node.js. Check if node is present on your system:

node --version

The version must be greater than 14

Install the GrucCloud command-line utility gc with npm

npm i -g @grucloud/core

Check the current version of gc:

gc --version

Code Architecture

In this section, we'll create the files needed to describe an infrastructure with GruCloud:

package.json: specifies the npm dependencies and other information.
config.js: the config function.
iac.js: exports createStack with provider and resources associated
hooks.js: optionally provides hook functions called after deployment or destruction.

The source code for this example in on GitHub.

Create a new project

Create a new directory, for instance ec2-example:

mkdir ec2-example
cd ec2-example

package.json

Let's create a new package.json with the npm init command:

npm init

Let's install the GruCloud AWS provider called @grucloud/provider-aws, as well as the GruCloud core library @grucloud/core.

npm install @grucloud/core @grucloud/provider-aws

config.js

The configuration is a Javascript function located in config.js

// config.js
const pkg = require("./package.json");
module.exports = ({ stage }) => ({
  projectName: pkg.name,
  ec2Instance: {
    name: "web-server",
    properties: {
      InstanceType: "t2.micro",
      ImageId: "ami-00f6a0c18edb19300", // Ubuntu 18.04
    },
  },
});

You will have to find out the ImageId for your specific region. One way to retrieve to list of images is with the aws cli:

aws ec2 describe-images --filters "Name=description,Values=Ubuntu Server 20.04 LTS" "Name=architecture,Values=x86_64"

This step will be automated in a future episode with the help of the Amazon Managed Image resource.

iac.js

Create a file called iac.js which stands for infrastructure as code.
We'll first import AwsProvider from @grucloud/provider-aws

iac.js must export the createStack function which returns the provider and the resources.

Then, instantiate AwsProvider by providing the config function.

In the case, an EC2 Instance is defined with provider.makeEC2.

// iac.js
const { AwsProvider } = require("@grucloud/provider-aws");

exports.createStack = async ({ stage }) => {
  const provider = AwsProvider({ config: require("./config"), stage });
  const { config } = provider;
  const ec2Instance = await provider.makeEC2({
    name: config.ec2Instance.name,
    properties: () => config.ec2Instance.properties,
  });

  return {
    provider,
    resources: { ec2Instance },
  };
};

GruCloud CLI

Info

As a way to verify that gc can connect to the AWS API, one can use the info command:

gc info

  - provider:
      name: aws
      type: aws
    stage: dev
    accountId: 840541460064
    zone: eu-west-2a
    config:
      projectName: ec2-simple
      ec2Instance:
        name: web-server
        properties:
          InstanceType: t2.micro
          ImageId: ami-00f6a0c18edb19300
      stage: dev
      zone: [object Function]
      accountId: [object Function]
      region: eu-west-2

Command "gc info" executed in 1s

Deploy

We are ready to deploy with apply command:

gc apply

Querying resources on 1 provider: aws
✓ aws
  ✓ Initialising
  ✓ Listing 6/6
  ✓ Querying
    ✓ EC2 1/1
┌──────────────────────────────────────────────────────────────────────────┐
│ 1 EC2 from aws                                                           │
├────────────┬──────────┬──────────────────────────────────────────────────┤
│ Name       │ Action   │ Data                                             │
├────────────┼──────────┼──────────────────────────────────────────────────┤
│ web-server │ CREATE   │ InstanceType: t2.micro                           │
│            │          │ ImageId: ami-00f6a0c18edb19300                   │
│            │          │ MaxCount: 1                                      │
│            │          │ MinCount: 1                                      │
│            │          │ TagSpecifications:                               │
│            │          │   - ResourceType: instance                       │
│            │          │     Tags:                                        │
│            │          │       - Key: Name                                │
│            │          │         Value: web-server                        │
│            │          │       - Key: ManagedBy                           │
│            │          │         Value: GruCloud                          │
│            │          │       - Key: CreatedByProvider                   │
│            │          │         Value: aws                               │
│            │          │       - Key: stage                               │
│            │          │         Value: dev                               │
│            │          │       - Key: projectName                         │
│            │          │         Value: ec2-simple                        │
│            │          │                                                  │
└────────────┴──────────┴──────────────────────────────────────────────────┘


┌─────────────────────────────────────────────────────────────────────────┐
│ Plan summary for provider aws                                           │
├─────────────────────────────────────────────────────────────────────────┤
│ DEPLOY RESOURCES                                                        │
├────────────────────┬────────────────────────────────────────────────────┤
│ EC2                │ web-server                                         │
└────────────────────┴────────────────────────────────────────────────────┘
? Are you sure to deploy 1 resource, 1 type on 1 provider? › (y/N)

At this point, you are given the opportunity to look at what is going to be deployed.
Type y to accept:

Deploying resources on 1 provider: aws
✓ aws
  ✓ Initialising
  ✓ Deploying
    ✓ EC2 1/1
1 resource deployed of 1 type and 1 provider
Running OnDeployedGlobal resources on 1 provider: aws
Command "gc a" executed in 1m 29s

When gc apply is executed a second time, resources should not be created or destroyed.

gc a

Querying resources on 1 provider: aws
✓ aws
  ✓ Initialising
  ✓ Listing 13/13
  ✓ Querying
    ✓ EC2 1/1
┌─────────────────────────────────────────────────────────────────────────────────────────┐
│ Plan summary for provider aws                                                           │
└─────────────────────────────────────────────────────────────────────────────────────────┘
Nothing to deploy
Running OnDeployedGlobal resources on 1 provider: aws
Command "gc a" executed in 9s

As expected, nothing to deploy, the target resources defined in the code match the live resources.

List

To list all the resources, and generate a diagram:

gc list --graph --all

This will include the default AWS resources such as VPC, subnet, internet gateway, and security group.

To list only the resources created by GruCloud:

gc list --our

Listing resources on 1 provider: aws
✓ aws
  ✓ Initialising
  ✓ Listing 13/13
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ 1 EC2 from aws                                                                                          │
├────────────┬─────────────────────────────────────────────────────────────────────────────────────┬──────┤
│ Name       │ Data                                                                                │ Our  │
├────────────┼─────────────────────────────────────────────────────────────────────────────────────┼──────┤
│ web-server │ AmiLaunchIndex: 0                                                                   │ Yes  │
│            │ ImageId: ami-00f6a0c18edb19300                                                      │      │
│            │ InstanceId: i-016628a1b6a6a2f91                                                     │      │
│            │ InstanceType: t3.micro                                                              │      │
│            │ LaunchTime: 2021-04-29T23:10:44.000Z                                                │      │
│            │ Monitoring:                                                                         │      │
│            │   State: disabled                                                                   │      │
│            │ Placement:                                                                          │      │
│            │   AvailabilityZone: eu-west-2a                                                      │      │
│            │   GroupName:                                                                        │      │
│            │   Tenancy: default                                                                  │      │
│            │ PrivateDnsName: ip-172-31-2-146.eu-west-2.compute.internal                          │      │
│            │ PrivateIpAddress: 172.31.2.146                                                      │      │
│            │ ProductCodes: []                                                                    │      │
│            │ PublicDnsName: ec2-18-132-98-216.eu-west-2.compute.amazonaws.com                    │      │
│            │ PublicIpAddress: 18.132.98.216                                                      │      │
│            │ State:                                                                              │      │
│            │   Code: 16                                                                          │      │
│            │   Name: running                                                                     │      │
│            │ StateTransitionReason:                                                              │      │
│            │ SubnetId: subnet-0f6f085fc384bf8ce                                                  │      │
│            │ VpcId: vpc-bbbafcd3                                                                 │      │
│            │ Architecture: x86_64                                                                │      │
│            │ BlockDeviceMappings:                                                                │      │
│            │   - DeviceName: /dev/sda1                                                           │      │
│            │     Ebs:                                                                            │      │
│            │       AttachTime: 2021-04-29T23:10:45.000Z                                          │      │
│            │       DeleteOnTermination: true                                                     │      │
│            │       Status: attached                                                              │      │
│            │       VolumeId: vol-0a88a34248fe18740                                               │      │
│            │ ClientToken: 7ef5fd60-e962-400f-a2b9-37719d8329ed                                   │      │
│            │ EbsOptimized: false                                                                 │      │
│            │ EnaSupport: true                                                                    │      │
│            │ Hypervisor: xen                                                                     │      │
│            │ ElasticGpuAssociations: []                                                          │      │
│            │ ElasticInferenceAcceleratorAssociations: []                                         │      │
│            │ NetworkInterfaces:                                                                  │      │
│            │   - Association:                                                                    │      │
│            │       IpOwnerId: amazon                                                             │      │
│            │       PublicDnsName: ec2-18-132-98-216.eu-west-2.compute.amazonaws.com              │      │
│            │       PublicIp: 18.132.98.216                                                       │      │
│            │     Attachment:                                                                     │      │
│            │       AttachTime: 2021-04-29T23:10:44.000Z                                          │      │
│            │       AttachmentId: eni-attach-02744addcff43ecbb                                    │      │
│            │       DeleteOnTermination: true                                                     │      │
│            │       DeviceIndex: 0                                                                │      │
│            │       Status: attached                                                              │      │
│            │       NetworkCardIndex: 0                                                           │      │
│            │     Description:                                                                    │      │
│            │     Groups:                                                                         │      │
│            │       - GroupName: default                                                          │      │
│            │         GroupId: sg-f4139a96                                                        │      │
│            │     Ipv6Addresses: []                                                               │      │
│            │     MacAddress: 06:ee:ef:d7:1c:48                                                   │      │
│            │     NetworkInterfaceId: eni-06e9d87ff776cf40f                                       │      │
│            │     OwnerId: 840541460064                                                           │      │
│            │     PrivateDnsName: ip-172-31-2-146.eu-west-2.compute.internal                      │      │
│            │     PrivateIpAddress: 172.31.2.146                                                  │      │
│            │     PrivateIpAddresses:                                                             │      │
│            │       - Association:                                                                │      │
│            │           IpOwnerId: amazon                                                         │      │
│            │           PublicDnsName: ec2-18-132-98-216.eu-west-2.compute.amazonaws.com          │      │
│            │           PublicIp: 18.132.98.216                                                   │      │
│            │         Primary: true                                                               │      │
│            │         PrivateDnsName: ip-172-31-2-146.eu-west-2.compute.internal                  │      │
│            │         PrivateIpAddress: 172.31.2.146                                              │      │
│            │     SourceDestCheck: true                                                           │      │
│            │     Status: in-use                                                                  │      │
│            │     SubnetId: subnet-0f6f085fc384bf8ce                                              │      │
│            │     VpcId: vpc-bbbafcd3                                                             │      │
│            │     InterfaceType: interface                                                        │      │
│            │ RootDeviceName: /dev/sda1                                                           │      │
│            │ RootDeviceType: ebs                                                                 │      │
│            │ SecurityGroups:                                                                     │      │
│            │   - GroupName: default                                                              │      │
│            │     GroupId: sg-f4139a96                                                            │      │
│            │ SourceDestCheck: true                                                               │      │
│            │ Tags:                                                                               │      │
│            │   - Key: Name                                                                       │      │
│            │     Value: web-server                                                               │      │
│            │   - Key: stage                                                                      │      │
│            │     Value: dev                                                                      │      │
│            │   - Key: CreatedByProvider                                                          │      │
│            │     Value: aws                                                                      │      │
│            │   - Key: ManagedBy                                                                  │      │
│            │     Value: GruCloud                                                                 │      │
│            │   - Key: projectName                                                                │      │
│            │     Value: ec2-simple                                                               │      │
│            │ VirtualizationType: hvm                                                             │      │
│            │ CpuOptions:                                                                         │      │
│            │   CoreCount: 1                                                                      │      │
│            │   ThreadsPerCore: 2                                                                 │      │
│            │ CapacityReservationSpecification:                                                   │      │
│            │   CapacityReservationPreference: open                                               │      │
│            │ HibernationOptions:                                                                 │      │
│            │   Configured: false                                                                 │      │
│            │ Licenses: []                                                                        │      │
│            │ MetadataOptions:                                                                    │      │
│            │   State: applied                                                                    │      │
│            │   HttpTokens: optional                                                              │      │
│            │   HttpPutResponseHopLimit: 1                                                        │      │
│            │   HttpEndpoint: enabled                                                             │      │
│            │ EnclaveOptions:                                                                     │      │
│            │   Enabled: false                                                                    │      │
│            │                                                                                     │      │
└────────────┴─────────────────────────────────────────────────────────────────────────────────────┴──────┘


List Summary:
Provider: aws
┌────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ aws                                                                                                    │
├────────────────────┬───────────────────────────────────────────────────────────────────────────────────┤
│ EC2                │ web-server                                                                        │
└────────────────────┴───────────────────────────────────────────────────────────────────────────────────┘
1 resource, 7 types, 1 provider
Command "gc l -o" executed in 7s

Note that tags have been added to the EC2 Instance, it gives GruCloud a way to identify the resources under its control. Unlike other infrastructure as code tools such as Terraform and Pulumi, GruCloud does not need a state file. Hence removing a lot of complexity and issues.

Update

The ec2 instance configuration might change, for instance, let's modify the machine type to t3.micro located in config.js

The plan command is a read-only command which fetches the live resources and compares them with the target resources defined in the code.

gc plan

Querying resources on 1 provider: aws
✓ aws
  ✓ Initialising
  ✓ Listing 13/13
  ✓ Querying
    ✓ EC2 1/1
┌──────────────────────────────────────────────────────────────────────────────────────────┐
│ 1 EC2 from aws                                                                           │
├────────────┬──────────┬──────────────────────────────────────────────────────────────────┤
│ Name       │ Action   │ Data                                                             │
├────────────┼──────────┼──────────────────────────────────────────────────────────────────┤
│ web-server │ UPDATE   │ added:                                                           │
│            │          │ updated:                                                         │
│            │          │   InstanceType: t3.micro                                         │
│            │          │                                                                  │
└────────────┴──────────┴──────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────────────────────────┐
│ Plan summary for provider aws                                                           │
├─────────────────────────────────────────────────────────────────────────────────────────┤
│ DEPLOY RESOURCES                                                                        │
├────────────────────┬────────────────────────────────────────────────────────────────────┤
│ EC2                │ web-server                                                         │
└────────────────────┴────────────────────────────────────────────────────────────────────┘
1 resource to deploy on 1 provider
Command "gc plan" executed in 5s

Let's apply the change:

gc apply

Bear in mind that the machine requires to be stopped and restarted.

Trust but verify, hence, list the EC2 instances and check the InstanceType has been changed.

gc l -t EC2

Output

Another useful command is gc output, which extract information for a specific field of a given resource

gc output -t EC2 --name web-server -f InstanceType

t3.micro

Nested field can be accessed too, for example, let's retrieve the public IP address attached to the EC2 instance:

gc output -t EC2 --name web-server -f 'NetworkInterfaces[0].Association.PublicIp'

18.130.34.212

The temptation to ping is high:

ping `gc output -t EC2 --name web-server -f 'NetworkInterfaces[0].Association.PublicIp'`

PING 18.130.34.212 (18.130.34.212): 56 data bytes
64 bytes from 18.130.34.212: icmp_seq=0 ttl=38 time=153.799 ms
64 bytes from 18.130.34.212: icmp_seq=1 ttl=38 time=153.583 ms
64 bytes from 18.130.34.212: icmp_seq=2 ttl=38 time=140.622 ms

Destroy

Time to destroy the resources allocated and therefore save a lot of £$€.

gc destroy

Find Deletable resources on 1 provider: aws
✓ aws
  ✓ Initialising
  ✓ Listing 6/6
? Are you sure to destroy 1 resource, 1 type on 1 provider? › (y/N)

Once again, you are given the opportunity to look at what is going to be destroyed.
Type 'y' to confirm the destruction:

Destroying resources on 1 provider: aws
✓ aws
  ✓ Initialising
  ✓ Destroying
    ✓ EC2 1/1
1 resource destroyed, 1 type on 1 provider
Running OnDestroyedGlobal resources on 1 provider: aws
Command "gc destroy" executed in 1m 29s

Let's run the gc list command with the E2 filter to verify the EC2 is gone:

gc l -t EC2

Listing resources on 1 provider: aws
✓ aws
  ✓ Initialising
  ✓ Listing 6/6
List Summary:
Provider: aws
┌─────────────────────────────────────────────────────────────────────────┐
│ aws                                                                     │
└─────────────────────────────────────────────────────────────────────────┘
0 resources, 0 types, 1 provider
Command "gc list -t EC2" executed in 3s

This example demonstrates how to code a very basic infrastructure with one EC2 instance, and how can we use the gc apply, gc list, gc destroy and gc graph to manage the infrastructure.

It paves the way for more AWS examples

Graph

A picture is worth a thousand words, GruCloud generates an SVG file describing the resources and their relationship.

gc graph

Here is the graph of a typical web application managed by Kubernetes running on AWS where the master node is managed by EKS.

Kubernetes defined as Javascript Code with GruCloud

Frederic — Wed, 28 Apr 2021 23:11:22 +0000

Introduction

The Kubernetes Grucloud provider allows to define and describe Kubernetes manifests in Javascript, removing the need to write YAML or template files.

The GruCloud Command Line Interface gc reads a description in Javascript and connects to the k8s control plane to apply the new or updated resource definitions.

For this tutorial, we will define a Namespace, a Service, and a Deployment to deploy an Nginx web server.

This diagram is generated from the code with gc graph

Requirements

Ensure kubectl is installed, and minikube is started: K8s Requirements

It is always a good idea to verify the current kubectl context, especially when switching k8s clusters:

kubectl config current-context

Getting GruCloud CLI: gc

The GruCloud CLI, gc, is written in Javascript and run on Node.js, hence node is required:

node --version

Install gc in just one command:

npm i -g @grucloud/core

Verify gc is installed properly by displaying the version:

gc --version

Project Content

We'll describe in the next sections the 4 files required for this infrastructure as code project:

The link to the source code for this tutorial

Let's create a new project directory

mkdir tuto
cd tuto

package.json

The npm init command will create a basic package.json:

npm init

Let's install the GruCloud Kubernetes provider and the SDK. We'll also install axios and rubico, needed for the post-deployment hooks, which does some final health check.

npm install @grucloud/core @grucloud/provider-k8s rubico axios

config.js

Create the config.js which contains the configuration for this project:

// config.js
const pkg = require("./package.json");
module.exports = () => ({
  projectName: pkg.name,
  namespace: "myapp",
  appLabel: "nginx-label",
  service: { name: "nginx-service" },
  deployment: {
    name: "nginx-deployment",
    container: { name: "nginx", image: "nginx:1.14.2" },
  },
});

iac.js

Let's create the iac.js with the following content:

// iac.js
const { K8sProvider } = require("@grucloud/provider-k8s");

// Create a namespace, service and deployment
const createResource = async ({ provider }) => {
  const { config } = provider;

  const namespace = await provider.makeNamespace({
    name: config.namespace,
  });

  const service = await provider.makeService({
    name: config.service.name,
    dependencies: { namespace },
    properties: () => ({
      spec: {
        selector: {
          app: config.appLabel,
        },
        type: "NodePort",
        ports: [
          {
            protocol: "TCP",
            port: 80,
            targetPort: 80,
            nodePort: 30020,
          },
        ],
      },
    }),
  });

  const deployment = await provider.makeDeployment({
    name: config.deployment.name,
    dependencies: { namespace },
    properties: ({}) => ({
      metadata: {
        labels: {
          app: config.appLabel,
        },
      },
      spec: {
        replicas: 1,
        selector: {
          matchLabels: {
            app: config.appLabel,
          },
        },
        template: {
          metadata: {
            labels: {
              app: config.appLabel,
            },
          },
          spec: {
            containers: [
              {
                name: config.deployment.container.name,
                image: config.deployment.container.image,
                ports: [
                  {
                    containerPort: 80,
                  },
                ],
              },
            ],
          },
        },
      },
    }),
  });

  return { namespace, service, deployment };
};

exports.createStack = async ({ config }) => {
  const provider = K8sProvider({ config });
  const resources = await createResource({ provider });
  return { provider, resources, hooks: [require("./hook")] };
};

hook.js

When the resources are created, any code can be invoked, defined in hook.js, useful to perform some final health check.

In this case, the kubectl port-forward is called with the right option:

kubectl --namespace myapp port-forward svc/nginx-service 8081:80

Then, we'll use the axios library to perform HTTP calls to the web server, retrying if necessary.

When the website is up, it will open a browser at http://localhost:8081

// hook.js
const assert = require("assert");
const Axios = require("axios");
const { pipe, tap, eq, get, or } = require("rubico");
const { first } = require("rubico/x");
const { retryCallOnError } = require("@grucloud/core").Retry;
const shell = require("shelljs");

module.exports = ({ resources, provider }) => {
  const localPort = 8081;
  const url = `http://localhost:${localPort}`;

  const servicePort = pipe([
    () => resources.service.properties({}),
    get("spec.ports"),
    first,
    get("port"),
  ])();

  const kubectlPortForwardCommand = `kubectl --namespace ${resources.namespace.name} port-forward svc/${resources.service.name} ${localPort}:${servicePort}`;

  const axios = Axios.create({
    timeout: 15e3,
    withCredentials: true,
  });

  return {
    onDeployed: {
      init: async () => {},
      actions: [
        {
          name: `exec: '${kubectlPortForwardCommand}', check web server at ${url}`,
          command: async () => {
            // start kubectl port-forward
            var child = shell.exec(kubectlPortForwardCommand, { async: true });
            child.stdout.on("data", function (data) {});

            // Get the web page, retry until it succeeds
            await retryCallOnError({
              name: `get ${url}`,
              fn: () => axios.get(url),
              shouldRetryOnException: ({ error }) =>
                or([
                  eq(get("code"), "ECONNREFUSED"),
                  eq(get("response.status"), 404),
                ])(error),
              isExpectedResult: (result) => {
                assert(result.headers["content-type"], `text/html`);
                return [200].includes(result.status);
              },
              config: { retryCount: 20, retryDelay: 5e3 },
            });
            // Open a browser
            shell.exec(`open ${url}`, { async: true });
          },
        },
      ],
    },
    onDestroyed: {
      init: () => {},
    },
  };
};

Workflow

We'll describe the most useful gc commands: apply, list, destroy, and plan.

Deploy

We are now ready to deploy the resources with the apply command:

gc apply

The first part is to find out the plan, i.e what is going to be deployed.
You will be prompted if you accept or abort.
When typing: 'y', the resources will be deployed: a namespace, a service, and a deployment.

Querying resources on 1 provider: k8s
✓ k8s
  ✓ Initialising
  ✓ Listing 7/7
  ✓ Querying
    ✓ Namespace 1/1
    ✓ Service 1/1
    ✓ Deployment 1/1
┌──────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ 1 Namespace from k8s                                                                                 │
├──────────┬──────────┬────────────────────────────────────────────────────────────────────────────────┤
│ Name     │ Action   │ Data                                                                           │
├──────────┼──────────┼────────────────────────────────────────────────────────────────────────────────┤
│ myapp    │ CREATE   │ apiVersion: v1                                                                 │
│          │          │ kind: Namespace                                                                │
│          │          │ metadata:                                                                      │
│          │          │   name: myapp                                                                  │
│          │          │   annotations:                                                                 │
│          │          │     Name: myapp                                                                │
│          │          │     ManagedBy: GruCloud                                                        │
│          │          │     CreatedByProvider: k8s                                                     │
│          │          │     stage: dev                                                                 │
│          │          │     projectName: @grucloud/example-k8s-tuto1                                   │
│          │          │                                                                                │
└──────────┴──────────┴────────────────────────────────────────────────────────────────────────────────┘


┌──────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ 1 Service from k8s                                                                                   │
├──────────────────────┬──────────┬────────────────────────────────────────────────────────────────────┤
│ Name                 │ Action   │ Data                                                               │
├──────────────────────┼──────────┼────────────────────────────────────────────────────────────────────┤
│ myapp::nginx-service │ CREATE   │ spec:                                                              │
│                      │          │   selector:                                                        │
│                      │          │     app: nginx-label                                               │
│                      │          │   type: NodePort                                                   │
│                      │          │   ports:                                                           │
│                      │          │     - protocol: TCP                                                │
│                      │          │       port: 80                                                     │
│                      │          │       targetPort: 8080                                             │
│                      │          │ apiVersion: v1                                                     │
│                      │          │ kind: Service                                                      │
│                      │          │ metadata:                                                          │
│                      │          │   name: nginx-service                                              │
│                      │          │   annotations:                                                     │
│                      │          │     Name: nginx-service                                            │
│                      │          │     ManagedBy: GruCloud                                            │
│                      │          │     CreatedByProvider: k8s                                         │
│                      │          │     stage: dev                                                     │
│                      │          │     projectName: @grucloud/example-k8s-tuto1                       │
│                      │          │   namespace: myapp                                                 │
│                      │          │                                                                    │
└──────────────────────┴──────────┴────────────────────────────────────────────────────────────────────┘


┌──────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ 1 Deployment from k8s                                                                                │
├─────────────────────────┬──────────┬─────────────────────────────────────────────────────────────────┤
│ Name                    │ Action   │ Data                                                            │
├─────────────────────────┼──────────┼─────────────────────────────────────────────────────────────────┤
│ myapp::nginx-deployment │ CREATE   │ metadata:                                                       │
│                         │          │   labels:                                                       │
│                         │          │     app: nginx-label                                            │
│                         │          │   name: nginx-deployment                                        │
│                         │          │   annotations:                                                  │
│                         │          │     Name: nginx-deployment                                      │
│                         │          │     ManagedBy: GruCloud                                         │
│                         │          │     CreatedByProvider: k8s                                      │
│                         │          │     stage: dev                                                  │
│                         │          │     projectName: @grucloud/example-k8s-tuto1                    │
│                         │          │   namespace: myapp                                              │
│                         │          │ spec:                                                           │
│                         │          │   replicas: 1                                                   │
│                         │          │   selector:                                                     │
│                         │          │     matchLabels:                                                │
│                         │          │       app: nginx-label                                          │
│                         │          │   template:                                                     │
│                         │          │     metadata:                                                   │
│                         │          │       labels:                                                   │
│                         │          │         app: nginx-label                                        │
│                         │          │     spec:                                                       │
│                         │          │       containers:                                               │
│                         │          │         - name: nginx                                           │
│                         │          │           image: nginx:1.14.2                                   │
│                         │          │           ports:                                                │
│                         │          │             - containerPort: 80                                 │
│                         │          │ apiVersion: apps/v1                                             │
│                         │          │ kind: Deployment                                                │
│                         │          │                                                                 │
└─────────────────────────┴──────────┴─────────────────────────────────────────────────────────────────┘


┌─────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Plan summary for provider k8s                                                                       │
├─────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ DEPLOY RESOURCES                                                                                    │
├────────────────────┬────────────────────────────────────────────────────────────────────────────────┤
│ Namespace          │ myapp                                                                          │
├────────────────────┼────────────────────────────────────────────────────────────────────────────────┤
│ Service            │ myapp::nginx-service                                                           │
├────────────────────┼────────────────────────────────────────────────────────────────────────────────┤
│ Deployment         │ myapp::nginx-deployment                                                        │
└────────────────────┴────────────────────────────────────────────────────────────────────────────────┘
✔ Are you sure to deploy 3 resources, 3 types on 1 provider? … yes
Deploying resources on 1 provider: k8s
✓ k8s
  ✓ Initialising
  ✓ Deploying
    ✓ Namespace 1/1
    ✓ Service 1/1
    ✓ Deployment 1/1
3 resources deployed of 3 types and 1 provider
Running OnDeployedGlobal resources on 1 provider: k8s
Command "gc a" executed in 30s

In the case of the Deployment type manifest, gc will query the pod that is started by the deployment through the replica set, when one of the container's pod is ready, the deployment can proceed.

Later on, when we deal with the Ingress type, gc will wait for the load balancer to be ready.

The command gc apply is the equivalent of kubectl apply -f mymanifest.yaml but it waits for resources to be up and running, ready to serve.

We could try to run the gc apply or the gc plan, we should not expect any deployment or destruction of resources.

In the mathematical and computer science world, we could say that apply (and destroy) commands are idempotent: "property of certain operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the initial application".

List

Let's verify that the resources are deployed with the gc list command:

A live diagram will be also generated.

gc list --our --all --graph

List Summary:
Provider: k8s
┌─────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ k8s                                                                                                 │
├────────────────────┬────────────────────────────────────────────────────────────────────────────────┤
│ Namespace          │ myapp                                                                          │
├────────────────────┼────────────────────────────────────────────────────────────────────────────────┤
│ Service            │ myapp::nginx-service                                                           │
├────────────────────┼────────────────────────────────────────────────────────────────────────────────┤
│ Deployment         │ myapp::nginx-deployment                                                        │
├────────────────────┼────────────────────────────────────────────────────────────────────────────────┤
│ ReplicaSet         │ myapp::nginx-deployment-66cdc8d56b                                             │
├────────────────────┼────────────────────────────────────────────────────────────────────────────────┤
│ Pod                │ myapp::nginx-deployment-66cdc8d56b-4d8lz                                       │
└────────────────────┴────────────────────────────────────────────────────────────────────────────────┘
5 resources, 15 types, 1 provider
Command "gc list --our --all --graph" executed in 0s

Notice the relationship between the Pod, ReplicaSet and Deployment.

The Deployment creates a ReplicaSet which creates a one or more Pod(s).

When querying the k8s-api-server for the live resources, the pod contains information about its ReplicaSet parent, who has itself information about its parent Deployment. This allows gc to find out the links between the resources.

Post Deploy Hook

Would like to check the health of the system? You can run the onDeployed hook any time with the following command:

gc run --onDeployed

Running OnDeployed resources on 1 provider: k8s
Forwarding from 127.0.0.1:8081 -> 80
Forwarding from [::1]:8081 -> 80
Handling connection for 8081
✓ k8s
  ✓ Initialising
  ✓ default::onDeployed
    ✓ exec: 'kubectl --namespace myapp port-forward svc/nginx-service 8081:80', check web server at http://localhost:8081
Command "gc run --onDeployed" executed in 5s

Update

Now that the initial deployment is successful, some changes will be made, for instance, let's change the Nginx container version, located at config.js.

Browse the list of Nginx images at https://hub.docker.com/_/nginx

Let's try the version nginx:1.20.0-alpine.

For a preview of the change that will be made, use the plan command:

gc plan

Querying resources on 1 provider: k8s
✓ k8s
  ✓ Initialising
  ✓ Listing 7/7
  ✓ Querying
    ✓ Namespace 1/1
    ✓ Service 1/1
    ✓ Deployment 1/1
┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ 1 Deployment from k8s                                                                                             │
├─────────────────────────┬──────────┬──────────────────────────────────────────────────────────────────────────────┤
│ Name                    │ Action   │ Data                                                                         │
├─────────────────────────┼──────────┼──────────────────────────────────────────────────────────────────────────────┤
│ myapp::nginx-deployment │ UPDATE   │ added:                                                                       │
│                         │          │ deleted:                                                                     │
│                         │          │ updated:                                                                     │
│                         │          │   spec:                                                                      │
│                         │          │     template:                                                                │
│                         │          │       spec:                                                                  │
│                         │          │         containers:                                                          │
│                         │          │           0:                                                                 │
│                         │          │             image: nginx:1.20.0-alpine                                       │
│                         │          │                                                                              │
└─────────────────────────┴──────────┴──────────────────────────────────────────────────────────────────────────────┘


┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Plan summary for provider k8s                                                                                    │
├──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ DEPLOY RESOURCES                                                                                                 │
├────────────────────┬─────────────────────────────────────────────────────────────────────────────────────────────┤
│ Deployment         │ myapp::nginx-deployment                                                                     │
└────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────┘
? Are you sure to deploy 1 resource, 1 type on 1 provider? › (y/N)

Notice this time the action is not CREATE but UPDATE. gc fetched the live resources from the kubernetes-api-server, compared them with the target resources defined in the code, and has found out the deployment needs to be updated.

Now we can apply the change:

gc apply

The updated Nginx image should be up and running.

Let's double-check the state of the Nginx deployment, filtering by type and name

gc list -t Deployment --name nginx-deployment

Listing resources on 1 provider: k8s
✓ k8s
  ✓ Initialising
  ✓ Listing 6/6
┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ 1 Deployment from k8s                                                                                             │
├─────────────────────────┬──────────────────────────────────────────────────────────────────────────────────┬──────┤
│ Name                    │ Data                                                                             │ Our  │
├─────────────────────────┼──────────────────────────────────────────────────────────────────────────────────┼──────┤
│ myapp::nginx-deployment │ metadata:                                                                        │ Yes  │
│                         │   name: nginx-deployment                                                         │      │
│                         │   namespace: myapp                                                               │      │
│                         │   uid: 7c9bf366-cbf4-47d9-a7b7-e3da900b75dc                                      │      │
│                         │   resourceVersion: 7111                                                          │      │
│                         │   generation: 2                                                                  │      │
│                         │   creationTimestamp: 2021-04-28T19:51:37Z                                        │      │
│                         │   labels:                                                                        │      │
│                         │     app: nginx-label                                                             │      │
│                         │   annotations:                                                                   │      │
│                         │     CreatedByProvider: k8s                                                       │      │
│                         │     ManagedBy: GruCloud                                                          │      │
│                         │     Name: nginx-deployment                                                       │      │
│                         │     deployment.kubernetes.io/revision: 2                                         │      │
│                         │     projectName: @grucloud/example-k8s-tuto1                                     │      │
│                         │     stage: dev                                                                   │      │
│                         │ spec:                                                                            │      │
│                         │   replicas: 1                                                                    │      │
│                         │   selector:                                                                      │      │
│                         │     matchLabels:                                                                 │      │
│                         │       app: nginx-label                                                           │      │
│                         │   template:                                                                      │      │
│                         │     metadata:                                                                    │      │
│                         │       creationTimestamp: null                                                    │      │
│                         │       labels:                                                                    │      │
│                         │         app: nginx-label                                                         │      │
│                         │     spec:                                                                        │      │
│                         │       containers:                                                                │      │
│                         │         - name: nginx                                                            │      │
│                         │           image: nginx:1.20.0-alpine                                             │      │
│                         │           ports:                                                                 │      │
│                         │             - containerPort: 80                                                  │      │
│                         │               protocol: TCP                                                      │      │
│                         │           resources:                                                             │      │
│                         │           terminationMessagePath: /dev/termination-log                           │      │
│                         │           terminationMessagePolicy: File                                         │      │
│                         │           imagePullPolicy: IfNotPresent                                          │      │
│                         │       restartPolicy: Always                                                      │      │
│                         │       terminationGracePeriodSeconds: 30                                          │      │
│                         │       dnsPolicy: ClusterFirst                                                    │      │
│                         │       securityContext:                                                           │      │
│                         │       schedulerName: default-scheduler                                           │      │
│                         │   strategy:                                                                      │      │
│                         │     type: RollingUpdate                                                          │      │
│                         │     rollingUpdate:                                                               │      │
│                         │       maxUnavailable: 25%                                                        │      │
│                         │       maxSurge: 25%                                                              │      │
│                         │   revisionHistoryLimit: 10                                                       │      │
│                         │   progressDeadlineSeconds: 600                                                   │      │
│                         │ status:                                                                          │      │
│                         │   observedGeneration: 2                                                          │      │
│                         │   replicas: 1                                                                    │      │
│                         │   updatedReplicas: 1                                                             │      │
│                         │   readyReplicas: 1                                                               │      │
│                         │   availableReplicas: 1                                                           │      │
│                         │   conditions:                                                                    │      │
│                         │     - type: Available                                                            │      │
│                         │       status: True                                                               │      │
│                         │       lastUpdateTime: 2021-04-28T19:51:39Z                                       │      │
│                         │       lastTransitionTime: 2021-04-28T19:51:39Z                                   │      │
│                         │       reason: MinimumReplicasAvailable                                           │      │
│                         │       message: Deployment has minimum availability.                              │      │
│                         │     - type: Progressing                                                          │      │
│                         │       status: True                                                               │      │
│                         │       lastUpdateTime: 2021-04-28T20:03:08Z                                       │      │
│                         │       lastTransitionTime: 2021-04-28T19:51:37Z                                   │      │
│                         │       reason: NewReplicaSetAvailable                                             │      │
│                         │       message: ReplicaSet "nginx-deployment-675bd9f4f7" has successfully progre… │      │
│                         │                                                                                  │      │
└─────────────────────────┴──────────────────────────────────────────────────────────────────────────────────┴──────┘


List Summary:
Provider: k8s
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ k8s                                                                                                              │
├────────────────────┬─────────────────────────────────────────────────────────────────────────────────────────────┤
│ Deployment         │ myapp::nginx-deployment                                                                     │
└────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────┘
1 resource, 5 types, 1 provider
Command "gc list -t Deployment --name nginx-deployment" executed in 0s

Great, as expected, the new version has been updated.

Destroy

To destroy the resources allocated in the right order:

gc destroy

┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Destroy summary for provider k8s                                                                                 │
├────────────────────┬─────────────────────────────────────────────────────────────────────────────────────────────┤
│ Namespace          │ myapp                                                                                       │
├────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────┤
│ Service            │ myapp::nginx-service                                                                        │
├────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────┤
│ Deployment         │ myapp::nginx-deployment                                                                     │
└────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────┘
✔ Are you sure to destroy 3 resources, 3 types on 1 provider? … yes
Destroying resources on 1 provider: k8s
✓ k8s
  ✓ Initialising
  ✓ Destroying
    ✓ Namespace 1/1
    ✓ Service 1/1
    ✓ Deployment 1/1
  ✓ default::onDestroyed
3 resources destroyed, 3 types on 1 provider
Running OnDestroyedGlobal resources on 1 provider: k8s
Command "gc d" executed in 1m 17s

At this stage, all the Kubernetes resources should have been destroyed.
We could try to run gc destroy command again, nothing should be destroyed or deployed:

gc d

Find Deletable resources on 1 provider: k8s
✓ k8s
  ✓ Initialising
  ✓ Listing 7/7

No resources to destroy
Running OnDestroyedGlobal resources on 1 provider: k8s
Command "gc d" executed in 0s

As expected, the destroy command is idempotent.

Debugging

A benefit of using a general-purpose programming such as Javascript, is debugging. Thanks Visual Studio Code for providing such an easy way to debug Javascript applications.

This example contains a vs code file called launch.json, which defines various debug targets for gc apply, gc destroy and so on.

Conclusion

This tutorial described how to deploy, list, and destroy Kubernetes manifests from Javascript code.
In this case, a namespace, a service, and a deployment.

What's next? Let's see how to deploy a full stack application on minikube.

Ready to try Kubernetes on EKS, the Amazon Elastic Kubernetes Service? Have a look at the project full stack application on EKS.

Maybe you prefer using kops to set up your cluster? The tutorial Setup Kops on AWS with Grucloud explains how to automate the kops setup

Are you looking to install the cert manager, web ui dashboard, prometheus, and more? Browse the GruCloud K8s Modules and find out how to install and use these npm packages into your code.

Setup KOPS on AWS with GruCloud

Frederic — Mon, 26 Apr 2021 15:46:54 +0000

The aim of this tutorial is to automatically create and destroy the AWS resources required by kops, a tool to create a Kubernetes cluster.

The section 'setup your environment' from the official kops documentation will be automated with GruCloud

Below is the diagram generated from the target code, it illustrates the resources and their association with each other:

Regarding this DNS scenario, the case of a subdomain where a top-level hosted zone already exists is implemented.

TD;DR

Get this example code and install the dependencies.
Edit the configuration file and set the domain name, the subdomain name, the region, and the zone.
gc apply

All the AWS resources required by kops should have been created. The environment file kops.env containing the necessary information should have been generated too.

You are now ready to create a cluster with kops,

Here are a few npm scripts wrapper: npm run kops:create, npm run kops:update and npm run kops:validate.

Steps

Here is a description of the steps that are automated:

IAM

create a kops group, attach 5 IAM policies.
create a kops user, attach the user to the kops group.
create access and secret key for the kops user.

Route53

create a hosted zone for a subdomain.
create a DNS record of type NS in the top-level hosted zone with the DNS servers as values from the subdomain hosted zone.

S3

create an S3 bucket with encryption and versioning.

kops.env file

create a file containing the environment variable for kops

You will be free from performing all these commands manually. The same applies to the destruction of all these resources.

Requirements

Access to the AWS console
AWS CLI configured
A domain name registered on Route53.
Node.js
GruCloud CLI

Install

Clone this code, change to the kops folder, install the npm dependencies:

git clone https://github.com/grucloud/grucloud
cd grucloud/examples/aws/kops
npm install

Configuration

Edit config.js and set the domainName, the subDomainName, the zone and the region:

Double check your configuration with gc info:

  - provider:
      name: aws
      type: aws
    stage: dev
    config:
      projectName: @grucloud/create-aws-kops
      kops:
        domainName: grucloud.org
        subDomainName: kops.example.grucloud.org
        groupName: kops
        userName: kops
      stage: dev
      zone: us-east-1a
      accountId: 4444454555555
      region: us-east-1

The domain name must be registered with Route53 for the current AWS user.
Let's also verify that a top level hosted zone already exists.
You could use the gc list command with the Route53Domain and HostedZone type filter:

gc list  -t Route53Domain -t HostedZone

[...Truncated]
┌─────────────────────────────────────────────────────────────────────────────────────────────┐
│ aws                                                                                         │
├────────────────────┬────────────────────────────────────────────────────────────────────────┤
│ Route53Domain      │ grucloud.org                                                           │
├────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ HostedZone         │ grucloud.org.                                                          │
└────────────────────┴────────────────────────────────────────────────────────────────────────┘

iac.js

For your information, the architecture is described in iac.js.
In this use, the cloud provider is AWS, so we'll use the GruCloud AWS Provider to create the resources.

Target Graph

Another way to explore the iac.js is to generate a diagram of the target resources:

gc graph

Deploying

Ready to deploy the user, group, s3 bucket, route53 hosted zone and record ?

gc apply

The AWS resources should have been deployed.
Let's find out our live resources as well as a diagram showing the dependencies between these resources:

gc list --graph --our

┌─────────────────────────────────────────────────────────────────────────────────────────────┐
│ aws                                                                                         │
├────────────────────┬────────────────────────────────────────────────────────────────────────┤
│ IamPolicy          │ AmazonEC2FullAccess                                                    │
│                    │ AmazonRoute53FullAccess                                                │
│                    │ AmazonS3FullAccess                                                     │
│                    │ IAMFullAccess                                                          │
│                    │ AmazonVPCFullAccess                                                    │
├────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ IamGroup           │ kops                                                                   │
├────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ S3Bucket           │ kops.example.grucloud.org                                              │
├────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ IamUser            │ kops                                                                   │
├────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ HostedZone         │ kops.example.grucloud.org.                                             │
│                    │ grucloud.org.                                                          │
├────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ Route53Record      │ kops.example.grucloud.org-ns                                           │
└────────────────────┴────────────────────────────────────────────────────────────────────────┘
11 resources, 15 types, 1 provider

Envirornment variables

At the end of the deployment, the environment file kops.env is generated with the variables required by kops:

# kops.env
export AWS_ACCESS_KEY_ID=XXXXXXNBM2ZQEPXXXXX
export AWS_SECRET_ACCESS_KEY=XXXXXiXmSB3aZTK/AxOOvSPcGby3XXXXXX
export NAME=kops.example.grucloud.org
export KOPS_STATE_STORE=s3://kops.example.grucloud.org
export REGION=eu-west-2
export ZONE=eu-west-2a

Source with variables with:

source kops.env

When the deploment is destroyed with gc destroy, kops.env is removed.

The file hook.js is the place where this logic is implemented.

NPM kops scripts

The following npm scripts manage the kops commands, the environment variables are sourced from kops.env.

npm run kops:create
npm run kops:update
npm run kops:validate

List Resources

Let's fetch all the live resources, we'll see that kops creates many resources such as autoscaling groups, ec2 instances, subnets, vpc, internet gateway, volumes, key pair and so on:

gc list --graph --all --default-exclude --types-exclude Certificate --types-exclude Route53Domain --types-exclude NetworkInterface

You could inspect and generate a diagram of any existing AWS infrastruture for the most used resources

Destroy

To destroy the resources created by GruCloud, use the destroy command.

Ensure the cluster is destroyed before.

npm run kops:destroy
gc destroy

Alternatively, gc could also destroy all the resources created by kops, use the all flag:

gc destroy --all

Further Step

Congratulations, you know how to create and destroy a Kubernetes cluster with kops.
What about a load balancer, DNS records, SSL certificates ? Grucloud provides some ready made modules distributed with npm, the node package manager.

Have a look at:

@grucloud/module-aws-certificate: Create a certificate and a Route53 record for validation.
@grucloud/module-aws-load-balancer: Manage a load balancer, target groups, listeners and rules. A leaner alternative the AWS Load Balancer Controller which runs on the cluster.
@grucloud/module-aws-vpc: Contains the base resources required to create a Kubernetes cluster.

On the Kubernetes side, be aware of the GruCloud Kubernetes Provider. In a nutshell, instead of writing YAML manifest, Javascript is used instead to define the manifests, no more templating engine, enjoy a real programming language instead.

Would you like to deploy a full stack application on EKS ? Choose the flavour depending on who is reponsible to create the load balancer, target groups, listener and rules:

Load balancer resources created inside the cluster with the AWS Load Balancer Controller: eks-lbc.
A leaner solution where the load balancer resources are created by GruCLoud outside the cluster: eks-lean.

DEV Community: Frederic

Full Stack App Automated deployment on AWS EKS

Full Stack Application on AWS EKS

High-level description

Modules for AWS resources

Modules for K8s resources

Resources

Mindmap

Diagram

Workflow

Requirements

AWS CLI

GruCloud CLI:

Getting the source code

Configuration

Amazon EKS

K8s

GruCloud workflow

Listing

Deploying

Updating

Destroying

Links

Deploy an HTTPS Static Website on GCP with GruCloud

Hosting a secure static website on GCP, domain managed by AWS Route53

Resources

Hooks

Requirements

GruCloud CLI

GCP

AWS

Route53 Domain

Route53 Hosted Zone

Getting the source code

Config

Initialise

Add the service account as the domain owner

GruCloud CLI Workflow

Deploy

List the resources.

Update

Destroy

Links

Deploy, update, and destroy AWS EC2 Instance automatically

Requirements

AWS Account

AWS CLI

Access and Secret Key

Configure AWS CLI

Getting the GruCloud CLI

Code Architecture

Create a new project

package.json

config.js

iac.js

GruCloud CLI

Info

Deploy

List

Update

Output

Destroy

Graph

Links

Kubernetes defined as Javascript Code with GruCloud

Introduction

Requirements

Getting GruCloud CLI: gc

Project Content

package.json

config.js

iac.js

hook.js

Workflow

Deploy

List

Post Deploy Hook

Update

Destroy

Debugging