DEV Community: Ryan Tiffany

Getting Started with IBM Cloud CLI Search

Ryan Tiffany — Thu, 27 Jan 2022 16:29:27 +0000

Overview

This guide will show you how to get started using the IBM Cloud CLI resource search function to locate resources on your account.

The examples are divided in to two sections:

Cloud = IBM Cloud Resources.
IaaS = Classic Infrastructure (SoftLayer) Resources.

If you do not have the IBM Cloud CLI installed you can refer to this doc for installation instructions. Alternately you can use IBM Cloud Shell which has all of the tools we will need out of the box.

Search Cloud Resources (Cloud)

$ ibmcloud resource search 'name:devcluster'

Search by resource name and return CRN (Cloud)

$ ibmcloud resource search 'name:devcluster' --output json | jq -r '.items[].crn'

Search by resource tag (Cloud)

$ ibmcloud resource search 'tags:ryantiffany' --output json

Return resource names (Cloud)

$ ibmcloud resource search 'tags:ryantiffany' --output json | jq -r '.items[].name'

Return resource CRNs (Cloud)

$ ibmcloud resource search 'tags:ryantiffany' --output json | jq -r  '.items[].crn'

Return resource types (Cloud)

$ ibmcloud resource search 'tags:ryantiffany' --output json | jq -r  '.items[].type'

Search classic infrastructure (IaaS)

$ ibmcloud resource search -p classic-infrastructure --output json

Search classic infrastructure by tag (IaaS)

$ ibmcloud resource search "tagReferences.tag.name:ryantiffany" -p classic-infrastructure --output json

Return resource types (IaaS)

$ ibmcloud resource search "tagReferences.tag.name:ryantiffany" -p classic-infrastructure --output json | jq -r '.items[].resourceType'

Search by tag and filter on virtual instances (IaaS)

$ ibmcloud resource search "tagReferences.tag.name:ryantiffany _objectType:SoftLayer_Virtual_Guest" -p classic-infrastructure --output json

Search IaaS Virtual instances by Tag and return FQDNs

$ ibmcloud resource search "tagReferences.tag.name:ryantiffany _objectType:SoftLayer_Virtual_Guest" -p classic-infrastructure --output json | jq -r '.items[].resource.fullyQualifiedDomainName'

Search IaaS Virtual instances by Tag and return instance ID's


shell
$ ibmcloud resource search "tagReferences.tag.name:<tag>

Deploy a Consul cluster to an IBM Cloud VPC using Terraform and Ansible

Ryan Tiffany — Fri, 04 Dec 2020 14:46:38 +0000

Lately I have been on a little bit of an Ansible + Terraform kick so I thought I would throw together a code example for deploying a Consul cluster in to an IBM Cloud VPC using these tools.

Consul is a service mesh control plane with baked in service discovery, configuration, and segmentation functionality. As more and more of our deployed applications and services are spread out between clouds, Consul allows us a secure communication layer regardless of where our infrastructure is hosted.

You can get $500 (USD) in credit towards VPC resources in IBM by adding the code VPC500 to your account.

Prerequisites

Tfswitch installed
Ansible installed
An IBM Cloud API Key

Use Terraform to Create Infrastructure

Terraform is an infrastructure as code tool that allows you to provision and manage a wide range of clouds, infrastructure, and services. Using Terraform allows us to create consistent, repeatable deployments.

Steps

Clone repository:

$ git clone https://github.com/cloud-design-dev/ibm-vpc-consul-terraform-ansible.git
$ cd ibm-vpc-consul-terraform-ansible

Copy terraform.tfvars.template to terraform.tfvars:

$ cp terraform.tfvars.template terraform.tfvars

Edit terraform.tfvars to match your environment.
Run tfswitch to point to the right Terraform version for this solution:

$ tfswitch

Deploy all resources:

$ terraform init
$ terraform plan -out default.tfplan 
$ terraform apply default.tfplan

If the plan completes successfully you should see something like the following output:

Apply complete! Resources: 27 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate

Outputs:

bastion_instance_ip = 10.242.0.36
bastion_public_ip = x.y.x.y
consul_instance_ip = [
  "10.242.0.4",
  "10.242.0.6",
  "10.242.0.5",
]
consul_names = [
  "default-041430-eu-gb-1-consul1",
  "default-041430-eu-gb-1-consul2",
  "default-041430-eu-gb-1-consul3",
]

Our Terraform deployment has also generated:

An Ansible inventory file
A variables file that will be used by the ansible playbook
A temporary ansible.cfg file for use with our playbook

After the plan completes we can move on to deploying Consul using Ansible.

Run Ansible Playbook to Create the Consul Cluster

Whereas Terraform is best suited for the deployment of infrastructure, when it comes to configuration management I prefer Ansible. In this example Ansible will be used to:

Update the base operating system
Add the consul public key to the server
Install the consul binary
Bootstrap a 3 node cluster using Ansible templates

$ cd ansible 
$ ansible-playbook -i inventory playbooks/consul-cluster.yml

If you would like a little more insight in to what Ansible is doing behind the scenes, add -vv to your ansible-playbook command:

$ ansible-playbook -vv -i inventory playbooks/consul-cluster.yml

Verify that the cluster is running

Since we bound the Consul agent to the main private IP of the VPC instances we first need to set the environmental variable for CONSUL_HTTP_ADDR. Take one of the consul instance IPs and run the following command:

$ ansible -m shell -b -a "CONSUL_HTTP_ADDR=\"http://CONSUL_INSTANCE_IP:8500\" consul members" CONSUL_INSTANCE_NAME -i inventory

Example output

ansible -m shell -b -a "CONSUL_HTTP_ADDR=\"http://10.241.0.36:8500\" consul members" dev-011534-us-east-1-consul1 -i inventory
dev-011534-us-east-1-consul1 | CHANGED | rc=0 >>

Node                          Address           Status  Type    Build  Protocol  DC       Segment
dev-011534-us-east-1-consul1  10.241.0.36:8301  alive   server  1.9.0  2         us-east  <all>
dev-011534-us-east-1-consul2  10.241.0.38:8301  alive   server  1.9.0  2         us-east  <all>
dev-011534-us-east-1-consul3  10.241.0.37:8301  alive   server  1.9.0  2         us-east  <all>

Asciinema Recording of a Test Run

Site to Site IPsec Tunnel to IBM Cloud VPC

Ryan Tiffany — Wed, 07 Oct 2020 14:27:15 +0000

Overview

This article will walk you through the process of connecting on on-prem IPsec tunnel to the IBM Cloud VPC VPN as a service offering. This will allow you to communicate from your local machine to the private IP addresses assigned to your VPC compute instances. In this guide you will walk through the following steps:

Provisioning an instance of VPC VPN as a Service
Adding a Peer connection to VPC VPN to connect to your local network
Install strongSwan on a local machine/VM
Configuring the local IPsec peer
Bringing up the local IPsec tunnel and pinging VPC resources

Provision and configure the VPC VPNaaS

We’ll start by deploying an instance of VPNaaS. From the main VPC landing page click on VPN Gateways on the left hand navigation bar:

Make sure you select the region where your VPC resides and then click Create VPN.

At the top of the screen give the VPN a name, select the VPC where you would like the VPN deployed, and then select the subnet to use with the VPN. Note: Only the resources in the same zone as the subnet you choose can connect through this VPN gateway.

With the subnet selected scroll down and make sure the New VPN Connection for VPC option is enabled. Give the new connection a name and provide the local and peer subnets along with the pre-shared key.

Note: If you need to generate a pre-shared key, launch Cloud Shell by clicking the terminal icon in the upper right of the IBM Cloud portal.

Once your Cloud Shell session starts run the following command to generate a 32 character pre-shared key

tr -dc "[:alpha:][:alnum:]" < /dev/urandom | head -c 32

In this example my VPC utilizes the 10.240.0.0/24 subnet and my local network uses 172.16 IPs. The Peer gateway address is the public IP on your local network. If you are unsure of what this is pull up a browser and head to IP Chicken.

With all the details added click Create VPN Gateway in the right hand navigation bar to deploy the VPN. We'll give the new VPN a few moments to deploy and then copy down the Peer address that we'll need for the local tunnel configuration.

Install strongSwan on local machine

In my example I have a local Ubuntu 18 VM that I will be using as the local IPsec peer. The first step is to install strongSwan.

$ apt-get update && apt-get install strongswan -y

With strongSwan installed we need to add IP forwarding to our kernel parameters:

$ sudo cat >> /etc/sysctl.conf << EOF
net.ipv4.ip_forward = 1 
net.ipv4.conf.all.accept_redirects = 0 
net.ipv4.conf.all.send_redirects = 0
EOF

$ sysctl -p /etc/sysctl.conf

Configure local ipsec peer

Next we'll update the /etc/ipsec.secrets file. The syntax for the file is:

LOCAL_PEER_IP VPC_VPN_PEER_IP : PSK "Pre-shared key"

For example if your local public IP was 192.168.20.2, the VPC VPN Peer address was 192.168.30.5, and the pre-shared key was XtemrMYFfmmMCpxgdCwSYoRBKdjQ1ndb the file would look like this:

192.168.20.2 192.168.30.5 : PSK "XtemrMYFfmmMCpxgdCwSYoRBKdjQ1ndb"

With the secrets file updated we'll now move on to updating the strongSwan configuration file:

# ipsec.conf - strongSwan IPsec configuration file
# basic configuration
config setup
    # strictcrlpolicy=yes
    # uniqueids = no
        charondebug="all"
        uniqueids=yes
        strictcrlpolicy=no

# connection to us-east-vpc
conn home-to-vpc
  authby=secret
  left=%defaultroute
  leftid=<Local Server Public IP>
  leftsubnet=<Local Internal Subnet range>
  right=<VPC VPN Endpoint IP>
  rightsubnet=<VPC Subnet range>,166.8.0.0/14,161.26.0.0/16
  ike=aes256-sha2_256-modp1024!
  esp=aes256-sha2_256!
  keyingtries=0
  ikelifetime=1h
  lifetime=8h
  dpddelay=30
  dpdtimeout=120
  dpdaction=restart
  auto=start

We add in the ranges 166.8.0.0/14 and 161.26.0.0/16 so that we can communicate with IBM Cloud services over their private IP address space.

With the ipsec configuration updated, add an iptables rule for post-routing. Again for my tunnel the VPC Subnet is 10.240.0.0/24 and my local internal subnet is 172.16.0.0/24 so adjust the following command to meet your needs.

$ iptables -t nat -A POSTROUTING -s 10.240.0.0/24 -d 172.16.0.0/24 -J MASQUERADE

Now restart the ipsec service and check the status of the tunnel:

$ sudo ipsec restart
Stopping strongSwan IPsec...
Starting strongSwan 5.6.2 IPsec [starter]...

$ sudo ipsec status
Security Associations (1 up, 0 connecting):
 home-to-vpc[1]: ESTABLISHED 16 seconds ago, 10.0.0.67[x.x.x.x]...52.y.y.y[52.y.y.y]
 home-to-vpc{1}:  INSTALLED, TUNNEL, reqid 1, ESP in UDP SPIs: c2225f47_i ccc3d826_o
 home-to-vpc{1}:   10.0.0.0/18 === 161.26.0.0/16 166.8.0.0/14 192.168.0.0/18

Test connectivity to VPC instance

In my VPC I have an instance with a private IP of 10.240.0.6:

$ ibmcloud is instances --output json | jq -r '.[] | select(.vpc.name=="us-south-vpc-rt") | .network_interfaces[].primary_ipv4_address'
10.240.0.6

$ ping -c2 -q 10.240.0.6
PING 10.240.0.6 (10.240.0.6) 56(84) bytes of data.

--- 10.240.0.6 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 40.836/41.497/42.159/0.692 ms

Build container images with Code Engine

Ryan Tiffany — Wed, 23 Sep 2020 15:31:56 +0000

Overview

This guide will show you how to use the experimental IBM Code Engine to build a container image from a Source control repository. Behind the scenes Code Engine will use Tekton pipelines to pull our source code from a Github repository and then create a container image using the supplied Docker file. After the build is complete Code Engine will push the new container image in to the IBM Cloud Container Registry.

NOTE: Code Engine is currently an experimental offering and all resources are deleted every 7 days.

Start a session in IBM Cloud Shell

In the IBM Cloud console, click the IBM Cloud Shell icon

A session starts and automatically logs you in through the IBM Cloud CLI.

Target Resource Group

In oder to interact with the Code Engine CLI we first need to target the Resource Group where the Code Engine project will be created:

$ ibmcloud target -g <Your Resource Group>

Create a Code Engine Project

The first step is to create a Code Engine project.

Keep in mind during the Beta phase you are limited to one Code Engine project per region. If you already have a Code Engine project you can simply target that project using the command ibmcloud ce project target -n <name of project>

We'll specify the --target option to automatically have the Code Engine cli target our new project:

$ ibmcloud ce project create -n <Project Name> --target

The project creation can take a few minutes, but when it completes you should see something like this:

$ ibmcloud ce project create -n ce-demo-project --target
Creating project 'ce-demo-project'...
Waiting for project 'ce-demo-project' to be in ready state...
Now selecting project 'ce-demo-project'.
OK

Create an API Key for Code Engine for Registry Access

As part of our Build process we are going be pulling a public Github repo but then pushing the built container in to IBM Cloud Container Registry. In order for our Code Engine project to be able to push to the registry we'll need to create an API key.

$ ibmcloud iam api-key-create <Project Name>-cliapikey -d "API Key for talking to Image registry from Code Engine" --file key_file

Create Code Engine Registry Secret

With our API key created, we will now add the IBM Cloud Container Registry to Code Engine. When using the IBM Container Registry the username will always be iamapikey. If you would like to push to an alternate IBM Container Registry endpoint update the --server flag accordingly.

$ export CR_API_KEY=`jq -r '.apikey' < key_file`

$ ibmcloud ce registry create --name ibmcr --server us.icr.io --username iamapikey --password "${CR_API_KEY}"

You can view all of your registry secrets by running the command: ibmcloud ce registry list.

$ ibmcloud ce registry list 
Project 'demo-rt' and all its contents will be automatically deleted 7 days from now.
Listing image registry access secrets...
OK

Name   Age  
ibmcr  11s

Create Code Engine Build Definition

With the Registry access added we can now create our Build definition. If you do not already have a Container namespace to push images to, please follow [this guide to create one.

$ ibmcloud ce build create --name go-app-example-build --source https://github.com/greyhoundforty/ce-build-example-go --strategy kaniko --size medium --image us.icr.io/<namespace>/go-app-example --registry-secret ibmcr

The breakdown of the command:

name: The name of the build definition
source: The Source control repository where our code lives
strategy: The build strategy we will use to build the image. In this case since our repository has a Dockerfile we will use kaniko
size: The size of the build defines how CPU cores, memory, and disk space are assigned to the build
image: The Container Registry namespace and image name to push our built container image
registry-secret: The Container Registry secret that allows Code Engine to push and pull images

Submit the Build Job

Before the Build run is submitted (the actual process of building the container image), we’ll want to target the underlying Kubernetes cluster that powers Code Engine. This will allow us to see the pods that are spun up for the build as well as track it’s progress. To have kubctl within Cloud Shell target our cluster run the following command: ibmcloud ce project target -n <Name of Project> -k

You should see output similar to this:

$ ibmcloud ce project target -n demo-rt -k 
Selecting project 'demo-rt'...
Added context for 'demo-rt' to the current kubeconfig file.
OK

With kubectl properly configured we can now launch the actual build of our container image using the buildrun command. We specify the build definition we created previously with the --build flag:

$ ibmcloud ce buildrun submit --name go-app-buildrun-v1 --build go-app-example-build

You can check the status of the build run using the command ibmcloud ce buildrun get --name <Name of build run>

$ ibmcloud ce buildrun get --name go-app-buildrun-v1
Project 'demo-rt' and all its contents will be automatically deleted 7 days from now.
Getting build run 'go-app-buildrun-v1'...
OK

Name:          go-app-buildrun-v1
ID:            d378e865-ecf4-4e26-932d-acb437eef0ef
Project Name:  demo-rt
Project ID:    ab07a001-9a77-4fd8-82e8-d4f8395ad735
Age:           36s
Created:       2020-09-23 09:13:33 -0500 CDT
Status:
  Reason:      Running
  Registered:  Unknown

Instances:
  Name                                Running  Status   Restarts  Age
  go-app-buildrun-v1-xpqfq-pod-hqchd  2/4      Running  0         34s

You can also check on the status of the Kubernetes pods by running kubectl get pods

$ kubectl get pods
NAME                                 READY   STATUS      RESTARTS   AGE
go-app-buildrun-v1-xpqfq-pod-hqchd   2/4     Running     0          41s

If the build completes successfully the pods will show Completed and the build run will show Succeeded

$ kubectl get pods
NAME                                 READY   STATUS      RESTARTS   AGE
go-app-buildrun-v1-xpqfq-pod-hqchd   0/4     Completed   0          4m10s

$ ibmcloud ce buildrun get --name go-app-buildrun-v1
Project 'demo-rt' and all its contents will be automatically deleted 7 days from now.
Getting build run 'go-app-buildrun-v1'...
OK

Name:          go-app-buildrun-v1
ID:            d378e865-ecf4-4e26-932d-acb437eef0ef
Project Name:  demo-rt
Project ID:    ab07a001-9a77-4fd8-82e8-d4f8395ad735
Age:           4m26s
Created:       2020-09-23 09:13:33 -0500 CDT
Status:
  Reason:      Succeeded
  Registered:  True

Instances:
  Name                                Running  Status     Restarts  Age
  go-app-buildrun-v1-xpqfq-pod-hqchd  0/4      Succeeded  0         4m24s

Use IBM Cloud Code Engine to Sync Object Storage Buckets Between Accounts

Ryan Tiffany — Thu, 20 Aug 2020 18:44:23 +0000

Overview

In this guide I will show you how to sync ICOS bucket objects between accounts using Code Engine. Code Engine provides a platform to unify the deployment of all of your container-based applications on a Kubernetes-based infrastructure. The Code Engine experience is designed so that you can focus on writing code without the need for you to learn, or even know about, Kubernetes.

Code Engine is currently an experimental offering and all resources are deleted every 7 days.

Steps

Preparing Accounts
Source Account
- Create Service ID
- Create Reader access policy for newly created service id
- Generate HMAC credentials tied to our service ID
Destination Account
- Create Service ID
- Create Reader access policy for newly created service id
- Generate HMAC credentials tied to our service ID
Create Code Engine Project via Cloud Shell
Create Code Engine Secrets
Create Code Engine Project Job definition with environmental variables
Submit Code Engine Job

Preparing Accounts

We will be using Cloud Shell to generate Service IDs and Object Storage credentials for both the source and destination accounts.

Source Account

We will create a service ID on the source account. A service ID identifies a service or application similar to how a user ID identifies a user. We can assign specific access policies to the service ID that restrict permissions for using specific services: in this case it gets read-only access to an IBM Cloud Object Storage bucket.

Create Service ID

$ ibmcloud iam service-id-create <name-of-your-service-id> --description "Service ID for read-only access to bucket" --output json

Create Reader access policy for newly created service id

Now we will limit the scope of this service ID to have read only access to our source Object Storage bucket.

$ ibmcloud iam service-policy-create <Service ID> --roles Reader --service-name cloud-object-storage --service-instance <Service Instance GUID> --resource-type bucket --resource <bucket-name>

Service Instance GUID - This is the GUID of the Cloud Object Storage instance. You can retrieve this with the command: ibmcloud resource service-instance <name of icos instance>

Generate HMAC credentials tied to our service ID

In order for the Minio client to talk to each Object Storage instance it will need HMAC credentials (Access Key and Secret Key in S3 parlance).

$ ibmcloud resource service-key-create source-icos-service-creds Reader --instance-id <Service Instance GUID> --service-id <Service ID> --parameters '{"HMAC":true}'

Save the access_key_id and secret_access_key as we will be using these in our Code Engine project.

Destination Account

We will create a service ID on the destination account. A service ID identifies a service or application similar to how a user ID identifies a user. We can assign specific access policies to the service ID that restrict permissions for using specific services: in this case it gets write access to an IBM Cloud Object Storage bucket.

Create Service ID

$ ibmcloud  iam service-id-create <name-of-your-service-id> --description "Service ID for write access to bucket" --output json

Create Reader access policy for newly created service id

Now we will limit the scope of this service ID to have read only access to our source Object Storage bucket.

$ ibmcloud iam service-policy-create <Service ID> --roles Writer --service-name cloud-object-storage --service-instance <Service Instance GUID> --resource-type bucket --resource <bucket-name>

Service Instance GUID - This is the GUID of the Cloud Object Storage instance. You can retrieve this with the command: ibmcloud resource service-instance <name of icos instance>

Generate HMAC credentials tied to our service ID

We'll follow the same procedure as last time to generate the HMAC credentials, but this time on the destination account.

$ ibmcloud resource service-key-create destination-icos-service-creds Writer --instance-id <Service Instance GUID> --service-id <Service ID> --parameters '{"HMAC":true}'

Save the access_key_id and secret_access_key as we will be using these in with our Code Engine project.

Create Code Engine Project via Cloud Shell

In order to create our Code Engine project we need to make sure that our cloud shell session is targeting the correct resource group. You can do this by using the target -g option with the IBM Cloud CLI.

$ ibmcloud target -g <Resource Group>

With the correct Resource Group set, we can now create our Code Engine project. We add the --target flag to ensure that future Code Engine commands are targeting the correct project.

$ ibmcloud ce project create -n <project_name> --target

Create Code Engine Secrets

In order for our minio powered container to sync objects between the accounts it needs access to the Access and Secret keys we created earlier. We will use the secret create option to store all of values in a single secret that we can then reference in our job definition.

SOURCE_ACCESS_KEY: Access Key generated on Source account
SOURCE_SECRET_KEY: Secret Key generated on Source account
SOURCE_REGION: Cloud Object Storage endpoint for the Source bucket
SOURCE_BUCKET: Name of bucket on Source account
DESTINATION_ACCESS_KEY: Access Key generated on Destination account
DESTINATION_SECRET_KEY: Secret Key generated on Destination account
DESTINATION_REGION: Cloud Object Storage endpoint for the Destination bucket
DESTINATION_BUCKET: Name of bucket on Destination account

$ ibmcloud ce secret create --name ce-sync-secret --from-literal SOURCE_ACCESS_KEY=VALUE --from-literal SOURCE_SECRET_KEY=VALUE --from-literal SOURCE_REGION=VALUE --from-literal SOURCE_BUCKET=VALUE --from-literal DESTINATION_ACCESS_KEY=VALUE --from-literal DESTINATION_SECRET_KEY=VALUE --from-literal DESTINATION_REGION=VALUE --from-literal DESTINATION_BUCKET=VALUE

Create Code Engine Project Job definition with environmental variables

Now that our project has been created we need to create our Job definition. In Code Engine terms a job is a stand-alone executable for batch jobs. Unlike applications, which react to incoming HTTP requests, jobs are meant to be used for running container images that contain an executable that is designed to run one time and then exit.

$ ibmcloud ce jobdef create --name JOBDEF_NAME --image IMAGE_REF --env-from-secret SECRET_NAME

You can view the definition of the job using the command ibmcloud ce jobdef get -n <name of job definition>

$ ibmcloud ce jobdef get -n ce-mc-sync-jobdef
Project 'ce-minio-sync' and all its contents will be automatically deleted 7 days from now.
Getting job definition 'ce-mc-sync-jobdef'...
Name:        ce-mc-sync-jobdef  
Project ID:  1d7514d2-ce89  
Metadata:    
  Creation Timestamp:  2020-08-17 14:51:00 +0000 UTC  
  Generation:          1  
  Resource Version:    223595401  
  Self Link:           /apis/codeengine.cloud.ibm.com/v1alpha1/namespaces/1d7514d2-ce89/jobdefinitions/ce-mc-sync-jobdef  
  UID:                 0fb11f71-a912-44f9-88e3-1d1612f8e8ab  
Spec:        
  Containers:  
    Image:  greyhoundforty/icos-ce-sync:1  
    Name:   ce-mc-sync-jobdef  
    Commands:  
    Arguments:  
    Env:    
      Name:  SOURCE_BUCKET  
      Value From Secret Key Ref:  
        Key:   SOURCE_BUCKET  
        Name:  ce-sync-secret  
    Env:    
      Name:  SOURCE_REGION  
      Value From Secret Key Ref:  
        Key:   SOURCE_REGION  
        Name:  ce-sync-secret  
    Env:    
      Name:  SOURCE_SECRET_KEY  
      Value From Secret Key Ref:  
        Key:   SOURCE_SECRET_KEY  
        Name:  ce-sync-secret  
    Env:    
      Name:  DESTINATION_ACCESS_KEY  
      Value From Secret Key Ref:  
        Key:   DESTINATION_ACCESS_KEY  
        Name:  ce-sync-secret  
    Env:    
      Name:  DESTINATION_BUCKET  
      Value From Secret Key Ref:  
        Key:   DESTINATION_BUCKET  
        Name:  ce-sync-secret  
    Env:    
      Name:  DESTINATION_REGION  
      Value From Secret Key Ref:  
        Key:   DESTINATION_REGION  
        Name:  ce-sync-secret  
    Env:    
      Name:  DESTINATION_SECRET_KEY  
      Value From Secret Key Ref:  
        Key:   DESTINATION_SECRET_KEY  
        Name:  ce-sync-secret  
    Env:    
      Name:  SOURCE_ACCESS_KEY  
      Value From Secret Key Ref:  
        Key:   SOURCE_ACCESS_KEY  
        Name:  ce-sync-secret  
    Resource Requests:  
      Cpu:     1  
      Memory:  128Mi  
OK

Submit Code Engine Job

It is now time to submit our Job to Code Engine. The maximum time a job can run is 10 hours, but in most cases ICOS syncing takes significantly less time to complete.

$ ibmcloud ce job run --name <name of job> --jobdef <name of job definition>

In my testing I am only syncing a few times so by the time I check the Kubernetes pods, the job has already completed. Looking at the logs I am able to verify that the contents have been synced between the Object Storage buckets.

ryan@cloudshell:~$ ibmcloud ce job run --name  ce-mc-sync-jobv1 --jobdef ce-mc-sync-jobdef
Project 'ce-minio-sync' and all its contents will be automatically deleted 7 days from now.
Creating job 'ce-mc-sync-jobv1'...
OK

ryan@cloudshell:~$ ibmcloud ce project target -n ce-minio-sync --kubecfg
Targeting project 'ce-minio-sync'...
Added context for 'ce-minio-sync' to the current kubeconfig file.
OK
Now targeting environment 'ce-minio-sync'.

ryan@cloudshell:~$ kubectl get pods 
NAME                   READY   STATUS      RESTARTS   AGE
ce-mc-sync-jobv1-0-0   0/1     Completed   0          53s

ryan@cloudshell:~$ ibmcloud ce job list
Project 'ce-minio-sync' and all its contents will be automatically deleted 7 days from now.
Listing jobs...
Name               Age   
ce-mc-sync-jobv1   2m13s   
OK
Command 'job list' performed successfully

ryan@cloudshell:~$ ibmcloud ce job logs -n ce-mc-sync-jobv1
Project 'ce-minio-sync' and all its contents will be automatically deleted 7 days from now.
Logging job 'ce-mc-sync-jobv1' on pod '0'...
Added `source_acct` successfully.
Added `destination_acct` successfully.
`source_acct/wandering-thunder-68-source/as-policy.png` -> `destination_acct/sparkling-sky-47-destination/as-policy.png`
`source_acct/wandering-thunder-68-source/create-source-service-policy.png` -> `destination_acct/sparkling-sky-47-destination/create-source-service-policy.png`
`source_acct/wandering-thunder-68-source/add-backup-repository.png` -> `destination_acct/sparkling-sky-47-destination/add-backup-repository.png`
`source_acct/wandering-thunder-68-source/direct-link-standard.png` -> `destination_acct/sparkling-sky-47-destination/direct-link-standard.png`
`source_acct/wandering-thunder-68-source/create-workspace.png` -> `destination_acct/sparkling-sky-47-destination/create-workspace.png`
`source_acct/wandering-thunder-68-source/direct-link-byoip.png` -> `destination_acct/sparkling-sky-47-destination/direct-link-byoip.png`
`source_acct/wandering-thunder-68-source/add-scale-out.png` -> `destination_acct/sparkling-sky-47-destination/add-scale-out.png`
`source_acct/wandering-thunder-68-source/filter-vpc.png` -> `destination_acct/sparkling-sky-47-destination/filter-vpc.png`
`source_acct/wandering-thunder-68-source/k8s-storage.png` -> `destination_acct/sparkling-sky-47-destination/k8s-storage.png`
`source_acct/wandering-thunder-68-source/Picture1.png` -> `destination_acct/sparkling-sky-47-destination/Picture1.png`
`source_acct/wandering-thunder-68-source/iks-storage-Page-1.png` -> `destination_acct/sparkling-sky-47-destination/iks-storage-Page-1.png`
`source_acct/wandering-thunder-68-source/dl-copy.png` -> `destination_acct/sparkling-sky-47-destination/dl-copy.png`
`source_acct/wandering-thunder-68-source/rt-us-east.gv.png` -> `destination_acct/sparkling-sky-47-destination/rt-us-east.gv.png`
`source_acct/wandering-thunder-68-source/ns-secrets-cm.png` -> `destination_acct/sparkling-sky-47-destination/ns-secrets-cm.png`
Total: 0 B, Transferred: 2.30 MiB, Speed: 1.66 MiB/s

OK
Command 'job logs' performed successfully

If you need to sync contents from the source bucket to the destination bucket again, simply run another job (with a new name) and Code Engine will take care of it for you.

Quick Project Templates in shell

Ryan Tiffany — Tue, 14 Jul 2020 15:55:45 +0000

As someone that has to jump between a host of Terraform deployment environments having a standard template of files for each environment is a real time saver. Today I want to highlight a shell utility I have been using to help streamline this process: prm

prm allows you to use CRUD methodology for projects within your shell. Upon activation, each projects runs its associated start up script and upon deactivation, it can run scripts to clean up the environment.

These start and stop scripts can be used for changing directories, setting environment variables, cleanup, etc.

Example

One of the IBM Cloud Services I interact with the most is Schematics. Schematics is a hosted Terraform environment for defining and deploying Infrastructure as Code both within and outside of IBM Cloud.

Everytime I start a new Schematics project I need to do the following:

Create a Directory structure in a specified location
Copy a set of example Terraform files
Initialize a new Git repository
Open Visual Studio Code in the newly created project directory

prm allows me to do that with the following start up script:

# Command line arguments can be used, $3 would be the first argument after your project name.
PROJECT_NAME="$3"
TEMPLATE_DIR="${HOME}/Sync/Code/Terraform/templates/prm/schematics"
TF_DIR="${HOME}/Sync/Code/Terraform"

dt=$(date "+%Y.%m.%d")

if [[ ! -d "${TF_DIR}/${PROJECT_NAME}" ]]; then
    mkdir "${TF_DIR}/${PROJECT_NAME}" && cd $_
    export TF_PROJECT_DIR="${TF_DIR}/${PROJECT_NAME}"
else
    echo "Directory already exists, creating new one with date appended"
    mkdir "${TF_DIR}/${PROJECT_NAME}-${dt}" && cd $_
    export TF_PROJECT_DIR="${TF_DIR}/${PROJECT_NAME}-${dt}"
fi

cp "${TEMPLATE_DIR}"/Terraform.gitignore "${TF_PROJECT_DIR}"/.gitignore
cp "${TEMPLATE_DIR}/main.tf" "${TF_PROJECT_DIR}/main.tf"
cp "${TEMPLATE_DIR}/variables.tf" "${TF_PROJECT_DIR}/variables.tf"
cp "${TEMPLATE_DIR}/providers.tf" "${TF_PROJECT_DIR}/providers.tf"
cp "${TEMPLATE_DIR}/install.yml" "${TF_PROJECT_DIR}/install.yml"
cp "${TEMPLATE_DIR}/installer.sh" "${TF_PROJECT_DIR}/installer.sh"
cp "${TEMPLATE_DIR}/data.tf" "${TF_PROJECT_DIR}/data.tf"

git init 
code .

Invoking PRM

To start a prm Schematics project I simply run prm start schematics <name of project>.

tycho ◲ prm start schematics testfordevto
Starting project schematics
Initialized empty Git repository in /Users/ryan/Sync/Code/Terraform/testfordevto/.git/

~/Sync/Code/Terraform/testfordevto master*
tycho ◲ ls-l 
-rw-r--r--  1 ryan  staff    68 Jul 14 10:40 data.tf
-rw-r--r--  1 ryan  staff  2440 Jul 14 10:40 install.yml
-rw-r--r--  1 ryan  staff  2259 Jul 14 10:40 installer.sh
-rw-r--r--  1 ryan  staff   599 Jul 14 10:40 main.tf
-rw-r--r--  1 ryan  staff   121 Jul 14 10:40 providers.tf
-rw-r--r--  1 ryan  staff  1136 Jul 14 10:40 variables.tf

When running prm stop schematics prm invokes the following actions:

Checks if latest code changes have been pushed to Source control and if not prompts the user to do so.
Clears out any local *.tfplan or *.tfvars files.
Prompts if any new TODO items need to be added to the README file.
Changes directory back to $HOME

Deploying IBM Cloud infrastructure using Terraform and Gitlab

Ryan Tiffany — Fri, 29 May 2020 17:47:27 +0000

Today I will be walking you through how to set up Environmental Variables and a .gitlab-ci.yml file to deploy IBM Cloud resources using Terraform and the Gitlab CI/CD.

Setting CI/CD Variables in Gitlab

All of my automated IBM Cloud Terraform projects land under the same Gitlab group. There are many reasons to use Groups in Gitlab but for me it is mainly so that I don't have to set per-project Environmental Variables.

Step 1: From the top navigation bar click on Groups and select Your Groups.

Step 2: Go to your Gitlab Group page and from the left hand navigation click Settings > CI/CD.

Step 3 Click Expand in the Variables section

The variables you will want to set are:

IC_API_KEY = Your IBM Cloud API Key
SL_API_KEY = Your IBM Cloud IaaS (SoftLayer) API Key
SL_USERNAME = Your IBM Cloud IaaS (SoftLayer) username

Check the Masked option for the variables. Setting the masked variable option means that the value of the variables will be hidden in job logs during the CI/CD runs. When you're variables are set click Save.

A Note About Remote States: It is highly recommended to use a remote state for your Terraform deployments. With remote state, Terraform writes the state data to a remote data store, which can then be shared between all members of a team. If you will be using a remote state for your Terraform backend make sure you set the appropriate environmental variables for the backend provider. For instance if you are using the Consul backend provider you would want to set the CONSUL_TOKEN and CONSUL_HTTP_ADDR environmental variables.

Test Gitlab Automation

In order to test our Gitlab automation let's deploy a single Ubuntu 18 Virtual Instance. The first step is to create new Project in Github. When creating the project click the Import project tab and click on Repo by URL. Under the Git Repository URL section enter https://git.cloud-design.dev/ryan/ibm-tf-gitlab-example.git. Give your newly imported project a name, set it's visibility level and the click Create Project.

After a few moments Gitlab will create the new project. Once the project has been created you can dive in to the code to tweak the example deployment. You will need to do the following at the very least:

Update the main.tf file with the name of your IaaS SSH Key in the data.ibm_compute_ssh_key resource.
You will need to rename the example.gitlab-ci.yml file to .gitlab-ci.yml.

With your changes complete go ahead and commit your code to start the Gitlab CI/CD Pipeline. You can watch the progress of the Pipeline by clicking on the CI/CD left hand navigation link and selecting Pipelines:

From there you can view the progress of the Pipeline.

Notes

Remote States
If your deployments will be using a remote state make sure to change terraform init to terraform init -backend-config="lock=true" in the before_script section.

Gitlab and SSL
If you're using Let's Encrypt generated certificates you may see issues with the certificate not being trusted. To get around this you can add the following to the .gitlab-ci.yml file.

variables:
  GIT_SSL_NO_VERIFY: "1"

Targetting Gitlab Runners
If you need to use specific Gitlab Runners for your deployments you will want to add a tag decleration. For instance is you are targetting runners with the docker tag you would want to add the following to all the Ci/CD stages:

tags: 
    - docker

Alternative

IBM Cloud has recently launched a hosted Terraform offering called Schematics. IBM Cloud Schematics supports all IBM Cloud resources that are provided by the IBM Cloud Provider plug-in for Terraform with the advantage that you don't have to install the Terraform CLI and the IBM Cloud Provider plug-in. You can find some good Schematics example templates here.

Migrating IBM Cloud Object Storage Data Between Accounts

Ryan Tiffany — Thu, 28 May 2020 15:08:10 +0000

Today I will be showing you how you can migrate the contents of one IBM Cloud Object Storage bucket to a different instance of ICOS (IBM Cloud Object Storage). We will be using the tool rclone in order to sync the contents between buckets. In this scenario the ICOS instances exist on the same account but the process will work between distinct IBM Accounts as well.

Pre-reqs

HMAC credentials generated for each instance of Cloud Object Storage. See this guide for generating ICOS credentials with HMAC.
rclone installed. See the official installation docs here.

Configuring rclone

Once you have rclone installed you will need to generate a configuration file that will define our 2 ICOS instances. You can do this by running the command rclone config:

$ rclone config
2020/01/16 09:39:33 NOTICE: Config file "/Users/ryan/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> icos-instance-1
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / 1Fichier
   \ "fichier"
 2 / Alias for an existing remote
   \ "alias"
 3 / Amazon Drive
   \ "amazon cloud drive"
 4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, etc)
   \ "s3"
 5 / Backblaze B2
   \ "b2"
...

Choose option 3 to get a list of S3 compatible offerings, then choose IBM COS S3

Storage> 4
** See help for s3 backend at: https://rclone.org/s3/ **

Choose your S3 provider.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Amazon Web Services (AWS) S3
   \ "AWS"
 2 / Alibaba Cloud Object Storage System (OSS) formerly Aliyun
   \ "Alibaba"
 3 / Ceph Object Storage
   \ "Ceph"
 4 / Digital Ocean Spaces
   \ "DigitalOcean"
 5 / Dreamhost DreamObjects
   \ "Dreamhost"
 6 / IBM COS S3
   \ "IBMCOS"
 7 / Minio Object Storage
   \ "Minio"
 8 / Netease Object Storage (NOS)
   \ "Netease"
 9 / Wasabi Object Storage
   \ "Wasabi"
10 / Any other S3 compatible provider
   \ "Other"

Add your HMAC Access Key and Secret Key

env_auth> 1
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").
access_key_id> xxxxxxxxxxxxxxxxxxxxx
AWS Secret Access Key (password)
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").
secret_access_key> xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Region to connect to.
Leave blank if you are using an S3 clone and you don't have a region.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Use this if unsure. Will use v4 signatures and an empty region.
   \ ""
 2 / Use this only if v4 signatures don't work, eg pre Jewel/v10 CEPH.
   \ "other-v2-signature"
region> 1

Next you will want to chose the IBM Cloud Object Storage endpoint to use and the storage tier for the bucket you will be using. In this instance I am targetting the US-Cross regional endpoint and a standard tier bucket:

Endpoint for IBM COS S3 API.
Specify if using an IBM COS On Premise.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / US Cross Region Endpoint
   \ "s3-api.us-geo.objectstorage.softlayer.net"
 2 / US Cross Region Dallas Endpoint
   \ "s3-api.dal.us-geo.objectstorage.softlayer.net"
 3 / US Cross Region Washington DC Endpoint
   \ "s3-api.wdc-us-geo.objectstorage.softlayer.net"
...
endpoint> 1

Location constraint - must match endpoint when using IBM Cloud Public.
For on-prem COS, do not make a selection from this list, hit enter
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / US Cross Region Standard
   \ "us-standard"
 2 / US Cross Region Vault
   \ "us-vault"
 3 / US Cross Region Cold
   \ "us-cold"
 4 / US Cross Region Flex
   \ "us-flex"
...

On the next prompt you will need to specify an ACL policy. I am choosing private:

Note that this ACL is applied when server side copying objects as S3
doesn't copy the ACL from the source but rather writes a fresh one.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Owner gets FULL_CONTROL. No one else has access rights (default). This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise COS
   \ "private"
 2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access. This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise IBM COS
   \ "public-read"
 3 / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. This acl is available on IBM Cloud (Infra), On-Premise IBM COS
   \ "public-read-write"
 4 / Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access. Not supported on Buckets. This acl is available on IBM Cloud (Infra) and On-Premise IBM COS
   \ "authenticated-read"
acl> 1

Skip the advanced config and rclone should present you with your new configuration details. Double check that everything is correct and then select n to add your second ICOS instance.

Edit advanced config? (y/n)
y) Yes
n) No
y/n> n
Remote config
--------------------
[icos-instance-1]
type = s3
provider = IBMCOS
env_auth = false
access_key_id = xxxxxx
secret_access_key = xxxxxxxxx
endpoint = s3-api.us-geo.objectstorage.softlayer.net
location_constraint = us-standard
acl = private
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name                 Type
====                 ====
icos-instance-1      s3

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q>

Follow the steps again to add your second ICOS instance and when you've verified that everything looks correct choose q to quit the configuration process.

Inspecting our ICOS buckets

With rclone configured we can now start the actual sync process between our buckets, but first I will start by listing the contents of our source and destination buckets:

Source

$ rclone ls icos-instance-1:source-bucket-on-instance-1
    45338 AddNFSAccess.png
    48559 AddingNFSAccess.png
    66750 ChooseGroup.png
     2550 CloudPakApplications.png
     4643 CloudPakAutomation.png
     4553 CloudPakData.png
     5123 CloudPakIntegration.png
     4612 CloudPakMultiCloud.png
    23755 CompletedAddingNFSAccess.png
   174525 CreateNetworkShare1.png
    69836 CreateNetworkShare2.png
    76863 CreateStoragePool.png
    50489 CreateStoragePool1.png
    56297 CreateStoragePool2.png
     2340 applications-icon.svg
     6979 automation-icon.svg
   120584 cloud-paks-leadspace.png
     9255 data-icon.svg

Destination

$ rclone ls icos-instance-2:destination-bucket-on-instance-2
$

Syncing Bucket Objects

In this example I am going to be syncing the contents of the bucket source-bucket-on-instance-1 from my first instance of ICOS to the bucket destination-bucket-on-instance-2 on my second instance of ICOS. The -P flag allows us to see the progress if the sync operation.

$ rclone sync -P icos-instance-1:source-bucket-on-instance-1 icos-instance-2:destination-bucket-on-instance-2
Transferred:      754.933k / 754.933 kBytes, 100%, 151.979 kBytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:           18 / 18, 100%
Elapsed time:        4.9

Now if we look at the destination-bucket-on-instance-2 bucket again we'll see our files have synced over:

$ rclone ls icos-instance-2:destination-bucket-on-instance-2
    45338 AddNFSAccess.png
    48559 AddingNFSAccess.png
    66750 ChooseGroup.png
     2550 CloudPakApplications.png
     4643 CloudPakAutomation.png
     4553 CloudPakData.png
     5123 CloudPakIntegration.png
     4612 CloudPakMultiCloud.png
    23755 CompletedAddingNFSAccess.png
   174525 CreateNetworkShare1.png
    69836 CreateNetworkShare2.png
    76863 CreateStoragePool.png
    50489 CreateStoragePool1.png
    56297 CreateStoragePool2.png
     2340 applications-icon.svg
     6979 automation-icon.svg
   120584 cloud-paks-leadspace.png
     9255 data-icon.svg

Taking it Further

Sync Options - The sync operation makes the source and destination identical, and modifies the destination only. Destination is updated to match source, including deleting files if necessary. If you need to modify this default behavior take a look these additional configuration options for the sync command.
Automated Sync - If you need to set an automatic sync between buckets you will need to use a scheduling took like Task Scheduler for Windows or crontab for Linux/macOS.
Supported rclone commands - The full list of rclone subcommands for interacting with Cloud Object Storage.

Time to cheat in the shell

Ryan Tiffany — Thu, 28 May 2020 15:00:41 +0000

Like most people that work in the cloud there are some commands that just flow from my fingers without even a moments hesitation, while others inevitably lead to the man pages or googling. Today we're going to look at a cool little utility for when you just need to grab the proper flags for a command or need a quick refresher on the proper syntax for your fingers to perform their magic. All hail the cheat command.

cheat allows you to create and view interactive cheatsheets on the command-line. It was designed to help remind nix system administrators of options for commands that they use frequently, but not frequently enough to remember.

Installation

cheat has no dependencies. To install it, download the executable from the releases page and place it on your PATH.

Adding Community Cheat Sheets

By default cheat itself does not come with any cheatsheets however there is a large number of community provided ones hosted on Github.

$ cd ~/.config/cheat/
$ git clone https://github.com/cheat/cheatsheets

Now you will need to edit the default ~/config/cheat/conf.yml file to ensure that the community cheatpath points to ~/.config/cheat/cheatsheets. If everything is set up correctly the community cheatpath entry should look like this:

  - name: community
    path: ~/.config/cheat/cheatsheets
    tags: [ community ]
    readonly: true

With the community added sheets in the correct location you can use the command cheat -l to view them:

$ cheat -l
title:        file:                                               tags:
7z            /Users/ryan/.config/cheat/cheatsheets/7z            community,compression
ab            /Users/ryan/.config/cheat/cheatsheets/ab            community
alias         /Users/ryan/.config/cheat/cheatsheets/alias         community
ansi          /Users/ryan/.config/cheat/cheatsheets/ansi          community
apk           /Users/ryan/.config/cheat/cheatsheets/apk           community,packaging
apparmor      /Users/ryan/.config/cheat/cheatsheets/apparmor      community
apt           /Users/ryan/.config/cheat/cheatsheets/apt           community,packaging
apt-cache     /Users/ryan/.config/cheat/cheatsheets/apt-cache     community,packaging
apt-get       /Users/ryan/.config/cheat/cheatsheets/apt-get       community,packaging
aptitude      /Users/ryan/.config/cheat/cheatsheets/aptitude      community,packaging
aria2c        /Users/ryan/.config/cheat/cheatsheets/aria2c        community
asciiart      /Users/ryan/.config/cheat/cheatsheets/asciiart      community
asterisk      /Users/ryan/.config/cheat/cheatsheets/asterisk      community
....

Diving in

One of the commands that I always struggle with is sed. I can never remember the syntax no matter how many times I've tried. Cheat to the rescue:

$ cheat sed
# To replace all occurrences of "day" with "night" and write to stdout:
sed 's/day/night/g' file.txt

# To replace all occurrences of "day" with "night" within file.txt:
sed -i 's/day/night/g' file.txt

# To replace all occurrences of "day" with "night" on stdin:
echo 'It is daytime' | sed 's/day/night/g'

# To remove leading spaces
sed -i -r 's/^\s+//g' file.txt

# To remove empty lines and print results to stdout:
sed '/^$/d' file.txt

# To replace newlines in multiple lines
sed ':a;N;$!ba;s/\n//g'  file.txt

Creating Custom CheatSheets

Create your own cheatsheets using the -e flag. This will open a new file with your default editor and place the new cheatsheet in the personal cheatsheet path. For instance at my job I often have to deal with authenticating against our IAM offering. No matter how many times I do this the command just don't stick for me so I decided to create a cheatsheet for that reason:

cheat -e ibmcloud-iam

Using the -e flag opens a new file in your default editor. The file is stored in the ~/.cheat directory as a regular text file.

$ cheat ibmcloud-iam
# Get IAM Token from API Key
curl -s -k -X POST --header "Content-Type: application/x-www-form-urlencoded" --header "Accept: application/json" --data-urlencode "grant_type=urn:ibm:params:oauth:grant-type:apikey" --data-urlencode "apikey=${IBMCLOUD_API_KEY}" "https://iam.cloud.ibm.com/identity/token"

Hot and Cold backups using IBM Bluemix Cloud Object Storage

Ryan Tiffany — Tue, 13 Jun 2017 15:02:47 +0000

Overview

In this walk through we will be looking at utilizing rsnapshot and s3cmd to have local "hot" backups, and "cold" backups in IBM Bluemix Cloud Object Storage (S3). The rsnapshot package will be used to generate the backups of the host system as well as remote linux systems if required. The s3cmd utility is used to push these backups to IBM Bluemix Cloud Object Storage (S3).

Prerequisites

One or more linux server with rsnapshot, rsync, and s3cmd installed.
- RHEL/CentOS: yum install s3cmd rsnapshot rsync (you may need to add the epel repository as outlined http://www.tecmint.com/how-to-enable-epel-repository-for-rhel-centos-6-5/)
- Ubuntu/Debian: apt-get install s3cmd rsnapshot rsync
SSH-Keys generated on your main backup server. In this guide that is referred to as 'Backupserver'
SSH port open on your servers firewall. Rsnapshot uses rsync, which in turn uses SSH to pull backups from the remote-hosts so you will want to ensure you have the proper port whitelisted.

Servers:

Backupserver - main server running rsnapshot. Local and remote backups are stored on File Storage mount and once a day pushed to COS
bck1 & bck2 - Two more servers we want to backup. Our Backupserver will pull the backups using rsync.

Generate and copy ssh-key to remote hosts

For a guide on generating your servers public ssh-key as well as how to copy it to your remote servers please see our KnowledgeLayer article Generating and using SSH-Keys for remote host authentication.

Setting up and Configuring rsnapshot

The rsnapshot utility will be used to backup our local system as well as remote-hosts to a single directory. This will allow us to then compress those backups and send them to Cloud Object Storage (S3) using the s3cmd utility.

We are including an example rsnapshot.conf file that you can use as well. The example file has the following defaults:

Backups are stored in the /backups/ directory. Rsnapshot can create the directory if it does not already exist.
The retention scheme is set to keep 6 alpha backups, 7 beta backups, and 4 gamma backups. We'll touch on the syntax a little but further down.
Rsnapshot will backup /home, /etc, and /usr/local. You will need to adjust this to fit your needs.

Example rsnapshot.conf

A Note About rsnapshot.conf - The rsnapshot configuration file is very picky when it comes to tabs vs spaces. Always use tabs when editing the file. If there is an issue running rsnapshot configtest will show you the offending line.

To use the example configuration file run the following commands:

$ mv /etc/rsnapshot.conf{,.bak}
$ wget -O /etc/rsnapshot.conf https://raw.githubusercontent.com/greyhoundforty/COSTooling/master/rsnapshot.conf

The syntax here breaks down to all local alphabackups will go to /backups/alpha.X/localhost and the alpha backup for the remote server bck1 will go to /backups/alpha.X/bck1.

root@backuptest:~# grep snapshot_root /etc/rsnapshot.conf
snapshot_root   /backups/

backup  /home/      localhost/
backup  /etc/       localhost/
backup  /usr/local/ localhost/
backup  root@10.176.18.15:/var/ bck1/

Test the rsnapshot configuration by running the command rsnapshot configtest. You can also do a dry-run backup that will show you what rsnapshot will actually do when running the backup job by passing the -t flag.

root@backuptest:~# rsnapshot configtest
Syntax OK

root@backuptest:~# rsnapshot -t alpha
echo 14407 > /var/run/rsnapshot.pid
/bin/rm -rf /backups/alpha.5/
mv /backups/alpha.4/ /backups/alpha.5/
mv /backups/alpha.3/ /backups/alpha.4/
mv /backups/alpha.2/ /backups/alpha.3/
mv /backups/alpha.1/ /backups/alpha.2/
/bin/cp -al /backups/alpha.0 /backups/alpha.1
/usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded \
    /home/ /backups/alpha.0/localhost/
/usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded /etc/ \
    /backups/alpha.0/localhost/
/usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded \
    /usr/local/ /backups/alpha.0/localhost/
/usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded \
    --rsh=/usr/bin/ssh root@10.176.18.15:/var/ /backups/alpha.0/bck1/
touch /backups/alpha.0/

Run a test backup

If the configtest and dry run don't return any errors proceed to run your first alpha backup job. Note: By default the rsnapshot command will not produce any output when running backup jobs. If you would like to see what it is doing in real time pass the -v flag.

root@backuptest:~# rsnapshot -v alpha
echo 25845 > /var/run/rsnapshot.pid
/bin/rm -rf /backups/alpha.5/
mv /backups/alpha.4/ /backups/alpha.5/
mv /backups/alpha.3/ /backups/alpha.4/
mv /backups/alpha.2/ /backups/alpha.3/
mv /backups/alpha.1/ /backups/alpha.2/
/bin/cp -al /backups/alpha.0 /backups/alpha.1
/usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded \
    /home/ /backups/alpha.0/localhost/
/usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded /etc/ \
    /backups/alpha.0/localhost/
/usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded \
    /usr/local/ /backups/alpha.0/localhost/
/usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded \
    --rsh=/usr/bin/ssh root@10.176.18.15:/var/ /backups/alpha.0/bck1/
touch /backups/alpha.0/
rm -f /var/run/rsnapshot.pid

root@backuptest:~# ls -l /backups/alpha.0
total 8
drwxr-xr-x 3 root root 4096 Feb 10 17:01 bck1
drwxr-xr-x 5 root root 4096 Feb  8 00:00 localhost

Add other servers and adjust your schedule

Now that you have tested rsnapshot go ahead and add additional servers to rsnapshot.conf and configure your backup frequency. The rsnapshot utility uses the terms alpha, beta, gamma, and delta but you can think of them as:

alpha = hourly backups
beta = daily backups 
gamma = weekly backups 
delta = monthly backups

By default rsnapshot ships with the following retention scheme.

retain  alpha   6
retain  beta    7
retain  gamma   4
#retain delta   3

This means that your server will keep 6 hourly backups, 7 daily backups, and 4 weekly backups. For example: When the alpha backup job runs for the 7th time the oldest backup is rotated out and deleted so that it only has 6 alpha backups. The rsnapshot utility also ships with a default cron.d file on Debian/Ubuntu. For RHEL/Centos the package does not ship with a default cron file. In either case you will want to run the following command

$ wget -O /etc/cron.d/rsnapshot https://raw.githubusercontent.com/greyhoundforty/COSTooling/master/rsnapshotcron

0 */4   * * *       root    /usr/bin/rsnapshot alpha
0 21    * * *       root    /usr/bin/rsnapshot beta
0  3    * * 1       root    /usr/bin/rsnapshot gamma
# 30 2  1 * *       root    /usr/bin/rsnapshot delta

By default the alpha job will run every 4 hours, the beta every day at 9pm, and so on. This is where you will want to customize to suit your needs. With rsnapshot taken care of, we will now move on to configuring s3cmd.

Configuring s3cmd

The s3cmd python script is an open-source utility that allows a *nix or osx box to talk to S3 compatible services. After the utility is installed all the customer has to do is download our example .s3cfg file and update it with their COS Access and Secret Key:

$ wget -O $HOME/.s3cfg https://raw.githubusercontent.com/greyhoundforty/COSTooling/master/s3cfg

The 5 lines that need to be updated are 2,30,31, and 55 (if using our example .s3cfg file). For lines 30 and 31 you will want to replace cos_endpoint with the Cloud Object Storage (S3) endpoint you are using. Once all the lines have been updated with the COS (S3) details from the Customer portal you can test the connection by issuing the command s3cmd ls which will list all the buckets on the account.

$ s3cmd ls                                                                                                                                                  
2017-02-03 14:52  s3://backuptest
2017-02-03 21:23  s3://largebackup
2017-02-07 20:49  s3://po9bmbnem531ehdreyfh-winbackup
2017-02-07 17:44  s3://winbackup

Creating our backup bucket

To create the bucket we will store our backups in we will use the s3cmd mb command.

A Few Notes About Buckets - Cloud Object Storage (S3) has a 100 bucket limit per account. Keep this in mind if you set up each backup to create its own bucket or do a per month bucket. Bucket names must be DNS-compliant. Names must be between 3 and 63 characters long, must be made of lowercase letters, numbers, and dashes, must be globally unique, and cannot appear to be an IP address. A common approach to ensuring uniqueness is to append a UUID or other distinctive suffix to bucket names.

$ s3cmd mb s3://coldbackups/
Bucket 's3://coldbackups/' created

Pushing backups to Cloud Object Storage (S3)

To manually push your backups to Cloud Object Storage (S3) I would recommend using tar to compress the backup directory with a date stamp for easier sorting should you need to pull the backups from Cloud Object Storage (S3) for restoration.

tar -czf $(date "+%F").backup.tar.gz /backups/
s3cmd put $(date "+%F").backup.tar.gz s3://coldbackups/

To automate this process you would likely want to a use a cron job to commpress the backups and send them to Cloud Object Storage (S3) at regular intervals.

A Note About Retention - Cloud Object Storage (S3) does not currently support a retention scheme. This means that whatever you push to COS (S3) will remain there until you delete it.

Restoring Files from Cloud Object Storage (S3)

To restore a file or directory from Cloud Object Storage (S3) you will need to use the get command to pull down the backup. Once the file or directory has been downloaded to your server you can use cp, rsync, or mv to restore the file. If the file was from a remote host backed up using rsnapshot you would use scp or rsync to move it back to the original host system.

To pull a single file or compressed backup

By default the file you download from Cloud Object Storage (S3) will be stored in the same directory you are currently in.

$ sc3md get s3://bucket/path/to/file

You can also specify the directory you would like the downloaded file stored in:

$ s3cmd get s3://bucket/path/to/file /local/path/

To pull a directory

In order to pull a directory as well as all sub-items, use the --recursive flag

$ s3cmd get s3://bucket/ /local/path/ --recursive

Automating the process

I have created a bash script that can be used to set up both rsnapshot and s3cmd. This script is offered as is and you are free to make any modifications you like. backup_script.sh

Deploying Hugo on Bluemix

Ryan Tiffany — Thu, 08 Jun 2017 19:48:03 +0000

In today's post we will be showing how to deploy a Hugo site to Bluemix using the Cloud Foundry command line and build tools. This guide assumes that you have Hugo installed already. If this is not the case please see the Hugo Documentation for installing and configuring Hugo.

You will want to pick a Hugo theme and run a build so that your static files are sent to the public folder. I am using the heather-hugo theme on my example site so the command is:

$ hugo -t heather-hugo

Once you have generated your static files you can move on to installing and configuring the Cloud Foundry cli to push your app to Bluemix.

Install and configure Bluemix Cloud Foundry cli

The installation will depend on your specific OS but you can use this page to get the command line interface installed. Once the install has completed you will need to login using the cf command:

$ cf login
API endpoint: https://api.ng.bluemix.net

Email> user@example.com

Password>
Authenticating...
OK

Targeted org dev_test

Select a space (or press enter to skip):
1. dev
2. testingground

Space> 1
Targeted space dev

API endpoint:   https://api.ng.bluemix.net (API version: 2.40.0)
User:           user@example.com
Org:            dev_test
Space:          dev

Create your manifest file

One of the most important requirements for a Cloud Foundry app is the manifest.yml file. This file defines metadata about your application. More information about manifest files can be found here: Deploying with Application Manifests.

Here is my hugo app manifest.yml file

applications:
- path: public/
  memory: 1024M
  instances: 1
  name: hugo
  host: hugo
  disk_quota: 1024M
  buildpack: https://github.com/cloudfoundry-incubator/staticfile-buildpack.git

Here is the breakdown of the file:

path: Specify what folder gets pushed. If not defined, defaults to the current directory.
memory: Specify how much memory your application needs.
instances: Specify the number of app instances that you want to start upon push.
name: Name of your application.
host: This defines the name for the subdomain (yourhost.myBluemix.net).
buildpack: Specify what kind of buildpack your application needs. More information here: Cloud Foundry stacks

Push application to Bluemix

Now that we have all our ducks in a row we can push our application to Bluemix using the Cloud Foundry cli:

$ cf push hugo
Using manifest file /Users/ryan/bluemix/cf-apps/hugo/manifest.yml

Updating app hugo in org user@example.com / space dev as user@example.com...
OK

Using route hugo.testingbig.blue
Uploading hugo...
Uploading app files from: /Users/ryan/bluemix/cf-apps/hugo/public
Uploading 78.9K, 23 files
Done uploading
OK

Stopping app hugo in org user@example.com / space tinylab as user@example.com...
OK

Starting app hugo in org user@example.com / space tinylab as user@example.com...
-----> Downloaded app package (40K)
-----> Downloaded app buildpack cache (4.0K)
Cloning into '/tmp/buildpacks/staticfile-buildpack'...
Submodule 'compile-extensions' (https://github.com/cloudfoundry/compile-extensions.git) registered for path 'compile-extensions'
Cloning into 'compile-extensions'...
Submodule path 'compile-extensions': checked out '26a578c06a62c763205833561fec1c5c6d34deb6'
-------> Buildpack version 1.3.1
Downloaded [https://pivotal-buildpacks.s3.amazonaws.com/concourse-binaries/nginx/nginx-1.9.10-linux-x64.tgz]
grep: Staticfile: No such file or directory
-----> Using root folder
-----> Copying project files into public/
-----> Setting up nginx
grep: Staticfile: No such file or directory
-----> Uploading droplet (2.6M)

0 of 1 instances running, 1 starting
1 of 1 instances running

App started
OK

App hugo was started using this command `sh boot.sh`

Showing health and status for app hugo in org user@example.com / space dev as user@example.com...
OK

requested state: started
instances: 1/1
usage: 1G x 1 instances
urls: hugo.mybluemix.net
last uploaded: Thu Feb 25 17:19:03 UTC 2016
stack: cflinuxfs2
buildpack: https://github.com/cloudfoundry-incubator/staticfile-buildpack.git

     state     since                    cpu    memory       disk         details
#0   running   2016-02-25 11:19:36 AM   0.0%   2.6M of 1G   5.6M of 1G

Create custom domain in Bluemix

For simple testing and proof of concept you can certainly keep using the mybluemix.net domain, but if you are wanting to use this as a full time site you can configure Bluemix to use a custom domain. The first step to using your custom domain is to create the domain in Bluemix and associate it with an organization. The syntax is cf create-domain ORG DOMAIN_NAME. In my case the ORG is tinylab and my domain is testingbig.blue:

$ cf create-domain [ORG] testingbig.blue
Creating domain testingbig.blue for org tinylab as user@example.com...
OK

In order to use your custom domain you have to map the domain to your Bluemix application using the map-route command:

$ cf map-route hugo testingbig.blue
Creating route testingbig.blue for org tinylab / space Production_US as user@example.com...
OK

Point your custom domain to Bluemix

My domain testingbig.blue is currently using the SoftLayer DNS service so I added a record to point my domain to the Bluemix DNS system using the awesome SoftLayer CLI. Disclaimer: I Work at SoftLayer

$ slcli dns record-add hugo.testingbig.blue @ A 75.126.81.68

Push to Bluemix to use the Custom domain

In order to see the site at the new custom domain we'll go ahead and push the app again.

$ cf push hugo