DEV Community: Scott McAllister

Building a web server: Containers

Scott McAllister — Fri, 07 Jun 2024 19:47:30 +0000

Welcome back to This Old Box! A series that covers the journey of building and running a web server out of an old 2014 Mac mini.

In this post we'll walk through how we containerized our first web application, configured our Kubernetes cluster with our new container, and exposed that cluster to the world.

The project we deployed is the most simple from an infrastructure standpoint. It's a static site that is mainly html, css, and a wee bit of client-side JavaScript. Logistically, the whole site fit into a single container.

Docker

There are several container runtimes in the industry today. We chose to use Docker as it seems to be the industry standard, especially on a Mac. Docker is also well documented, and thus should be the easiest to learn. Although, I am concerned because I've heard the runtime is a bit greedy with CPU and memory resources. I hope this Old Box can handle it.

We installed Docker Desktop in our last post. With that application running we built a docker image by writing the instructions in a Dockerfile.

This first website we containerized is a collection of static sites that my son created called AjMcWeb. The instructions for building the container for this site were relatively simple. All we really needed to do was copy all the html, javascript, css, and image files for the site into the container, run them on nginx, and expose port 3001 on the container. The Dockerfile to accomplish these steps looked like this.

FROM nginx
COPY . /usr/share/nginx/html
EXPOSE 3001

Using that Dockerfile, we ran the following docker build command inside the same folder to build a running image of that container.

docker build -t ajmcweb:0.1 .

This command built the Docker image. The -t argument told Docker to tag the image with the the name ajmcweb and gave it a version of 0.1. The trailing . told Docker to use the Dockerfile in the current directory.

Now that we had our site encapsulated inside a Docker container we needed to deploy it to the world. That's where Kubernetes came in.

Kubernetes

Docker Desktop includes a standalone Kubernetes server and client that we enabled in our last post. This enablement instantiated the images required to run the Kubernetes server as containers, and installed the kubectl utility--used to manage our Kubernetes cluster--on our machine.

ngrok for ingress

Before configuring our Kubernetes cluster, we needed to install the ngrok Ingress Controller so we could use ngrok to provide ingress to our application. Ingress is "an API object that manages external access to the services in a cluster."

ngrok is an ingress platform that helps provide access to a variety of things including Kubernetes clusters as well as devices and other networks. The reason why we chose ngrok for this particular project is because ngrok works behind a NAT.

Here's how we set up ngrok and its ingress controller.

Setting up ngrok

First, we signed up for a free ngrok and then upgraded to a paid plan so we could use our own domains for our sites--like www.ajmcweb.com for example.

In order to use ngrok we first needed to get our ngrok authtoken in our ngrok dashboard. Then, we used that to set an NGROK_AUTHTOKEN environment variable on our system.

Next we created an API Key and set that to an NGROK_API_KEY environment variable.

The last environment variable we set was NAMESPACE. This is going to be the Kubernetes namespace we'll use for our clusters on this box. In our case, we used macbox.

Both of these environment variables will be used when we install the ngrok ingress controller. But, before we leave ngrok, I'll mention how we set up our own domains inside of our ngrok account.

Setting up custom domains in ngrok

We followed the ngrok guide for setting up Custom Domains which can be boiled down to two steps. First, we went to the Domains section of the ngrok dashboard and clicked the New Domain button.

We typed in www.ajmcweb.com and clicked Continue. This gave us a Domain Record value that we copied and pasted into a CNAME record on our domain registrar.

Setting up ngrok ingress controller

To establish ingress, or provide user access, we wanted to incorporate the ngrok account that we configured previously. ngrok has an ingress controller for Kubernetes which we installed as a helm chart with the following steps:

We added the ngrok repo to our helm settings.

helm repo add ngrok https://ngrok.github.io/kubernetes-ingress-controller

Then, we installed the helm chart with the following command, referencing the environment variables we set earlier.

helm install ngrok-ingress-controller ngrok/kubernetes-ingress-controller \
  --namespace $NAMESPACE \
  --create-namespace \
  --set credentials.apiKey=$NGROK_API_KEY \
  --set credentials.authtoken=$NGROK_AUTHTOKEN

Configuring Kubernetese

Going back to our code we captured in a Docker container we were ready to define our application deployment in Kubernetes. We began with configuring a Service that is a pod running on http and port 80.

apiVersion: v1
kind: Service
metadata:
  name: ajmcweb
  namespace: macbox
spec:
  ports:
      - name: http
        port: 80
        targetPort: 80
  selector:
      app: ajmcweb

Then we defined a Deployment object to run our ajmcweb container we built earlier. In the code below you'll see that we're running version 0.3 of that container, which is denoted as ajmcweb:0.3.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ajmcweb
  namespace: macbox
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ajmcweb
  template:
    metadata:
      labels:
        app: ajmcweb
        version: v0.1
    spec:
      containers:
        - name: ajmcweb
          image: ajmcweb:0.3
          ports:
          - name: http
            containerPort: 80

And, finally, we configured our ingress object using the ngrok Kubernetes Operator that we installed earlier.

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ajmcweb
  namespace: macbox
spec:
  ingressClassName: ngrok
  rules:
  - host: www.ajmcweb.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: ajmcweb
            port:
              number: 80

Our ingress object had one rule which defined the host as www.ajmcweb.com and the backend would run the ajmcweb service on port 80.

We applied those Kubernetes configurations with a kubectl apply -f <filename>.yaml and www.ajmcweb.com came right up in the browser!

Success!

Conclusion

This felt great to be able to get our simple static website deployed on our server and have ngrok handle the network settings to make the site reachable.

Our next task will be to get a more complex application online. My son has an application he built to help track the time he spends practicing his saxophone. The app uses Node.js and MySQL for the backend and has a much more rich frontend. It will require some more effort both from a deployment standpoint and from a processing perspective. Stay tuned as we figure out how to get this application online and keep it running!

Building a web server: Installing the right software

Scott McAllister — Wed, 17 Jan 2024 07:42:21 +0000

Welcome back to This Old Box! A series where I chronicle the journey of building a web server out of an old 2014 Mac mini. My son and I both have web applications that are running on various platforms, and we want to learn more about how to set up the infrastructure to serve those applications.

In our previous posts we initialized our machine and set up a KVM switch to share hardware--keyboard, mouse, and monitor--with my newer Mac mini that is the current family machine.

Homebrew

In this post we're going to prepare the software to run our web server. As is the case when I set up most Macs, my first install is the Homebrew package manager. They have instructions on their site for how to install, and once you have Homebrew set up most of the rest of your software can be installed with a brew install <package_name> in your Terminal. With the exception of our next piece of software.

Docker Desktop

I've dabbled with containers and Kubernetes in the past, but have never run anything in a production environment. Since my container experience has been with Docker we're going to install Docker Desktop. It's the only option for installing Docker on a Mac. And, there's an option to also enable Kubernetes.

GitLab Setup

The code for my son's projects is in GitLab. Our next step was to generate SSH keys to securely work interact with his repositories.

To set the keys for your account you'll go to the User Settings / Edit Profile page. I found it by selecting my profile image in the left sidebar, and then clicking on my name at the top. This got me to User Settings. On that page you'll see the Edit Profile button on the right side of the header. Then, on the Profile page, you'll see the SSH Keys option on the left sidebar.

If you're new to SSH keys, or just need a refresher, GitLab provides a great resource called Use SSH Keys to Communicate with GitLab.

With our SSH keys in place we were ready to clone my son's projects onto our machine. We wanted to be fancy and perform the cloning using the Terminal. To clone a repo you use the following command in the directory where you want the code to live:

git clone git@gitlab.com:account_name/repo_name.git

On the topic of directory structure we've kept our server very basic. To keep with Mac conventions we decided to keep all our files in the ~/Documents folder. For my son's projects we're going to be running the infrastructure with Kubernetes, so we named the directory for all his code phippy. With code cloned to our machine we wanted to have just a single GUI-based code editor. We chose VS Code.

VS Code

We wanted this machine to be as lean as possible. There is only so much memory and processing power to go around. Remember, our machine has 3 GHz Dual-Core Intel Core i7 processor with 16 GB 1600 MHz DDR3 memory. We also wanted as much of that space and power to be used for serving up our web applications. However, we also wanted to have an additional option for editing any code files, in addition to vim.

VS Code has been my defacto GUI code editor for years, mainly because Microsoft has really nailed the lean part right when it comes to this application. It runs quite smoothly and efficiently.

In addition to installing the main application, we also enabled the VS Code CLI. This allows us to have more control with how we open VS Code. We can open files or whole directories through the CLI options. It's how I usually open my VS Code sessions on every machine I use.

Those are the applications we initially installed to get our projects running. We may install more in the future, but up until now this is the application stack we wanted to use.

In our next post we go over how we created our containers, configured our Kubernetes cluster, and exposed our web applications to the world.

This Old Box: Setting Up the KVM Switch

Scott McAllister — Sun, 17 Dec 2023 09:17:57 +0000

Welcome back to my series where I am building my own application server out of an old 2014 Mac mini.

In my last post, I reformatted the hard drive of my retired 2014 Mac mini so I could start with a clean system. My family's current Mac mini is still running on the same desk. So, with two Mac minis running in a limited space I needed a solution for having both machines sharing the same peripheral hardware--monitor, keyboard, and mouse--without the need of unplugging cables between the machines.

To make the switching between machines easy, we purchased this KVM Switch.

We picked this one because it appeared easy to set up to accomplish what we needed. Looking at the picture below, you can see the clearly labeled spots for Blue and Green. Those are the HDMI and USB ports for each of the machines you'll be switching between. Those cables come with the KVM switch, which was nice.

The cables coming out of the bottom are the HDMI cable going to my monitor and the USB cable going to my USB hub. The 4-port hub (not pictured) is where I connect the keyboard and mouse to be used by both machines.

Switching between machines only requires a push of the round button ringed by colored light. The color changes depending on which machine you're currently using. To keep things straight, we put Post-It notes on each box listing the color and either "New" or "Old".

The setup of the switch was relatively straight forward, without a lot of hiccups. However, if you have any issues or questions please post them in the comments. We'd love to help!

In our next post, we'll talk about all the software we installed on our server to run and serve our web applications.

Learning ngrok: Inspect and Replay

Scott McAllister — Thu, 16 Feb 2023 22:46:36 +0000

My first post on Dev.to, written back in 2019, was about creating a local web server using Python and ngrok. Little did I know that four years later I would be an crew member on the rocket ship that is ngrok.

I joined the company a couple weeks ago, and I'm in the middle of ramping up with the organization and learning more about the product. While I explore the features that are new to me I'm going to write about things I learn that may be valuable to the community. This post is going to cover the Inspect and Replay feature.

A popular use case of ngrok is to receive webhooks on your local machine. As explained in the post I mentioned above, you can spin up a lightweight web server and run ngrok http 80 on your command line to provide a destination for incoming webhooks to land. For this exercise, I wrote the following code in Go to receive webhooks at the root endpoint of the domain.

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        fmt.Printf("Webhook: %v", r)
    })

    port := ":80"
    err := http.ListenAndServe(port, nil)
    if err != nil {
        fmt.Println(err)
    }
}

The server is listening for requests on port 80, and prints the raw http.Request which ends up looking something like this for a webhook received from PagerDuty.

Webhook: &{POST / HTTP/1.1 1 1 map[Accept:[application/json]
 Accept-Encoding:[gzip] Content-Length:[1438] Content-Type:[application/json] User-Agent:[PagerDuty-Webhook/V3.0] 
X-Forwarded-For:[44.222.99.999] X-Forwarded-Proto:[https] 
X-Pagerduty-Signature:[v1=e3d475277secret24919127safe26759d684anotreal6285b25c4e15a89b7ccb9] 
X-Webhook-Id:[eec61a23-d1af-4b1b-8fe3-afea6fdead49] 
X-Webhook-Subscription:[PXXXXX]] 0x140001582c0 <nil> 1438 
[] false tunnel.ngrokpaperscissors.com map[] map[] <nil> 
map[] [::1]:57318 / <nil> <nil> <nil> 0x14000158300}

There is definitely more I could do to my code to get a better view of the incoming webhook. However, it appears ngrok already does a lot of that for us out the box. I've recently discovered the ngrok inspector. It can be accessed while ngrok is running by opening http://localhost:4040/ where you're shown a view like this.

All the HTTP requests are listed down the left side, and on right is a nicely formatted representation of the webhook JSON payload. There are also tabs to look at the Headers, the Raw request, and even the Binary (though, I'm not entirely sure why I would want that :thinking_face: If you know, please share in the comments).

There's also a Replay button that not only lets me replay the selected HTTP request, but you can also choose to modify the replayed request. For example, in my current situation I'm consuming webhooks from PagerDuty incidents. The webhooks are fired depending on whether the state of an incident changes. Some examples of incident state include triggered, acknowledged, and resolved. If my code that's consuming these webhooks behaves differently depending on what type of action occurring on the incident then to speed up my debugging I could select Replay with Modifications.

Find the event_type in the JSON body of the request and change it from incident.triggered to incident.acknowledged and click Replay.

That seems pretty handy. I could see myself using that in the future. Hopefully you found this useful. If you have any questions or constructive suggestions please feel free to leave a comment.

Building My Own Application Server: Post 1

Scott McAllister — Mon, 13 Feb 2023 06:08:17 +0000

Remember when blogs first came out and most of them were written records of journeys folks encountered? Some embarked on adventurous travels, some were (and still are) a platform for sports fans to shout into the void, while others documented their progress in a particular project. This is more of the latter. I have a late 2014 Mac mini that has been cold on my desk for several months. We initially kept it arround just in case we needed something. But, I think it's safe to say that the just in case time period has adequately passed.

Now my son and I want to turn this old Mac into a web server. We realize it would most definitely be easier to deploy our code onto one of the many application service providers (aka "clouds"), but what's the fun in that? While I'm a web application developer by trade, and my son has built his own web projects, neither of us have ever built and ran a production server before, so we're going to give it a shot.

These posts are going to be a journal of our progress, our missteps, and our learnings.

Progress Update

I reformatted harddrive on the Mac mini (Late 2014). It's running a 3 GHz Dual-Core Intel Core i7 processor with 16 GB 1600 MHz DDR3 memory. The machine contains a 2 TB Fusion Drive. My initial hypothesis is that the Fusion Drive is going to hurt our performance. But, hopefully, since we stripped the machine of all non-essential files and applications, we won't notice any degraded speeds.

Outside of the Mac mini the only other hardware of note is the KVM Switch I ordered so we can easily switch between the server and our family machine, which will share the same monitor, mouse, keyboard.

Next Steps

We don't have a master plan for how we're going to run and deploy things. I know I want to learn more about running containers and container orchestration, but that's probably further down the road. The next steps we're going to focus on is getting runtimes and other utilities installed so we can run our applications.

Please feel free to offer advice, encouragement, or constructive feedback in the comments.

Stay Informed with PagerDuty Webhook Subscriptions

Scott McAllister — Fri, 02 Sep 2022 23:43:41 +0000

PagerDuty webhooks are evolving.

With the recent introduction of v3 webhook subscriptions, v1 and v2 of PagerDuty webhooks will both be retired in the coming months (v1 EOL Oct. 2022 and v2 EOL March 2023). For those reasons alone, you'll want to update your code to use v3 webhook subscriptions as soon as possible. But, let's also talk about what you'll be gaining by adopting v3 webhook subscriptions.

Webhooks at PagerDuty

The first version of webhooks at PagerDuty covered a relatively small number of events. You essentially could only be notified when an incident was triggered, acknowledged, resolved, assigned, escalated, or delegated.

An incremental update to webhooks came in the form of a v2 release which allowed you to receive webhooks whenever an incident was annotated. However, as the functionality of incidents in PagerDuty increased, so has the need to receive notifications on more than just the core incident events. Things like when an incident has been reassigned, reopened, or the status update on an incident has been published were events that needed webhooks. In addition to the new incident actions, webhooks needed to be able to support other resources in PagerDuty. Resources like Services. V3 webhooks now support when a service is created, updated, or deleted.

The following table shows all the the events supported by webhooks and which version the events are supported in. The data for this table comes from the version comparison section of the PagerDuty documentation on webhooks.

Resource	Event	v1	v2	v3
incidents	`trigger/triggered`	✓	✓	✓
	`acknowledge/acknowledged`	✓	✓	✓
	`unacknowledge/unacknowledged`	✓	✓	✓
	`resolve/resolved`	✓	✓	✓
	`assign`	✓	✓	✓
	`escalate/escalated`	✓	✓	✓
	`delegate/delegated`	✓	✓	✓
	`annotate/annotated`		✓	✓
	`priority_updated`			✓
	`reassigned`			✓
	`reopened`			✓
	`responder.added`			✓
	`responder.replied`			✓
	`status_update_published`			✓
services	`created`			✓
	`deleted`			✓
	`updated`			✓

You'll notice a few of the original incident event types have two names, such as trigger and triggered. That's because the names of those events changed in v3 of webhooks. If there is a / in the table above the second name is the v3 version of the event name.

V3 Webhook Subscriptions

V3 webhook subscriptions can be created through the PagerDuty UI and by making calls to the API. For this article we're going to focus on managing subscriptions through the API. For information on managing subscriptions through the UI checkout the Webhooks support documentation.

Create Webhook Subscription Through API

You create a webhook subscription by making a call to the webhook_subscriptions endpoint with a webhook_subscription object defined in the body of the request. The subscription object has three main components--delivery method, events, and filter.

Delivery Method

The delivery_method of a webhook subscription defines how and where a message for an event will be delivered. There are two required fields: type and url. At this time the only valid value for type is http_delivery_method, and the url is the location of where the webhook should be sent.

Events

The events field is an array of event types the webhook subscription is interested in. For a complete list of possible event times, and the syntax for including them, see the Event Types section of the PagerDuty Webhook developer documentation.

Filter

The filter value sets the scope of the events that are being sent. You can filter on three types--service_reference,team_reference, and account_reference.

Putting all of those pieces together, creating a webhook_subscription with a service_reference filter would something like this:

{
  "webhook_subscription": {
    "delivery_method": {
      "type": "email",
      "url": "https://example.com/receive_a_pagerduty_webhook"
    },
    "description": "Sends webhook events somewhere interesting.",
    "events": [
      "incident.acknowledged",
      "incident.annotated",
      "incident.delegated",
      "incident.escalated",
      "incident.priority_updated",
      "incident.reassigned",
      "incident.reopened",
      "incident.resolved",
      "incident.responder.added",
      "incident.responder.replied",
      "incident.unacknowledged",
      "incident.status_update_published"

    ],
    "filter": {
      "id": "PJHZQ7R",
      "type": "service_reference"
    },
    "type": "webhook_subscription"
  }
}

NOTE: Something to be careful of, as of this writing, there isn't a check to see if the service ID you're passing is valid. Make sure to verify your service ID is correct before creating a webhook_subscription.

Webhook Payload

With the webhook subscription created it’s now ready to send webhooks on the subscribed events. The webhook payload for that event looks something like this:

{
  "event": {
    "id": "01D5RHYA931SGC5NALZ1HVKVE2",
    "event_type": "incident.priority_updated",
    "resource_type": "incident",
    "occurred_at": "2022-09-01T23:29:50.459Z",
    "agent": {
      "html_url": "https://pdt-stmcallister.pagerduty.com/users/P5JEPDQ",
      "id": "P5JEPDQ",
      "self": "https://api.pagerduty.com/users/P5JEPDQ",
      "summary": "Scott McAllister",
      "type": "user_reference"
    },
    "client": null,
    "data": {
      "id": "Q39FWCH8OEH7I0",
      "type": "incident",
      "self": "https://api.pagerduty.com/incidents/Q39FWCH8OEH7I0",
      "html_url": "https://pdt-stmcallister.pagerduty.com/incidents/Q39FWCH8OEH7I0",
      "number": 10,
      "status": "acknowledged",
      "incident_key": "59124bec832148a7b9dc5e8c43e1cf7b",
      "created_at": "2022-06-10T19:27:35Z",
      "title": "Something's Broken!",
      "service": {
        "html_url": "https://pdt-stmcallister.pagerduty.com/services/P0IQNQT",
        "id": "P0IQNQT",
        "self": "https://api.pagerduty.com/services/P0IQNQT",
        "summary": "Changie",
        "type": "service_reference"
      },
      "assignees": [
        {
          "html_url": "https://pdt-stmcallister.pagerduty.com/users/P5JEPDQ",
          "id": "P5JEPDQ",
          "self": "https://api.pagerduty.com/users/P5JEPDQ",
          "summary": "Scott McAllister",
          "type": "user_reference"
        }
      ],
      "escalation_policy": {
        "html_url": "https://pdt-stmcallister.pagerduty.com/escalation_policies/P1CSJJO",
        "id": "P1CSJJO",
        "self": "https://api.pagerduty.com/escalation_policies/P1CSJJO",
        "summary": "Default",
        "type": "escalation_policy_reference"
      },
      "teams": [],
      "priority": {
        "html_url": "https://pdt-stmcallister.pagerduty.com/account/incident_priorities",
        "id": "PDNVXWA",
        "self": "https://api.pagerduty.com/priorities/PDNVXWA",
        "summary": "P3",
        "type": "priority_reference"
      },
      "urgency": "high",
      "conference_bridge": null,
      "resolve_reason": null
    }
  }
}

As you can see, there’s a lot to that event object. Something to notice is the event.data object. This is the meat of the incident. It’s where you’ll find information like title and status, as well as the service, assignees, and escalation policy objects. Rather than go into detail about the structure of this payload here in this post, I invite you to head over to the Webhook Payload section of the Webhooks Developer documentation.

Let Us Know

We'd love to hear about your experience migrating to v3 Webhook Subscriptions. Let us know in the comments how things went, or if you have any questions during your migration process.

Requests for Go or Terraform Content?

Scott McAllister — Mon, 28 Mar 2022 23:00:37 +0000

I have found myself completely immersed in the Go and Terraform ecosystems lately. Most of my time is focused on learning both of these technologies. I'm learning new things all the time and would love to share them. I've written a couple posts (Write Terraform Files in Go with hclwrite
& Create Terraform Resource References in Go with hclwrite) on the hclwrite Go library for creating HashiCorp Configuration Language (HCL) and would love to write more on Go!

Are there any posts on either Go or Terraform that you would like to see?

Create Terraform Resource References in Go with hclwrite

Scott McAllister — Tue, 22 Mar 2022 06:50:26 +0000

In our last post I introduced the hclwrite library from HashiCorp which allows you to write HashiCorp Configuration Language (HCL) using Go. In that article I created a basic HCL file that defined 150 Business Service resources in a PagerDuty account. Truth be told, I picked Business Services because they didn't require reference values from other resources, because I didn't know how to construct those resource references using hclwrite...until now.

In this post we're going to create HCL that defines a PagerDuty Service. A service requires a reference to a PagerDuty Escalation Policy which, in turn, requires a reference to a PagerDuty Schedule. Each of these references are accomplished by using a resource reference expression. For example, the HCL for defining a service looks something like this, where the escalation_policy attribute is the ID of the escalation policy associated with the service.

resource "pagerduty_service" "me1" {
  name              = "Me Service 1"
  escalation_policy = pagerduty_escalation_policy.ep.id
}

We already know how to create new blocks and set primitive attribute values, as we covered those concepts in the last post. However, we need to first create a service resource block in order to set an escalation_policy attribute. That code for creating the service resource block looks something like this:

service := rootBody.AppendNewBlock("resource", 
    []string{"pagerduty_service", "me1")})
serviceBody := service.Body()
serviceBody.SetAttributeValue("name", cty.StringVal("Me Service 1")))

Now we're ready to add the escalation_policy attribute to serviceBody, which is where things get interesting. To create the name attribute in the code above we used the SetAttributeValue and the cty.StringVal functions to set the string value. But, a resource reference is not a string. It's essentially a variable. If you've been using Terraform for a while you might be thinking you could use the older expression syntax where you set the value as a string and wrapped expression with a ${ }. The hclwrite library actually watches for this syntax and tries to escape it by adding an extra $ to the front, like this $${ }.

There are a couple of approaches that will work for creating these expressions. The first is to use SetAttributeTraversal. A Traversal is designed to define a path of an expression. To define the escalation_policy as a Traversal it would look something like this:

serviceBody.SetAttributeTraversal("escalation_policy", hcl.Traversal{
    hcl.TraverseRoot{
        Name: "pagerduty_escalation_policy",
    },
    hcl.TraverseAttr{
        Name: "ep",
    },
    hcl.TraverseAttr{
        Name: "id",
    },    
})

This creates a perfectly formatted expression of pagerduty_escalation_policy.ep.id.

The other way to create the necessary reference syntax, and the one I prefer, is to use SetAttributeRaw. This function requires you to define the entire expression manually as a series of Tokens. A Token is a struct containing Type and Bytes fields. The Type comes from the hclsyntax.TokenType. And Bytes is a []byte with the value of the Token. In this case, where we're setting the escalation_policy value, the Tokens and subsequent SetAttributeRaw would look something like this:

tokens := hclwrite.Tokens{
  {
    Type: hclsyntax.TokenIdent, 
    Bytes: []byte(`pagerduty_escalation_policy.ep.id`),
  },
}

serviceBody.SetAttributeRaw("escalation_policy", tokens)

There are a few reasons why I prefer using the SetAttributeRaw method. The first advantage you'll notice from this approach is that it takes less code than the Traversal. However, the real advantage comes when creating attributes that contain a list value. We do just that when defining a pagerduty_schedule resource. In the layer block of the schedule resource we set a list of users.

The HCL for the entire PagerDuty Schedule looks something like this:

resource "pagerduty_schedule" "foo" {
  name      = "Me Schedule"
  time_zone = "America/Los_Angeles"

  layer {
    name                         = "This Shift"
    start                        = "2022-03-15T20:00:00-08:00"
    rotation_virtual_start       = "2022-03-15T20:00:00-08:00"
    rotation_turn_length_seconds = 86400
    users                        = [data.pagerduty_user.me.id]

    restriction {
      type              = "daily_restriction"
      start_time_of_day = "17:00:00"
      duration_seconds  = 54000
    }
  }
}

The Go code needed to create this pagerduty_schedule resource and layer blocks looks like this:

// create schedule
sched := rootBody.AppendNewBlock("resource",
  []string{"pagerduty_schedule", "foo"})
schedBody := sched.Body()
schedBody.SetAttributeValue("name", cty.StringVal("Me Schedule"))
schedBody.SetAttributeValue("time_zone", cty.StringVal("America/Los_Angeles"))
schedBody.AppendNewline()
schedLayer := schedBody.AppendNewBlock("layer", nil)
schedLayerBody := schedLayer.Body()
schedLayerBody.SetAttributeValue("name", cty.StringVal("This Shift"))
schedLayerBody.SetAttributeValue("start", cty.StringVal("2022-03-15T20:00:00-08:00"))
schedLayerBody.SetAttributeValue("rotation_virtual_start", cty.StringVal("2022-03-15T20:00:00-08:00"))

Creating the Tokens for the users list looks extremely similar to the code we used above to create the escalation_policy reference. The difference now is that we wrap our data.pagerduty_user.me.id value in opening and closing square brackets, like so:

userTokens := hclwrite.Tokens{
  {
    Type:  hclsyntax.TokenOBrack,
    Bytes: []byte(`[`),
  },
  {
    Type:  hclsyntax.TokenIdent,
    Bytes: []byte(`data.pagerduty_user.me.id`),
  },
  {
    Type:  hclsyntax.TokenCBrack,
    Bytes: []byte(`]`),
  },
}

schedLayerBody.SetAttributeRaw("users", userTokens)

Hopefully, you now have a better idea of how to use the SetAttributeRaw function when writing resource references in HCL when using the hclwrite library. You can find all the code that was described in this article in the Create Terraform Files in Go repository on GitHub.

Write Terraform Files in Go with hclwrite

Scott McAllister — Wed, 18 Aug 2021 07:49:37 +0000

I am one of the lead maintainers of the PagerDuty Terraform Provider which means I find myself writing a lot of Terraform definitions in HashiCorp Configuration Language (HCL). The provider itself contains a healthy collection of acceptance tests, but I often still write some of my own HCL to verify use cases and make sure bug fixes address specific issues raised by users.

To create these Terraform definitions I commonly write the HCL by hand. However, this can get tedious if I need to create, say, 150 of the same resource to test pagination coming from the PagerDuty API. Historically, I used a Python script that verbosely wrote out the HCL syntax like so

def createService(label, id, output):
    tf_code = "resource \"pagerduty_service\" \""+id+"\" {\n\
    name = \""+label+"\"\n\
    escalation_policy = pagerduty_escalation_policy."+str(id)+".id \n\
    alert_creation = \"create_alerts_and_incidents\" \n\
}\n\n"

    output.write(tf_code)

Functions like this createService above would be written for each resource and put inside of a loop to generate the HCL resource blocks that were needed for the definition. Honestly, this method worked fine for me. At least it did until I wondered if Go provided a better way. That's when I discovered the hclwrite package from HashiCorp.

The project documentation describes hclwrite as a package that, "deals with the problem of generating HCL configuration and of making specific surgical changes to existing HCL configurations." This turned out to be exactly the package I needed. Rather than writing out HCL syntax by hand the hclwrite package would do that for me while I just called functions to create the objects.

This article will help you get started with the hclwrite package by walking you through how I used it to generate a Terraform configuration for creating 150 Business Services in PagerDuty. I'll step through some of the concepts I struggled with and wished there were more examples of.

The hclwrite package is imported from github.com/hashicorp/hcl/hclwrite. This tripped me up a bit, partly because I'm still relatively new to Go and partly because the path to the package listed in the documentation is slightly different. For the rest of you also new to Go, your imports should look something like this:

import (
    "fmt"
    "os"

    "github.com/hashicorp/hcl/v2/hclwrite"
    "github.com/zclconf/go-cty/cty"
)

You'll notice another import for the go-cty package. This package (pronounced see-tie) provides some infrastructure for a type system that might be useful for applications that need to represent configuration values. You'll see it used throughout the examples when setting types to attribute values.

To get things started you'll need to create two different types of file objects. One for hclwrite and another for the filesystem. This can be done with the following code.

// create new empty hcl file object
hclFile := hclwrite.NewEmptyFile()

// create new file on system
tfFile, err := os.Create("bservelist.tf")
if err != nil {
    fmt.Println(err)
    return
}
// initialize the body of the new file object
rootBody := hclFile.Body()

The content of every object in hclwrite is stored in a body object. To add or append anything to an object you'll need to reference its body. In the code above you see that we named the body of the HCL document object rootBody. The first thing we need to do with rootBody is set up the provider block for the PagerDuty provider. The HCL for this block looks like this

provider "pagerduty" {
  token = "yeahRightN0tgo1ng2t3llyOuTh@t"
}

Constructing this block using the hclwrite package requires the AppendNewBlock function which is expecting two arguments. First is a string which will set the type of block and the second argument is an array of strings that act as labels for the block. In the case of this provider block we want to set the label simply as pagerduty. That would look something like this:

provider := rootBody.AppendNewBlock("provider", 
    []string{"pagerduty"})

Inside this block we need to set a token attribute to the value of a PagerDuty API Key. Remember, the attribute needs to be added to the body of the block. So, we'll first set that body value to the providerBody variable and then call SetAttributeValue, passing the label and value of the attribute as arguments.

providerBody := provider.Body()
providerBody.SetAttributeValue("token", 
    cty.StringVal(os.Getenv("PAGERDUTY_TOKEN")))

The next block we are going to create is the terraform block where we define which providers we're going to use and the versions of those providers. What's interesting about these definitions is they require two nested blocks that don't have any labels. The AppendNewBlock function handles his by accepting a nil argument for the labels. Creating the two blocks looks like this:

tfBlock := rootBody.AppendNewBlock("terraform", 
    nil)
tfBlockBody := tfBlock.Body()
reqProvsBlock := tfBlockBody.AppendNewBlock("required_providers", 
    nil)
reqProvsBlockBody := reqProvsBlock.Body()

Inside the required_providers block we need to define an attribute called pagerduty that contains a value of an object with two key-value pairs as fields. This is done by setting the value to a cty.ObjectVal as a map.

reqProvsBlockBody.SetAttributeValue("pagerduty", 
    cty.ObjectVal(map[string]cty.Value{
        "source":  cty.StringVal("PagerDuty/pagerduty"),
        "version": cty.StringVal("1.10.1"),
}))

The generated HCL for this code will look like this:

terraform {
  required_providers {
    pagerduty = {
      source  = "PagerDuty/pagerduty"
      version = "2.3.0"
    }
  }
}

Now it's time to start creating actual resources. Remember, our task was to create 150 Business Services. In HCL, a resource block is just like the other block types we've created. The main difference is resource blocks contain multiple labels. In this case, each resource block contains the pagerduty_business_service label for the resource type along with the identifier label for the resource. Because we only care about creating a whole lot of Business Services it doesn't matter what they're named. So, we're just going to use the index i from our loop to put a number variable into each Business Service name. For example, the name of the first Business Service will be "Business Service 1". The code for creating those resource blocks looks like this:

bs := rootBody.AppendNewBlock("resource", 
    []string{"pagerduty_business_service", 
        fmt.Sprintf("bs%v", i)})
bsBody := bs.Body()
bsBody.SetAttributeValue("name", 
    cty.StringVal(fmt.Sprintf("Business Service %v", i)))
rootBody.AppendNewline()

When the looping is done, and the HCL for all 150 Business Service definitions has been generated, the last thing to do is to write all the definitions out to a .tf file. There are a few ways to do this. I went for the way I was most familiar with, where I wrote the bytes from the hclwrite file object to the tfFile, like this:

tfFile.Write(f.Bytes())

And that's it. You should be able to run the configurations in the .tf file you created and populated. To see all the code used in these examples checkout the Create Terraform Files in Go project over on GitHub. This was also only an introduction to the hclwrite package. Go see the hclwrite documentation to see all the available functions.

Assigning a Google Domain to GitLab Pages

Scott McAllister — Sat, 03 Aug 2019 18:33:19 +0000

Before I jump into the nitty gritty of how to add a domain registered with Google Domains to a GitLab Pages site, I want to share a little backstory. If you're not interested in reading the human side of how and why I learned about custom domains with GitLab Pages, please skip ahead to the GitLab Pages section.

Backstory

A few years ago, I arrived home to my eight-year-old son excitedly telling me that he found a book at the library about HTML and built a website. This happened as a complete surprise and without any prodding from his software developer dad. After seeing the site, the dialog went something like this.

I told him, "that's great! Want me to put it on a server for you?"

"What's that!?" he asked.

"On a server, anyone can come see your site," I explained.

"Yeah!"

At the time, I was using a web host that required me to upload files via FTP. This proved to be a bottleneck in my son's workflow. Every time he had an update, he would bug me to upload it to the server.

Finally, I got wise and got out of his way. I put his code in a GitLab repository, and set up a GitLab Pages site. Then, I showed him the command line--which he tought was exceedingly cool--and taught him a few simple Git commands. And, voila! My son could publish changes as often as he wanted, without waiting for his old man.

This process continues to work well for both of us. But recently, I wanted him to feel a little more ownership of his site. You see, all of the sites he built were being served off of my generic GitLab domain https://stmcallister.gitlab.io/webkid/. So, rather than looking like he was piggybacking on my site, I wanted to get my son his own domain.

We went over to Google Domains and searched long and hard to find the name he wanted and arrived at http://ajmcweb.com. It's short. It's easy to spell. And, most importantly, he liked it!

Now, I needed to figure out how to assign the domain to the site. That's where the instruction piece of this article begins.

GitLab Pages

In the GitLab repository for the Pages site, go to Settings -> Pages. On the Settings page click the green New Domain button in the upper right corner.

On the New Pages Domain page enter the domain you want to use for your GitLab pages site. There are also options for entering certificates and private keys. Since the site we're simply static and not working with any kind of information, we did not set either of these values. When ready, click Create New Domain.

On the Pages Domain page you'll see a form entry for DNS that shows the CNAME entry of your new domain pointing to your GitLab Pages domain. Just below the DNS field is Verification status. Copy the value for Verification status, you'll need to add this to your Google Domain account to verify ownership of the domain you want to use.

Google Domains

In Google Domains, select your domain, go to the DNS section, and scroll down to Custom resource records. You will add two resource records to point your Google domain to your GitLab pages site.

First, take the value copied from the GitLab Verification status and create a resource record of type TXT. Paste the GitLab verification code in the Text field. The default value for TTL is set to 1H, but if you want to see changes occur more quickly, change it to 1m. Then, click Add.

TTL stands for Time to Live, and it stands for how long a copy of the record stays in cache before being discarded. That means if you leave the TTL set to 1m, then that resource is getting looked up every minute. So, after changing the value to 1m to see the DNS change populate, it would be a good idea to change it back to the 1H, or even 1D, value to reduce the numbers of look ups.

The next resource record to add is a DNS A record, which gives your domain an IP address for GitLab pages. As of this writing, that value is 35.185.44.232. You may also want to check the DNS A record documentation on GitLab Pages to make sure this value is still current.

With those two records set in Google Domains, go back to the Pages Domain page on GitLab and click the reload circle arrow to check if the domain now verifies. The unverified message should turn green and change to Verified. And, for a final test, point your browser to your new domain, and you should see your GitLab Pages site.

Now, my son is happy that his site has a custom domain. And--hopefully--you're happy that you found all this information in one article. You're welcome. ;)

Create a Quick Local Web Server with Python and Ngrok

Scott McAllister — Sun, 12 May 2019 04:50:06 +0000

One of the nice things about starting a new job is that you have to setup all of your development tools and environments from scratch. And some of the things you thought you'd never forget stubbornly hide behind the memories that are much easier to recall.

Such was the case for me recently. On a machine that was only a few weeks old, I needed to make an HTML file quickly accessible to the outside world. I had done this dozens of times before, but for the life of me, I could not remember the exact tools I needed to get this job done.

Then, after much searching, the fog cleared. All I needed was to run a single Python command to fire up a local web server. Then, using a tool called ngrok I could make that server accessible to the world.

This blog post is more of an insurance policy against my faulty memory, so I can easily remind myself of these things in the future. However, you are more than welcome to continue reading if this sounds interesting to you. First, let's get the Python web server running on port 8000.

Python

A super quick way to fire up a web server, with virtually no configuration (especially if you're on macOS) is to use Python's SimpleHttpServer. In Python 2.7 (which is the default version that comes with macOS) the server is started by running python -m SimpleHTTPServer 8000.

In Python 3 this can be done by running python -m http.server 8000.

With this command, the SimpleHTTPServer will serve up the contents of the current directory on port 8000. Any port can be specified here, but make sure it's one that is not currently in use. The contents can be accessed locally by pointing your browser to http://localhost:8000/. Now that the simple web server is running, it's time to configure ngrok to make this server accessible by the Internet at large.

ngrok

Ngrok is a tool that provides a secure tunnel for a local application to access the Internet, and for the Internet to access it. This tunneling comes in handy when demoing or testing web sites without having to deploy them. For instance, if you're building an application that subscribes to webhooks, and those webhooks need a callback URL to access the application on your machine, ngrok would provide that address for the webhooks to call. In my case, I just needed a simple way to serve up HTML files.

There are a lot of features to ngrok, but this article is only going to focus on providing a simple http tunnel on a specified port. This can be done with the following steps:

Download ngrok from the ngrok Download Page.
Extract the ngrok executable and put it somewhere that makes sense to you. I put it in /Applications/Utilities.
To get the ngrok executable into PATH so that it's executable from anywhere on your system, create a symlink in /usr/local/bin/ that points to where ever ngrok is saved. If it's in the Utilities directory, as mentioned above, the symlink command would look like this ln -s /Application/Utilities/ngrok /usr/local/bin/ngrok
Go the the directory where SimpleHTTPServer is running and run the following command ngrok http 8000

The ngrok output then shows the following:

The important piece you're looking for are the values for Forwarding. The local web server will now be visible at the x.ngrok.io domains. As requests are made to this tunnel, the results are logged in the Connections section of the ngrok CLI. For the free version of ngrok, this session will last for 8 hours. When the ngrok process expires, or is restarted, the subdomain value in the address will change. If you're looking for a consistent address, I believe you can purchase a license from ngrok.

Now with these two simple tools you can spin up publicly accessible web servers to your heart's content.