DEV Community: Brett Curtis

Using GitHub to Manage GitHub!

Brett Curtis — Sat, 21 Jan 2023 16:13:17 +0000

GitHub Organization Management Platform

Updated version here: https://www.osinfra.io/github-organization-management-platform

Managing thousands of repositories across hundreds of users in GitHub can become very complex and tedious. You'll want basic naming conventions on repositories and teams, ways to manage users in those teams, roles, and security settings enforced as a minimum. The Terraform provider for GitHub allows us to manage our GitHub organization as a platform, just like any other infrastructure. Doing this will reduce technical debt when onboarding, provide ongoing supportability and give a defined team interaction mode of GitHub-as-a-Service.

This open-source repository is an example of Infrastructure as Code (IaC) for managing a GitHub organization using Terraform, concepts from Team Topologies, and sensible default practices.

Currently, we have the following service interfaces defined by the beta GitHub Issue Forms that you can see in the README.md:

github_membership service interface: GitHub Issue
github_repository service interface: GitHub Issue

Any issue created by an exposed service interface in any platform we build is labeled as a good first issue and documented to make completion of the work possible within the individual author's cognitive load. These issues align with one of our key concepts of cultivating and developing a strong pool of talented individuals ready to take on work and grow within the organization. They are helping new Infrastructure as Code engineers learn the GitHub flow through muscle memory. Large organizations will fail at Infrastructure as Code if they expect "traditional" infrastructure engineers to take on the cognitive load of learning and treating infrastructure like code while learning cloud platforms. They are left with, at best, poor automation practices, not Infrastructure as Code.

After we merge a pull request, we use GitHub reusable workflows with OIDC to our Google Cloud Terraform backend Platform to run the terraform plan and use GitHub environments with a required reviewers protection rule before the apply job runs. Giving the pull request author a chance to look over the Terraform plan in detail and validate the changes are what they expect.

We also use Dependabot and reusable workflows to keep all our actions and terraform provider dependencies up to date. Along with CODEOWNERS to ensure the correct platform team is requested to review. The notifications are a bit noisy right now, but we hope to see an option to make a pull request draft by default so code reviewers will only receive notification once the pull request is ready for review.

In conclusion, GitHub is an extremely powerful tool that, hands down, enables collaboration and learning across engineers. GitHub powers Infrastructure as Code engineers and software engineers by creating a common language and skill set across the platform and stream-aligned teams with a comprehensive, end-to-end infrastructure and software development solution. With that and organizational design like Team Topologies in place, we see increased success in individuals, teams, organizations, and the software built for their customers.

Technical Documentation (WIP): docs.osinfra.io

GitHub Actions for Kitchen-Terraform Testing

Brett Curtis — Sat, 19 Sep 2020 18:07:29 +0000

In the last post, we worked on Kitchen-Terraform running locally, and now we want to run it from GitHub Actions.

First, we can talk at a high level about the GitHub flow in use. I'll probably write up a different post with more details around this because I find folks with a core operational background haven't used GitHub or understand the "whys" around doing some of the things developers do.

Create an issue
Create a branch to work on
- NOTE: Branches are deleted after a successful merge into mainline. Also, in the spirit of continuous delivery, these are not long-lived branches and should be integrated frequently
Create a pull request
- NOTE: Once the pull request is created, it will (based on our action configuration) run the GitHub Action
Add someone to for review after successful GitHub Action run
Squash and merge your commit after code review

To get your GitHub Action setup, the first thing we will need to do is create a self-hosted runner. For this example, I ended up using a small compute instance in Google Cloud Platform. You can pick any cloud provider you want or even a machine in your datacenter if needed. It just needs to have the tools installed to do whatever you're asking your action to run. In our case, all the bits to run Kitchen-Terraform.

Adding a GitHub self-hosted runner has to be the most accessible CI/CD tool "agent-based" install ever, so I don't think we need to go into the details here. I did try to create my own packaged Docker container action for Kitchen-Terraform here, but I wouldn't say I liked how it had to build the image each run.

Once your runner is up and running, you will need to tell GitHub what you want to do and where to do it. We do this using a workflow file. The workflow file is written in YAML and lives inside your GitHub repository in the .github/workflows directory. You can take a look at the GitHub Actions Cheat Sheet, it describes the contents of a workflow file. Ours is simple, one Job with three steps. First, we tell GitHub the name of the action and when to run it. In our case, on commit to master and pull requests as I talked about earlier. Next, where to run the action:

The runs-on has the self-hosted runner labels:

As mentioned in the previous post, we are passing BILLING_ID to terraform as a secret from GitHub. You can set secrets and an organizational level or a repository level.

Next, we define a default shell to use and the steps to run:

The first step is to use the GitHub Checkout Action to checkout the repository on the runner. The second is to use the Google Cloud Platform Setup gcloud Environment Action to configure our Google credentials. This action also allows us to set up Google Application Default Credentials (ADC) by setting export_default_credentials: true. This is slick because Terraform can use these by default without setting anything up! Finally, we end up at the last step, which runs bundle exec kitchen test on our self-hosted runner:

There we have it! You have yourself a nice little Terraform module you can continue to improve upon over time in a supportable way just like I quoted in another post about using some of the good practices we've learned in the software world like:

Using source control, adhering to the DRY principle, modularization, maintainability, and using automated testing and deployment.

In the next post, we can talk about putting multiple Terraform modules together with the inspec-gcp-cis-benchmark profile running across all the Google Cloud Platform resources created.

Using Kitchen-Terraform with the GCP CIS Benchmark Profile

Brett Curtis — Sat, 05 Sep 2020 19:27:13 +0000

In the last post we covered "local" development. In the spirit of continuous integration, we want to be able to have team members integrate their work. If you remember, the violations we had from running the inspec-gcp-cis-benchmark profile, we've got some work to do! We will do that using Kitchen-Terraform and run the workflow in GitHub Actions. In this post, we will cover Kitchen-Terraform.

Project Setup

The directory structure is essential when using Kitchen-Terraform. My current structure looks like this:

Fixtures

You'll need to have a fixtures directory.

This directory is where your Terraform root modules will live or in Kitchen-Terraform language, "test fixtures." It will call the child module, the module we are developing. That language aligns with the module documentation, so in our case, the child module is in the root of the repository, and it's the module we are publishing through GitHub for reuse. In this example, we only have one root module or test fixture we are testing against. There could and most likely will be more if we can't cover multiple test scenarios in a single fixture. I also have a shared directory for common code across numerous test fixtures if needed.

Integration

The integration directory will hold your profile with the Chef Inspec DSL controls you want to run against a test fixture.

For this post, we will focus only on the cis_benchmark.rb and the inspec.yml files. Since Google did all the heavy lifting for us by writing the controls for the specific benchmarks we are testing for, 3.1 Networking and 4.4 VMS, we can add them very quickly. First, you add the repository you depend on:

NOTE: The input gcp_project_id is what the inspec-gcp-cis-benchmark profile expects in order to know what project to run the tests against. Kitchen-Terraform can do attribute and Terraform output mapping. I'll touch on why that's important when we look at the kitchen.yml file.

Then to call a specific control, we add it into the cis_benchmark.rb like so:

kitchen.yml

The last thing we need to do is tell Kitchen-Terraform what to do! We do this by creating a kitchen.yml file:

You can read the documentation about the kitchen.yml if you want to know more. However, I do want to touch on a couple of things.

Lines three and four are a way you can pass -var="foo=bar" to Terraform from inside Kitchen-Terraform. In this example, I'm using the billing ID as a secret in GitHub. We can talk about that a bit more when we set up the workflow in GitHub Actions.

The other thing is the lines twenty-four and twenty-five. This has to do with the attribute and Terraform output mapping I talked about earlier. I took me a bit of learning out loud to get this squared away, but with some help, I was able to get it. To make a long story short, this is how I mapped my Terraform output of project_id to the attribute needed to run the inspec-gcp-cis-benchmark profile.

Running Kitchen-Terraform

Now we can run kitchen converge and see what we have:

This is going to do all kinds of work for you. It will verify the Terraform client version, initialize the Terraform working directory as well as create and switch to a workspace:

It will also validate the Terraform configuration files and finally run the Terraform apply:

Next, we can run the tests using kitchen verify and see the results:

Finally, we can destroy the infrastructure using kitchen destroy. You could also do all of the above in one command by running kitchen test.

In the next post, I will talk about running the testing workflow through GitHub Actions!

Terraform Development & Testing Introduction

Brett Curtis — Fri, 04 Sep 2020 21:49:14 +0000

I'm starting to play around with kitchen-terraform for testing and compliance of Terraform modules. I'm currently using the inspec-gcp resource pack and the inspec-gcp-cis-benchmark profile. This post will focus on the latter. The drivers for doing this are testing code's self-evident benefits and shifting some reasonable base security standards left.

To keep things simple, I wanted to start from what you might do first when you begin building cloud infrastructure out in Google. The base of any Google Cloud Platform service is a project. The module I will be talking about is here for reference and covers building a project. It's a reasonably opinionated terraform module to fit some use cases I had; however, I think it shows the example of what I'm trying to do without being too complex and confusing. Because honestly, this stuff is hard for non-developer, ex-operational mindset people like me.

NOTE: I'm not going into the basics of setting up Terraform, Ruby, Kitchen-Terraform, Chef Inspec, etc. If that's interesting, let me know, and maybe I can do another post.

"Local" Development

Learning from the practices of software development, we need to be able to develop "locally." When I say locally here, we can consider it a sandboxed area in the Google Cloud Platform resource hierarchy where you can create and build infrastructure safely. When you start your Terraform module development, you will most likely end up with something like this in your main.tf:

# Project Resource
# https://www.terraform.io/docs/providers/google/r/google_project.html

resource "google_project" "this" {
  name                     = var.project_id
  project_id               = var.project_id
  billing_account          = var.billing_id
  folder_id                = "folders/${var.folder_id}"
}

and a variables.tf file like this:

variable "billing_id" {
  description = "Billing ID for the project to use"
  type        = string
}

variable "project_id" {
  description = "Project ID (This will be used for the project name as well)"
  type        = string
}

variable "folder_id" {
  description = "Folder ID for the project to be created in."
  type        = string
}

We can now run the terraform and create our Google Cloud project:

terraform init

terraform plan -out plan.out -var="billing_id=00000C-AZAZAZ-EFEFEF" \
-var="project_id=test-del-me-4876des" \
-var="folder_id=993877078800"

terraform apply plan.out

Now you've got a project up and running, but you may be surprised that you have already violated a bunch of CIS Benchmarks! Thanks to Google and the work folks have done in the inspec-gcp-cis-benchmark GitHub repository; we can test it!

NOTE: This is not an officially supported Google product, but hopefully, they maintain this repo with help from the community.

One thing I love about this GitHub project is that it runs directly against a specific Google Cloud project. For me, that's precisely the level at which I want to run the tests. I'll get into more details on that later, but for now, lets inspec the project we just built to see what we need to do:

inspec exec https://github.com/GoogleCloudPlatform/inspec-gcp-cis-benchmark.git \
-t gcp:// --input gcp_project_id=test-del-me-4876des

After running that command, you're going to see a list of violations. In the spirit of Test Driven Development (TDD), we have our failing test, and now we can code to fix it. We will integrate all this is the next post. For the sake of simplicity, let's focus on two of them:

×  cis-gcp-4.4-vms: [VMS] Ensure oslogin is enabled for a Project
×  cis-gcp-3.1-networking: [NETWORKING] Ensure the default network does not exist in a project

The idea here is that we address these issues as far left in the process as possible. I'm still developing "locally." If you deploy a bunch of stuff on default networks, and three years later, your security team says, we need to get everything CIS compliant; you will have an unruly amount of operational work! Let's fix these compliance issues now. To do that we can add a few lines to the Terraform module code. Then, everyone in your organization that consumes it will meet the standard.

# Project Resource
# https://www.terraform.io/docs/providers/google/r/google_project.html

resource "google_project" "this" {
  name                = var.project_id
  project_id          = var.project_id
  billing_account     = var.billing_id
  folder_id           = "folders/${var.folder_id}"
  auto_create_network = false
}


# Project Metadata Resource
# https://www.terraform.io/docs/providers/google/r/compute_project_metadata.html

resource "google_compute_project_metadata" "this" {
  project = google_project.project.project_id
  metadata = {
    enable-oslogin = true
  }
}

Next, you can destroy your previous project and recreate it using this code, and you will see you've passed the tests:

✔  cis-gcp-3.1-networking: [NETWORKING] Ensure the default network does not exist in a project
✔  cis-gcp-4.4-vms: [VMS] Ensure oslogin is enabled for a Project

Not only that but we went from:

Profile Summary: 4 successful controls, 16 control failures, 21 controls skipped
Test Summary: 12 successful, 40 failures, 79 skipped

to:

Profile Summary: 5 successful controls, 11 control failures, 22 controls skipped
Test Summary: 7 successful, 12 failures, 80 skipped

I feel like the results speak for themselves. In the next post, we will focus on kitchen-terraform.

Infrastructure as Code (IaC)

Brett Curtis — Sat, 29 Aug 2020 22:30:01 +0000

With the cloud, we obviously can go fast, sometimes too fast. We can quickly stand things up via infrastructure automation; however, it can often lead to supportability problems. We have too many things and keep producing more and surprise, surprise no one is getting any less busy. Infrastructure automation is excellent in some cases, for example, short-lived projects, tossing up POC, or even services that are not business-critical. However, longer-lived projects and business-critical services need to support infrastructure as code (IaC). The monumental difference is the two words "as code," meaning:

"All the good practices we've learned in the software world should be applied to infrastructure. Using source control, adhering to the DRY principle, modularization, maintainability, and using automated testing and deployment are all critical practices. Those of us with a deep software and infrastructure background need to empathize with and support colleagues who do not. Saying "treat infrastructure like code" isn't enough; we need to ensure the hard-won learnings from the software world are also applied consistently throughout the infrastructure realm." - ThoughtWorks

The problem I see and hear across many organizations in this space is we are not dedicating resources to it, and often the pay now live later mentality that comes along with IaC gets shut down in favor of timelines. I also see that people are so embedded with their current work, done their way, using their tools that they are their own worst enemy, creating systems that only they can support. That could be from a complexity standpoint or even an access/control standpoint. That means they don't even have the time to learn something new like IaC and in some cases do not want to. Simply put, we are just stumbling over the existing cloud implementations and the comfort of doing things the same old way.

Back when we were ordering and racking servers in the data center, you could see workloads, security issues, and problems add up. Now, with the speed, flexibility, and ease of cloud, these things are multiplied, and you can see it. I think Yevgeniy Brikman, Co-Founder of Gruntworks, describes the problem well in the first 3 minutes of his story he tells back at HashiConf'17. The funny thing here is, he's only talking about one application on a single cloud provider, and it's overwhelming. Guess what, if you're in any organization larger than 500 people, I bet you have at least a few applications, if not hundreds, even across different cloud providers.

I have some hope though, hope that treating infrastructure like code is a solution. With a dedicated cross-organizational team of development, operations, testing, and security with a focus of continuous improvement, the idea that our applications' infrastructure foundation can benefit from "the hard-won learnings from the software world" can be realized!

I recently started this adventure and would be interested in hearing if IaC has worked for your organization. What are some of the essential practices you've adopted from software development? I'd also be interested in learning about some of the challenges and issues (people or technology) you had to solve along the way.