DEV Community: Raphaël Pinson

How to Automatically Issue Badges for Instruqt Labs

Raphaël Pinson — Thu, 17 Oct 2024 09:00:00 +0000

In the first blog post, we talked about making labs fun and enjoyable by adding elements of gamification. Issuing badges is a fantastic way to motivate learners, giving them a sense of accomplishment for the skills they've gained, and Isovalent issues hundreds of them every month for the Cilium labs!

Issuing Credentials

Obviously, issuing badges can be done manually, but this is not scalable or ideal for creating a seamless experience. So, let's automate it!

Credly is a widely recognized provider of digital badges, so we will be using this solution to issue badges whenever a user finishes an Instruqt lab.

We'll be using Instruqt webhooks, coupled with the Credly API, to automatically issue badges when labs are completed.

And thanks to Isovalent's open-sourced Go libraries for both Instruqt and Credly APIs, you will find this automation process smooth and straightforward.

isovalent / instruqt-go

A Go library for the Instruqt API

instruqt-go

instruqt-go is a Go client library for interacting with the Instruqt platform. It provides a simple and convenient way to programmatically access Instruqt's APIs, manage content, retrieve user data and track information.

Features

Manage Instruqt Teams and Challenges: Retrieve team information, challenges, and user progress.

Installation

To install the instruqt-go library, run:

go get github.com/isovalent/instruqt-go

Example Usage

package main
import (
    "github.com/isovalent/instruqt-go/instruqt"
    "cloud.google.com/go/logging"
)

func main() {
    // Initialize the Instruqt client
    client := instruqt.NewClient("your-api-token", "your-team-slug")

    // Get all tracks
    tracks, err := client.GetTracks()

    // Add context to calls
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()
    
    clientWithTimeout := client.WithContext(ctx)
    userInfo, err := clientWithTimeout.GetUserInfo("user-id")

    // Attach a logger
    logClient, err := logging

…

View on GitHub

isovalent / credly-go

A Go library for the Credly API

credly-go

credly-go is a Go client library for interacting with the Credly platform. It provides a simple and convenient way to programmatically access Credly's APIs and handle badges and templates.

Features

Badge Management: Issue, retrieve, and manage badges using the Credly API.

Installation

To install the credly-go library, run:

go get github.com/isovalent/credly-go

Example Usage

package main

import (
    "github.com/isovalent/credly-go/credly"
)

func main() {
    // Initialize the Credly client
    client := credly.NewClient("your-api-token", "your-credly-org")

    // Get all badges for user joe@example.com
    badges, err := client.GetBadges("joe@example.com")
}

Contributing

We welcome contributions! Please follow these steps to contribute:

Fork the repository.
Create a new branch with your feature or bug fix.
Make your changes and add tests.
Submit a pull request with a detailed description of your changes.

Running Tests

To run the tests, use:

go test ./...

Make sure…

View on GitHub

Overview

In this post, we'll take you step by step through the process:

Setting up the environment and harnessing Google Cloud Functions.
Initializing imports, constants, and setting up secret environment variables.
Implementing the webhook and explaining each step.
Setting up the webhook in Instruqt and adding signature verification to secure it.
Testing locally using Docker and Docker Compose.
Deploying the webhook and required secrets to Google Cloud Platform.
Wrapping up with some final considerations.

Let's dive in!

Pre-requisites

As for the first blog post, you will need an Instruqt account (with an API key) and a Google Cloud project.

In addition, you will also need a Credly account with an API key this time.

Setting Up the Environment

First, create a directory for your function and initialize the Go environment.

mkdir instruqt-webhook
cd instruqt-webhook

go mod init example.com/labs

Just as in the first post, we create a cmd directory so we can build and test the function locally:

mkdir cmd

Create a main.go file in that directory, with the following content:

package main

import (
    "log"
    "os"

    // Blank-import the function package so the init() runs
    // Adapt if you replaced example.com earlier
    _ "example.com/labs"

    "github.com/GoogleCloudPlatform/functions-framework-go/funcframework"
)

func main() {
    // Use PORT environment variable, or default to 8080.
    port := "8080"
    if envPort := os.Getenv("PORT"); envPort != "" {
        port = envPort
    }
    if err := funcframework.Start(port); err != nil {
        log.Fatalf("funcframework.Start: %v\n", err)
    }
}

Back to the instruqt-webhook directory, create a file named webhook.go to contain the function logic. This file will serve as the webhook handler for incoming events from Instruqt.

Setting Up the Basics

In webhook.go, begin by adding the necessary imports, constants, and initializing the function:

package labs

import (
    "fmt"
    "net/http"
    "os"
    "strings"

    "github.com/GoogleCloudPlatform/functions-framework-go/functions"

    "github.com/isovalent/instruqt-go/instruqt"
    "github.com/isovalent/credly-go/credly"

)

func init() {
    functions.HTTP("InstruqtWebhookCatch", instruqtWebhookCatch)
}

const (
    instruqtTeam = "yourInstruqtTeam"   // Replace with your own team name
    credlyOrg    = "yourCredlyOrg"      // Replace with your own credly organization ID
)

Implementing the Webhook Receiver

Now, let's write the instruqtWebhookCatch function to receive the event.

We will take advantage of the methods provided by the Isovalent instruqt-go library to manage the Instruqt webhook:

func instruqtWebhookCatch(w http.ResponseWriter, r *http.Request) {
    webhookSecret := os.Getenv("INSTRUQT_WEBHOOK_SECRET")
    wbHandler := instruqt.HandleWebhook(processWebhook, webhookSecret)
    wbHandler(w, r)
}

This function works as a proxy between the HTTP connection handler provided by the Google Cloud Functions framework and the instruqt.HandleWebhook method provided by Isovalent's library to manage the Svix webhook.

It allows us to set up a webhook manager by passing the webhook's secret. We will see later where to find the value for the webhook secret.

The instruqt.HandleWebhook method will automatically:

Verify the webhook signature using svix.
Parse the incoming event payload.
Check if the event is valid.
Retrieve the information into an instruqt.WebhookEvent structure.

Step 4: The `processWebhook()` Function

Next, we need to implement the processWebhook function, where our logic will be placed.

This function will receive 3 parameters:

the HTTP connection handlers (http.ResponseWriter and *http.Request) inherited from the GCP Function handler;
the instruqt.Webhook structure parsed by instruqt.HandleWebhook and passed down to us.

Here's the complete implementation:

func processWebhook(w http.ResponseWriter, r *http.Request, webhook instruqt.WebhookEvent) (err error) {
    // Return early if the event type is not track.completed
    if webhook.Type != "track.completed" {
        w.WriteHeader(http.StatusNoContent)
        return
    }

    // Setup the Instruqt client
    instruqtToken := os.Getenv("INSTRUQT_TOKEN")
    if instruqtToken == "" {
        w.WriteHeader(http.StatusInternalServerError)
        return
    }
    instruqtClient := instruqt.NewClient(instruqtToken, instruqtTeam)

    // Setup the Credly client
    credlyToken := os.Getenv("CREDLY_TOKEN")
    if credlyToken == "" {
        w.WriteHeader(http.StatusInternalServerError)
        return
    }
    credlyClient := credly.NewClient(credlyToken, credlyOrg)

    // Get user info from Instruqt
    user, err := instruqtClient.GetUserInfo(webhook.UserId)
    if err != nil {
        fmt.Printf("Failed to get user info: %v", err)
        w.WriteHeader(http.StatusInternalServerError)
        return
    }

    // Get track details to extract badge template ID from tags
    track, err := instruqtClient.GetTrackById(webhook.TrackId)
    if err != nil {
        fmt.Printf("Failed to get track info: %v", err)
        w.WriteHeader(http.StatusInternalServerError)
        return
    }

    // Extract badge template ID from track tags
    var templateId string
    for _, tag := range track.TrackTags {
        // Use strings.Split to parse the tag and extract the badge template ID
        parts := strings.Split(tag.Value, ":")
        if len(parts) == 2 && parts[0] == "badge" {
            templateId = parts[1]
            break
        }
    }

    if templateId == "" {
        fmt.Printf("No badge template ID found for track %s", webhook.TrackId)
        w.WriteHeader(http.StatusBadRequest)
        return
    }

    // Issue badge through Credly
    _, badgeErr := credlyClient.IssueBadge(templateId, user.Email, user.FirstName, user.LastName)
    // Check if the badge has already been issued
    if badgeErr != nil {
        if strings.Contains(badgeErr.Error(), credly.ErrBadgeAlreadyIssued) {
            fmt.Printf("Badge already issued for %s", user.Email)
            w.WriteHeader(http.StatusConflict)
            return
        }
        fmt.Printf("Failed to issue badge: %v", badgeErr)
        w.WriteHeader(http.StatusInternalServerError)
        return
    }

    w.WriteHeader(http.StatusOK)
    return
}

This function does the following:

Check if the event is of type track.completed, exit otherwise.
Instantiate Instruqt and Credly clients using environment variables for the tokens.
Retrieve user information from the Instruqt API. This requires to ensure that Instruqt has that information. See the first blog post to find how to do that with a proxy.
Get track information from Instruqt. We will use set a badge: special tag on the track to store the Credly badge ID to issue.
Parse track tags to find the badge template ID.
Issue the badge using the Credly library.

Setting Up the Webhook on Instruqt

To enable Instruqt to call your webhook, navigate to the Instruqt UI, go to Settings -> Webhooks, and click "Add Endpoint" to set up a new webhook that points to your Google Cloud Function URL.

Select track.completed in the list of events to fire up this endpoint.

Since we'll be hosting the function on Google Cloud Functions, the URL will be in the form https://<zone>-<project>.cloudfunctions.net/<name>. For example, if your function is called instruqt-webhook and is deployed in the labs GCP project in the europe-west1 zone, then the URL will be https://europe-west1-labs.cloudfunctions.net/instruqt-webhook. If in doubt, put a fake URL and you can modify it later.

Create "Create", then locate the "Signing secret" field to the right side of the panel and copy its value.

Export it in your terminal as the INSTRUQT_WEBHOOK_SECRET value:

export INSTRUQT_WEBHHOOK_SECRET=whsec_v/somevalueCopiedFromUi

Then use it to create a new GCP secret called instruqt-webhook-secret:

echo -n "$INSTRUQT_WEBHHOOK_SECRET" | gcloud secrets create instruqt-webhook-secret --data-file=-

Give it the proper permissions to be usable in your function (see first blog post for details):

PROJECT_NUMBER=$(gcloud projects describe $(gcloud config get-value project) --format="value(projectNumber)")
gcloud secrets add-iam-policy-binding instruqt-webhook-secret \
    --member="serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com" \
    --role="roles/secretmanager.secretAccessor"

Also create a secret for your Credly token:

export CREDLY_TOKEN=yourCredlyToken
echo -n "$CREDLY_TOKEN" | gcloud secrets create credly-token --data-file=-
gcloud secrets add-iam-policy-binding credly-token \
    --member="serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com" \
    --role="roles/secretmanager.secretAccessor"

Testing the Code

Let's check that this function builds and runs fine.

First, update your go.mod and go.sum files with:

go get ./...
go mod tidy

Now, run the function:

FUNCTION_TARGET=InstruqtWebhookCatch go run ./cmd/main.go

The function should compile and run fine. You can try sending queries to it on localhost:8080:

curl -i localhost:8080

Expect to get an error since the Svix webhook authentication is not set up properly in the payload:

HTTP/1.1 405 Method Not Allowed
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Tue, 08 Oct 2024 13:20:47 GMT
Content-Length: 23

Invalid request method

It would be possible to emulate this, but it's a bit complex, so let's just deploy to GCP now!

Alternative testing: using Docker

If you'd like to use Docker to test your function locally, you can create a Dockerfile in your current directory:

FROM golang:1.23

WORKDIR /app

COPY . .

RUN go build -o myapp ./cmd/main.go

ENV DEV=true
ENV PORT=8080

EXPOSE $PORT

CMD ["./myapp"]

Add a docker-compose.yaml file:

version: '3'
services:
  proxy:
    build: ./
    ports:
      - "8080:8080"
    environment:
      INSTRUQT_WEBHOOK_SECRET: ${INSTRUQT_WEBHOOK_SECRET}
      INSTRUQT_TOKEN: ${INSTRUQT_TOKEN}
      CREDLY_TOKEN: ${CREDLY_TOKEN}
      FUNCTION_TARGET: InstruqtWebhookCatch

Finally, build and launch your container:

docker-compose up --build

And you can send requests to localhost:8080 just the same as before!

Deploy the Function

You can then deploy the function (adapt the region if needed), giving it access to all three secret values:

gcloud functions deploy "instruqt-webhook" \
  --gen2 --runtime=go122 --region=europe-west1 --source=. \
  --entry-point="InstruqtWebhookCatch" --trigger-http --allow-unauthenticated \
  --set-secrets="INSTRUQT_WEBHOOK_SECRET=instruqt-webhook-secret:latest" \
  --set-secrets="INSTRUQT_TOKEN=instruqt-token:latest" \
  --set-secrets="CREDLY_TOKEN=credly-token:latest"

This will upload and build your project, and return the URL to access the function.

If necessary, update the URL in your Instruqt webhook configuration.

Testing

Now for the moment of truth: testing!

Create a badge on Credly. Publish it and copy its template ID.
Add a tag to the Instruqt track you want to associate the badge with. Name the tag badge:<template_ID>, replacing template_ID with the ID you just copied.
Publish the track.
Take the track and complete it!

You should get the badge in your email!

Further Considerations

User Information: Make sure you read the first blog post to understand how to send user information to Instruqt.
Make it worth it!: Getting badges is fun, but it's better if users deserve them. Consider adding exam steps to your tracks to make earning the badges a challenge.
Rate Limiting and Retries: Consider rate limiting incoming webhook requests to prevent abuse and adding retry logic to handle temporary failures when interacting with Credly.
Manage more Events: This webhook manager only manages track.completed events. You can extend it to do a lot more things with all the events provided by Instruqt! I typically like to capture lots of events to send them to Slack for better visibility.
Logs: Consider adding more logging (for example using the GCP logging library) to the code.

Streamlining Access to Embedded Instruqt Labs

Raphaël Pinson — Thu, 03 Oct 2024 11:58:29 +0000

How do you teach a very technical topic to prospects and customers? How do you make it a smooth ride?

At Isovalent, we're passionate about making the learning experience as seamless as possible for our users. Isovalent are the creators of Cilium, the de facto cloud networking platform for Kubernetes. While we love networking and security, we appreciate folks might find it a difficult topic. We thought we would make learning Kubernetes networking fun, so we make it a point to gamify the learning experience.
Instruqt provides a great platform to build hands-on labs that can be both technically advanced and engaging for users.

We also believe the user experience should be smooth and the processes fully automated.
Fortunately, a lot can be done by leveraging the Instruqt graphQL API.
To that purpose, we wrote our own instruqt-go library, which we've decided to open source. The library is designed to help developers automate and integrate with the Instruqt platform with ease.

isovalent / instruqt-go

A Go library for the Instruqt API

instruqt-go

Features

Manage Instruqt Teams and Challenges: Retrieve team information, challenges, and user progress.

Installation

To install the instruqt-go library, run:

go get github.com/isovalent/instruqt-go

Example Usage

package main
import (
    "github.com/isovalent/instruqt-go/instruqt"
    "cloud.google.com/go/logging"
)

func main() {
    // Initialize the Instruqt client
    client := instruqt.NewClient("your-api-token", "your-team-slug")

    // Get all tracks
    tracks, err := client.GetTracks()

    // Add context to calls
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()
    
    clientWithTimeout := client.WithContext(ctx)
    userInfo, err := clientWithTimeout.GetUserInfo("user-id")

    // Attach a logger
    logClient, err := logging

…

View on GitHub

One of the issues in publishing Instruqt labs is to link user information from Instruqt with that of your own database or CRM.
In this first post, we’ll guide you through building a proxy using instruqt-go that:

collects user identifiers (e.g., HubSpot tokens);
validates user identity;
redirects users to a lab with unique access tokens generated via the Instruqt API.

We will then publish the function on Google Cloud Functions.

Why a Proxy

There are various reasons to collect user information in labs:

It is useful to be able to generate badges (and we love badges) upon completion of the lab (more to come on that in a future post).
It allows to show users their progress through the labs so they know which ones to take (see for example the Cilium Labs Map).

How to Pass User Data

There are several methods to pass user data to Instruqt tracks.

Custom Parameters

Instruqt custom parameters are very useful to pass any kind of information when starting a track. These fields are simply added to the URL as query parameters, prefixed with icp_. These parameters can also be retrieved in Instruqt webhooks as well as through the Instruqt GraphQL API, making them practical to use.

Until recently, Instruqt encouraged track developers to pass user information (such as name, email, or token) using custom parameters.

However, there are a few downsides to using custom parameters:

They are not standardized, and Instruqt doesn't interpret them. This means user sessions will show as anonymous in Instruqt reports (and the unique user count might be wrong).
They are not encrypted by default. You can of course encrypt them for your own keys, but Instruqt will show you the encrypted value in play reports.
I have seen on multiple occasions custom parameters being lost when users restart a lab. I actually started my own cache database to counter this issue.

Invites

Instruqt invites allow to create a list of tracks and generate an invitation link that can be shared with users for easy access. Invites can be set to collect user data via a form.

This user data is then added to the user's details on Instruqt (user details are attached to user accounts, but are unique per Instruqt team).

This is extremely practical for workshops, but there's again a few limitations:

Using an invite to access all labs means that invite must contain all published labs.
Invites have their own landing page, so it wouldn't work with our Cilium Labs map or other kiosk approaches.

Note: Instruqt recently introduced landing pages, which is a form of invites with a way to tune the landing page, with the same advantages and limitations.

Third Party Forms

Recently, Instruqt added another way to pass user information, which combines the benefits of both previous methods.

The encrypted PII method allows to pass a pii_tpg query parameter to an embed URL. This means:

The data is encrypted, using a public key provided by Instruqt, so URLs don't contain readable user information.
Instruqt understands the pii_tpg data and has the private key to decrypt it. The information is used to fill in the user's details, just as if they had acceted an invite.
This is not linked to invites, so it can be used with any track.

We're going to use this new method in this example, as it is the most versatile today to pass information to Instruqt in a safe and reliable manner.

A Note on Embed Tokens

When you visit a track page on Instruqt, there is an option to embed the track.
This gives you a URL which contains a token unique to the track.

While it is perfectly valid to use that URL, it also means that whoever has access to this token can start the track whenever they want.

Instruqt has recently added an API call to generate one-time tokens for tracks, such that URLs using such tokens can only be used once.

The proxy we're coding will use one-time tokens, since we have access to the API and can easily generate them.

Creating the Proxy

Initial Steps

First, create a directory for your function:

mkdir instruqt-proxy

Move to this directory and initialize the Go environment:

# Replace example.com with the prefix of your choice
go mod init example.com/labs

Google Cloud Function Harnessing

For local testing, create a cmd directory:

mkdir cmd

Create a main.go file in that directory, with the following content:

package main

import (
    "log"
    "os"

    // Blank-import the function package so the init() runs
  // Adapt if you replaced example.com earlier
    _ "example.com/labs"

    "github.com/GoogleCloudPlatform/functions-framework-go/funcframework"
)

func main() {
    // Use PORT environment variable, or default to 8080.
    port := "8080"
    if envPort := os.Getenv("PORT"); envPort != "" {
        port = envPort
    }
    if err := funcframework.Start(port); err != nil {
        log.Fatalf("funcframework.Start: %v\n", err)
    }
}

Creating the Function

Back to the instruqt-proxy directory, create a proxy.go file and start by adding the init() function to it, along with the Go packages we will be using:

package labs

import (
    "fmt"
    "net/http"
    "net/url"
    "os"

    "github.com/GoogleCloudPlatform/functions-framework-go/functions"

    "github.com/isovalent/instruqt-go/instruqt"
)

func init() {
    functions.HTTP("InstruqtProxy", instruqtProxy)
}

This will allow Google Cloud Functions to call the instruqtProxy function when it is initialized.

Let's write that function:

const (
    // Replace team name with yours
    instruqtTeam = "isovalent"
)

func instruqtProxy(w http.ResponseWriter, r *http.Request) {
    instruqtToken := os.Getenv("INSTRUQT_TOKEN")

    if instruqtToken == "" {
        w.WriteHeader(http.StatusInternalServerError)
        return
    }

    instruqtClient := instruqt.NewClient(instruqtToken, instruqtTeam)

    // Get user from passed token
    utk := r.URL.Query().Get("utk")
    if utk == "" {
        w.WriteHeader(http.StatusUnauthorized)
        return
    }

    user, err := getUser(utk)
    if err != nil {
        w.WriteHeader(http.StatusUnauthorized)
        return
    }

    labSlug := r.URL.Query().Get("slug")

    url, err := getLabURL(instruqtClient, user, labSlug)
    if err != nil {
        w.WriteHeader(http.StatusNotFound)
        return
    }

    http.Redirect(w, r, url, http.StatusFound)
}

In this function, we:

get the Instruqt token from the INSTRUQT_TOKEN environment variable
initialize the Instruqt API client for the token and team
retrieve a utk parameter from the URL parameters in order to authenticate the user
get user information based on that UTK
get the lab slug from the URL parameters
retrieve the lab URL for the redirection
redirect the user using an http.Redirect function

Implement getLabURL()

The getLabURL function will generate the redirect URL for the lab based on user information, the requested lab slug, and dynamic information from the Instruqt API.

Let's write it:


const (
    // Replace with your sign-up page format
    labSignupPage = "https://isovalent.com/labs/%s"

    // Adapt to your values
    finishBtnText = "Try your next Lab!"
    finishBtnURL  = "https://labs-map.isovalent.com/map?lab=%s&showInfo=true"
)

func getLabURL(instruqtClient *instruqt.Client, u user, slug string) (string, error) {
    track, err := instruqtClient.GetTrackBySlug(slug)
    if err != nil {
        return "", err
    }

    // Unknown user
    if u.Email == "" {
        url := fmt.Sprintf(labSignupPage, slug)
        return url, nil
    }

    // Get one-time token
    token, err := instruqtClient.GenerateOneTimePlayToken(track.Id)
    if err != nil {
        return "", err
    }

    labURL, err := url.Parse(fmt.Sprintf("https://play.instruqt.com/embed/%s/tracks/%s", instruqtTeam, track.Slug))
    if err != nil {
        return "", err
    }

    // Prepare the fields to encrypt
    encryptedPII, err := instruqtClient.EncryptUserPII(u.FirstName, u.LastName, u.Email)
    if err != nil {
        return "", err
    }

    // Add params
    params := map[string]string{
        "token":             token,
        "pii_tpg":           encryptedPII,
        "show_challenges":   "true",
        "finish_btn_target": "_blank",
        "finish_btn_text":   finishBtnText,
        "finish_btn_url":    fmt.Sprintf(finishBtnURL, track.Slug),
    }

    q := labURL.Query()
    for key, value := range params {
        q.Set(key, value)
    }

    // Encode the parameters
    labURL.RawQuery = q.Encode()

    return labURL.String(), nil
}

First, note that we have defined some new constants that you can tune:

labSignupPage is the URL on your website where unauthenticated users will be redirected. It contains a variable for the lab slug.
finishBtnText is the text shown on the finish button of the lab.
finishBtnURL is the action of the button at the end of the lab. It also contains a variable for the lab slug.

Now let's explain the getLabURL() function steps:

Retrieve track information from the Instruqt API, error if it cannot be found.
If the user is unknown, redirect to sign-up page.
Generate a one-time token for the embedded track access.
Generate the redirect URL.
Encrypt user information using the PII key from the Instruqt API.
Add all parameters (one-time token, encrypted user information, finish button options) to the redirect URL.
Encode the URL.
Return the resulting URL.

The `getUser()` Function

The last piece missing in this proxy is the getUser() function. I can't help you much here, since this part is where you plug your own logic. You might be using a CRM like Hubspot to retrieve contact information from the UTK, or another database, it's up to you!

The code I'll show you here simply returns a sample user:

/*
 * This is where you add the logic to get user information from your CRM/database.
 */
type user struct {
    FirstName string
    LastName  string
    Email     string
}

func getUser(utk string) (u user, err error) {
    // Implement the logic to get your user information from UTK

    u = user{
        FirstName: "John",
        LastName:  "Doe",
        Email:     "john@doe.com",
    }

    return u, err
}

Testing the code

Now that we have our whole proxy.go function, let's test it!

First, update your go.mod and go.sum files with:

go get ./...
go mod tidy

In your Instruqt dashboard, go to "API keys" and get the value of your API key. Export it as a variable in your shell:

export INSTRUQT_TOKEN=<your_instruqt_token>

Next, launch the function on your local machine:

FUNCTION_TARGET=InstruqtProxy go run ./cmd/main.go

Finally, in another terminal, make test requests to localhost:8080 where your function will be running (you can pass a PORT environment variable above to change the port if necessary):

curl  -i "localhost:8080/?slug=cilium-getting-started&utk=someUtkValue"

Adapt to use a track slug that exists in your Instruqt organization. If the track exists, you should get a 302 response with the redirect URL containing a one-time token for access, as well as John Doe's information encrypted with the PII key, and a one-time token (starting with ott_)!

Alternative testing: using Docker

If you'd like to use Docker to test your function locally, you can create a Dockerfile in your current directory:

FROM golang:1.23

WORKDIR /app

COPY . .

RUN go build -o myapp ./cmd/main.go

ENV DEV=true
ENV PORT=8080

EXPOSE $PORT

CMD ["./myapp"]

Add a docker-compose.yaml file:

version: '3'
services:
  proxy:
    build: ./
    ports:
      - "8080:8080"
    environment:
      INSTRUQT_TOKEN: ${INSTRUQT_TOKEN}
      FUNCTION_TARGET: InstruqtProxy

Finally, build and launch your container:

docker-compose up --build

And you can send requests to localhost:8080 just the same as before!

Hosting the Proxy on Google Cloud Functions

In order to deploy to Google Cloud, first make sure you are logged in to your Google Cloud project:

gcloud auth application-default login

Create a Secret for the API Token

Next, let's create a new secret for the Instruqt token:

echo -n "$INSTRUQT_TOKEN" | gcloud secrets create instruqt-token --data-file=-

In order to adjust the permissions on this secret, you will need to get your project ID:

PROJECT_NUMBER=$(gcloud projects describe $(gcloud config get-value project) --format="value(projectNumber)")

Then, add the "Secret Manager Secret Accessor" role for the default Compute Engine service account for the project:

gcloud secrets add-iam-policy-binding instruqt-token \
    --member="serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com" \
    --role="roles/secretmanager.secretAccessor"

Your secret is now ready to be used by the function!

Deploy the Function

You can then deploy the function (adapt the region if needed):

gcloud functions deploy "instruqt-proxy" \
  --gen2 --runtime=go122 --region=europe-west1 --source=. \
  --entry-point="InstruqtProxy" --trigger-http --allow-unauthenticated \
  --set-secrets="INSTRUQT_TOKEN=instruqt-token:latest"

This will upload and build your project, and return the URL to access the function.

This URL will look something like https://europe-west1-<project>.cloudfunctions.net/instruqt-proxy.
You can then test the function using that URL instead of localhost:8080!

Further Considerations

This is a simplified approach to the lab proxy we use at Isovalent. There are things you might want to consider with this implementation:

If you have actual user (instead of marketing contact), switch to a proper authentication system (e.g. JWT) instead of UTK.
The current implementation will give access to any lab in your collection if you know its slug. You might want to filter them out (using for example track tags).
This implementation manages errors but is very basic in logging. We would recommend using Google Cloud logging to easily audit function runs.
You might want to pass extra information as custom parameters. For example, we like to pass some form of event or campaign ID. These can then be retrieved via the API as part or the Track structure.
If you're using a custom form and redirecting to the proxy, you might want to be sure your CRM/database has already gotten the user information. You can for example implement a retry logic in the proxy function.
Invite embed URLs contain the invite ID. If you want to support invites, the proxy could take an optional invite parameter and add it to the URL.

Using the Proxy

This proxy can typically be used to give access to authenticated users in a safe way, while preserving user information in Instruqt reports and making sure embed URLs are not reusable.

Here is an example of usage of this proxy:

Set up lab sign-up pages with the form system of your choice (e.g. using Hubspot forms).
Retrieve a user identifier from the page context (e.g. a Hubspot cookie token).
Redirect users to the proxy, passing the user identifier and lab slug as parameters.

This can allow you to build a series of public sign-up pages for your labs, similar to what we have built on the Isovalent website. It can also be used to build a Kiosk interface, or even a more creative landing page such as the Cilium Labs map, where clicks on the map redirect to the lab proxy.

By making a complex networking technology like Cilium fun with our labs, we have made it the experience for users less intimidating and more approachable. Using our proxy can help you provide a similar user experience to your prospects. Please get in touch if you have any questions.

Towards a Modular DevOps Stack

Raphaël Pinson — Wed, 23 Feb 2022 17:37:45 +0000

A year and a half ago, our infrastructure team at Camptocamp was faced with an increasingly problematic situation. We were provisioning more and more Kubernetes clusters, on different cloud providers. We used Terraform to deploy the infrastructure itself, and we had started to adopt Argo CD to deploy applications on top of the cluster.

We quickly ended up with many projects using similar logic, often borrowed from older projects, and most of these cluster were starting to use divergent code.

We thought it was time to put together a standard core in order to:

provision Kubernetes clusters;
deploy standard applications sets (monitoring, ingress controller, certificate management, etc.) on them;
provide an interface for developers to deploy their applications in a GitOps manner;
ensure all teams used similar approaches.

The DevOps Stack was born!

🌟 The Original Design

The original DevOps Stack design was monolithic, both for practical and technical reasons.

After all, we were trying to centralize best practices from many projects into a common core, so it made sense to put everything together behind a unified interface!

In addition to that, all the Kubernetes applications were created using an App of Apps pattern, because we had no ApplicationSets and no way to control Argo CD directly from Terraform.

As a result, the basic interface was very simple, for example:

module "cluster" {
  source = "git::https://github.com/camptocamp/devops-stack.git//modules/eks/aws?ref=v0.54.0"

  cluster_name = local.cluster_name
  vpc_id       = module.vpc.vpc_id

  worker_groups = [
    {
      instance_type        = "m5a.large"
      asg_desired_capacity = 2
      asg_max_size         = 3
    }
  ]

  base_domain     = "example.com"

  cognito_user_pool_id     = aws_cognito_user_pool.pool.id
  cognito_user_pool_domain = aws_cognito_user_pool_domain.pool_domain.domain
}

However, it could get very complex when application settings needed to be tuned!

With time, this architecture started being problematic for various reasons:

it was hard to deactivate or replace components (monitoring, ingress controller, etc.), making it hard to extend the core features;
default YAML values got quite mixed up and hard to understand;
as a result, overriding default values was unnecessarily complex;
adding new applications was done using extra_apps and extra_applicationsets parameters, which were monolithic and complex.

It was time to rethink the design.

🐙 Argo CD Provider

In order to escape the App of Apps pattern, we started looking into using a Terraform provider to control Argo CD resources. After various contributions, the provider was ready for us to start using in the DevOps Stack.

🗺 The Plan for Modularization

Using the Argo CD provider allowed us to split each component into a separate module. In a similar way to DevOps Stack v0, each of these modules would provide Terraform code to set up the component, optionally with:

cloud resources required to set up the component;
Helm charts to deploy the application using Argo CD.

💡 A Full Example

As a result, the user interface is much more verbose. The previous example would thus become:

module "cluster" {
  source = "git::https://github.com/camptocamp/devops-stack.git//modules/eks/aws?ref=v1.0.0"

  cluster_name = var.cluster_name
  base_domain = "demo.camptocamp.com"
  vpc_id       = module.vpc.vpc_id

  cluster_endpoint_public_access_cidrs = flatten([
    formatlist("%s/32", module.vpc.nat_public_ips),
    "0.0.0.0/0",
  ])

  worker_groups = [
    {
      instance_type        = "m5a.large"
      asg_desired_capacity = 2
      asg_max_size         = 3
      root_volume_type     = "gp2"
    },
  ]
}

provider "argocd" {
  server_addr = "127.0.0.1:8080"
  auth_token  = module.cluster.argocd_auth_token
  insecure = true
  plain_text = true
  port_forward = true
  port_forward_with_namespace = module.cluster.argocd_namespace

  kubernetes {
    host                   = module.cluster.kubernetes_host
    cluster_ca_certificate = module.cluster.kubernetes_cluster_ca_certificate
    token = module.cluster.kubernetes_token
  }
}

module "ingress" {
  source = "git::https://github.com/camptocamp/devops-stack-module-traefik.git//modules/eks"

  cluster_name     = var.cluster_name
  argocd_namespace = module.cluster.argocd_namespace
  base_domain      = module.cluster.base_domain
}

module "oidc" {
  source = "git::https://github.com/camptocamp/devops-stack-module-oidc-aws-cognito.git//modules"

  cluster_name     = var.cluster_name
  argocd_namespace = module.cluster.argocd_namespace
  base_domain    = module.cluster.base_domain

  cognito_user_pool_id     = aws_cognito_user_pool.pool.id
  cognito_user_pool_domain = aws_cognito_user_pool_domain.pool_domain.domain
}

module "monitoring" {
  source = "git::https://github.com/camptocamp/devops-stack-module-kube-prometheus-stack.git//modules"

  cluster_name     = var.cluster_name
  oidc             = module.oidc.oidc
  argocd_namespace = module.cluster.argocd_namespace
  base_domain    = module.cluster.base_domain
  cluster_issuer = "letsencrypt-prod"
  metrics_archives = {}

  depends_on = [ module.oidc ]
}

module "loki-stack" {
  source = "git::https://github.com/camptocamp/devops-stack-module-loki-stack.git//modules/eks"

  cluster_name     = var.cluster_name
  argocd_namespace = module.cluster.argocd_namespace
  base_domain      = module.cluster.base_domain

  cluster_oidc_issuer_url = module.cluster.cluster_oidc_issuer_url

  depends_on = [ module.monitoring ]
}

module "cert-manager" {
  source = "git::https://github.com/camptocamp/devops-stack-module-cert-manager.git//modules/eks"

  cluster_name     = var.cluster_name
  argocd_namespace = module.cluster.argocd_namespace
  base_domain      = module.cluster.base_domain

  cluster_oidc_issuer_url = module.cluster.cluster_oidc_issuer_url

  depends_on = [ module.monitoring ]
}

module "argocd" {
  source = "git::https://github.com/camptocamp/devops-stack-module-argocd.git//modules"

  cluster_name   = var.cluster_name
  oidc           = module.oidc.oidc
  argocd         = {
    namespace = module.cluster.argocd_namespace
    server_secrhttps://kubernetes.slack.com/archives/C01SQ1TMBSTetkey = module.cluster.argocd_server_secretkey
    accounts_pipeline_tokens = module.cluster.argocd_accounts_pipeline_tokens
    server_admin_password = module.cluster.argocd_server_admin_password
    domain = module.cluster.argocd_domain
  }
  base_domain    = module.cluster.base_domain
  cluster_issuer = "letsencrypt-prod"

  depends_on = [ module.cert-manager, module.monitoring ]
}

☸ The Cluster Module

As you can see, the main cluster module —which used to do all the work— is now only responsible for deploying the Kubernetes cluster and a basic Argo CD set up (in bootstrap mode).

🐙 The Argo CD Provider

We then set up the Argo CD provider using outputs from the cluster module:

provider "argocd" {
  server_addr = "127.0.0.1:8080"
  auth_token  = module.cluster.argocd_auth_token
  insecure = true
  plain_text = true
  port_forward = true
  port_forward_with_namespace = module.cluster.argocd_namespace

  kubernetes {
    host                   = module.cluster.kubernetes_host
    cluster_ca_certificate = module.cluster.kubernetes_cluster_ca_certificate
    token = module.cluster.kubernetes_token
  }
}

This provider is used in all (or most at least) other modules in order to deploy Argo CD applications, without going through a monolithic/centralized App of Apps.

🧩 Component Modules

Each module provides a single component, using Terraform and the Argo CD provider (optionally).

There are common interfaces passed as variables between modules. For example, the oidc variable can be provided as an output from various modules: Keycloak, AWS Cognito, etc. This oidc variable can then be passed to other component modules to configure the component's authentication layer.

In a similar fashion, the ingress module, which was so far only using Traefik, can now be replaced by another Ingress Controller implementation.

🐙 The Argo CD Module

The Argo CD module is a special one. A special entrypoint (called boostrap) is used in the cluster module to set up a basic Argo CD using Helm.

The user is then responsible for instantiating the Argo CD module a second time at the end of the stack, in order for Argo CD to manage itself and configure itself properly —including monitoring and authentication, which cannot be configured before the corresponding components are deployed.

🥾 The Bootstrap Pattern

Eventually, other modules might adopt the bootstrap pattern in order to solve chicken-and-egg dependencies.

For example, cert-manager requires Prometheus Operator CRDs in order to be monitored. However, the Kube Prometheus Stack might require valid certificates, thus depending on cert-manager being deployed. This could be solved by deploying a basic cert-manager instance (in a bootstrap endpoint), and finalizing the deploying at the end of the stack.

🚀 To Infinity

The modular DevOps Stack design is the current target for release 1.0.0.

While this refactoring is still in beta state, it can be tested by using the v1 branch of the project. You can also find examples in the tests directory (though not all distributions are ported yet).

Feedback is welcome, and you can contact us on the #camptocamp-devops-stack channel of the Kubernetes Slack.

A 15-year Puppet Journey

Raphaël Pinson — Sun, 06 Feb 2022 22:59:02 +0000

Note: this is a personal blog post. It does not concern Camptocamp's partnership with Puppet Inc., which remains unchanged.

In 2006, I landed my first gig as a Systems Administrator. One of my main roles was taking care of a Cfengine server and repository for about 4000 machines, and finishing its migration from Cfengine 1 to Cfengine 2.

Like many people in this position at that time, discovering Puppet was akin to a revelation. Leaving aside its slowness (in comparison to Cfengine), its DSL, file templates, and the extensibility of the Resource Abstraction Layer were nothing short of a little revolution.

Then Augeas was presented to me, and I enjoyed the concept so much I got involved in the project and started writing many lenses.

I joined Camptocamp in 2012 in large part because of the role the company played in the Puppet community. Together with my colleague Mickaël, we took to standardising and modernising our Puppet stack and modules, and got deeply involved in the community, writing plugins (puppet-lint plugins, facterdb/rspec-puppet-facter) and tools (prometheus-puppetdb-sd, puppetfile-updater, puppet-ghostbuster, etc.).

Over the years, I've had the pleasure of teaching Puppet training courses at all levels and consulting on all sorts of layers (modules, Ruby plugins, TDD, etc.) and stacks (Puppet Enterprise, Foreman, Docker-based, and more…).

In the last couple of years though, my work has mostly revolved around containers and Kubernetes. I have kept maintaining Puppet- and Augeas-related code, but often without using these projects myself for production needs.

For that reason, I started donating such projects to Voxpupuli, as I believe they will receive better care than I can currently give them.

Truth be told, I've delayed all this for months —maybe years— because the Puppet community is awesome and it's always been a pleasure to contribute to it. I've even gotten back quite a bit last year, reviving puppet-catalog-diff and participating in various Puppet Camps.

It's been fun, but I need to focus on other projects now, and it's probably better if things are set in a clear manner.

So farewell Puppet community, keep being an awesome and welcoming place, & thanks for all the 🐟!

How to allow dynamic Terraform Provider Configuration

Raphaël Pinson — Tue, 11 May 2021 11:47:57 +0000

Terraform relies heavily on the concept of providers, a base brick which consists of Go plugins enabling the communication with an API.

Each provider gives access to one or more resource types, and these resources then manage objects on the target API.

Most of the time, a provider's configuration is static, e.g.

provider "aws" {
  region = "us-east-1"
}

However, in some cases, it is useful to configure a provider dynamically, using the attribute values from other resources as input for the provider's configuration.

I'll use the example of the Argo CD provider. In a single Terraform run, we would like to:

install a Kubernetes cluster (using a DevOps Stack K3s Terraform module)
install Argo CD on the the cluster using the Helm provider
instantiate Argo CD resources (projects, applications, etc.) on this new Argo CD server.

Our code will look like this:

# Install Kubernetes & Argo CD using a local module
# (from https://devops-stack.io)
module "cluster" {
  source = "git::https://github.com/camptocamp/devops-stack.git//modules/k3s/docker?ref=master"

  cluster_name = "default"
  node_count   = 1
}

# /!\ Setup the Argo CD provider dynamically
# based on the cluster module's output
provider "argocd" {
  server_addr = module.cluster.argocd_server
  auth_token  = module.cluster.argocd_auth_token
  insecure    = true
  grpc_web    = true
}

# Deploy an Argo CD resource using the provider
resource "argocd_project" "demo_app" {
  metadata {
    name      = "demo-app"
    namespace = "argocd"
  }

  spec {
    description  = "Demo application project"
    source_repos = ["*"]

    destination {
      server    = "https://kubernetes.default.svc"
      namespace = "default"
    }

    orphaned_resources {
      warn = true
    }
  }

  depends_on = [ module.cluster ]
}

This requires to configure Argo CD dynamically, using the output of the Kubernetes cluster's resources.

Provider Initialization

Providers are initialized early in a Terraform run, as their initialization is required to compute the graph which defines in which order the resources are applied.

This means it is actually not possible to make a provider initialize after a secondary resource is created.

Officially, the story stops here, and Terraform has a bug report to track the feature allowing to dynamically configure providers.

So… it's game over then? 🎮 👾
Not really!

Leveraging Pointers

When a provider is configured in Terraform, it triggers a configuration function:

func Provider() *schema.Provider {
    return &schema.Provider{
      ConfigureFunc: func(d *schema.ResourceData) (interface{}, error) {
        // Create someObject
        return someObject, nil
      }
    }
}

This ConfigureFunc method is usually used to create a static client for the target API. In the Argo CD provider for example, it returns a ServerInterface structure, with pointers to several clients, instantiated from the provider parameters:

type ServerInterface struct {                                                   
    ApiClient            *apiclient.Client                                      
    ApplicationClient    *application.ApplicationServiceClient                  
    ClusterClient        *cluster.ClusterServiceClient                          
    ProjectClient        *project.ProjectServiceClient                          
    RepositoryClient     *repository.RepositoryServiceClient                    
    RepoCredsClient      *repocreds.RepoCredsServiceClient                      
    ServerVersion        *semver.Version                                        
    ServerVersionMessage *version.VersionMessage                                                                                                              
}

The return statement from the ConfigureFunc eventually looks like this:

return ServerInterface{                                                             
    &apiClient,                                                                     
    &applicationClient,                                                             
    &clusterClient,                                                                 
    &projectClient,                                                                 
    &repositoryClient,                                                              
    &repoCredsClient,                                                               
    serverVersion,                                                                  
    serverVersionMessage}, err

Let's add a new field to the ServerInterface to store the pointer to the provider's ResourceData object, which gives access to the provider's parameters:

type ServerInterface struct {                                                       
    ApiClient            *apiclient.Client                                          
    ApplicationClient    *application.ApplicationServiceClient                      
    ClusterClient        *cluster.ClusterServiceClient                              
    ProjectClient        *project.ProjectServiceClient                              
    RepositoryClient     *repository.RepositoryServiceClient                        
    RepoCredsClient      *repocreds.RepoCredsServiceClient                          
    ServerVersion        *semver.Version                                            
    ServerVersionMessage *version.VersionMessage                                    
    ProviderData         *schema.ResourceData                                       
}

Now in the ConfigureFunc, we'll instantiate the ServerInterface, providing only the ProviderData pointer. The first resource that needs to use the provider will then instantiate the clients, when the provider parameters are available. We'll need the ConfigureFunc method to return a pointer to a ServerInterface, so we can later cache the clients and avoid recreating them for every resource:

ConfigureFunc: func(d *schema.ResourceData) (interface{}, error) {                  
    server := ServerInterface{ProviderData: d}                                      
    return &server, nil                                                             
},

Initialize the Clients

Now we need to actually initialize the clients in each resource.

Each resource method gets the interface returned by the ConfigureFunc function as an empty interface parameter, usually called meta:

func resourceArgoCDProjectCreate(ctx context.Context, d *schema.ResourceData, meta interface{}) diag.Diagnostics {

These methods currently simply cast the meta parameter as a ServerInterface structure and use the pre-initialized clients:

server := meta.(ServerInterface)

We now need to cast meta as a pointer to a ServerInterface structure instead (since we'll need to modify the clients from within the resources), and initialize the clients:

server := meta.(*ServerInterface)                                                   
if err := server.initClients(); err != nil {                                        
    return []diag.Diagnostic{                                                       
        diag.Diagnostic{                                                            
            Severity: diag.Error,                                                   
            Summary:  fmt.Sprintf("Failed to init clients"),                        
            Detail:   err.Error(),                                                  
        },                                                                          
    }                                                                           
}

The initClients() method of the ServerInterface structure will be called, allowing to set up the clients using the current provider parameters.

Client Pool Caching

In the ServerInterface#initClients() method, we want to make sure we reuse existing clients. This is rather simple, since each client is stored as a pointer in the structure, so it defaults to nil:

func (p *ServerInterface) initClients() error {                                 
    d := p.ProviderData                                                         

    if p.ApiClient == nil {                                                     
        apiClient, err := initApiClient(d)                                      
        if err != nil {                                                         
            return err                                                          
        }                                                                       
        p.ApiClient = &apiClient                                                
    }

    // etc for all clients

    return nil
}

Conclusion

That's it, we're done. With these modifications, terraform plan now works. The resources get applied in the proper order, and the outputs from the cluster module get properly passed as configuration to the Argo CD clients.

March Cloud Native Romandie Meetup

Raphaël Pinson — Thu, 01 Apr 2021 13:44:49 +0000

Last week, we organized our last Cloud Native Romandie Meetup. Due to the current situation, this was an online event like the previous occurrences.

The meetup was recorded and can be viewed again on YouTube.

Subjects

For this edition, we had presentations by CloudBees, Exoscale, and Camptocamp.

CloudBees CI

Frédéric Gibelin from CloudBees presented CloudBees CI, a solution that helps users scale their Jenkins Enterprise platform in the Cloud.

See the slides

Exoscale SKS

Next Pierre-Yves Ritschard and Mathieu Corbin from Exoscale presented SKS, a Kubernetes as-a-Service, operating the user’s cluster on their behalf and taking care of the underlying infrastructure.

Camptocamp DevOps Stack

To finish, Raphaël Pinson presented Camptocamp's DevOps Stack project, a framework to deploy a standardized Kubernetes platform and its ecosystem, using a GitOps approach.

Future Meetups

Don’t miss any more and join the Cloud Native Romandie meetup group. This way, you can be part of a local community and stay up-to-date on different Cloud Native technologies.

We look forward to meeting and exchanging with you during our next virtual meetup scheduled for Thursday, June 17th.

Also, if you are keen to present a technology or you would like to see more of a technology, please let us know, we would be happy to support your interest. If it interests you, it also interests the community.

Immutability & loose coupling: a match made in heaven

Raphaël Pinson — Thu, 18 Mar 2021 07:31:29 +0000

When it comes to infrastructure and deployment automation, two opposite approaches share the podium: mutable vs immutable management.

Mutable Systems

Mutable systems usually have a long life cycle, typically in the order
of weeks to years. As their requirements change (new files,
configurations, users, packages, etc.), the systems are modified to
match a new target state. When left unmanaged, mutable systems tend to
drift away from their target state, in a divergent dynamic.

Automating mutable systems is often referred to as Configuration Management, and leverages tools such as Cfengine, Puppet, Chef, or Ansible. This tooling uses principles based on the concepts of target state, idempotence, and somewhat related to Mark Burgess’ Promise Theory. Configuration Management aims to make the system convergent, by running a tool on a regular basis, in order to resynchronize the system with its target state. Some of these tools (e.g. mgmt) also attempt to reach congruence by adopting a reactive approach, triggering corrective actions on events.

Immutable Systems

In an immutable system, any change requires a new deployment. Whether it be a change in configuration, new files, or new users, immutability demands that the system be destroyed and rebuilt from scratch.

An immutable approach can greatly simplify deployments. Avoiding convergence, immutable systems rely on artifacts that are built once, and can be deployed multiple times. These artifacts can increase trust in the ability to rebuild the system from scratch if necessary. It can also ease scalability, since the artifact can be precisely duplicated.

However, immutability comes with a high cost: in order to be done properly, it must be strict. Any change to the state involves a complete replacement of the artifact. A lack of abiding to that rule results in a convergent system (at best), which cannot be managed in an immutable manner.

Mutable vs Immutable

If immutable systems are easier to maintain, what are the reasons for not using them? The system’s complexity is probably the main justification. A traditional mutable system features multiple layers (Operating System, Middleware, Applications) that are usually strongly coupled. For example, the flavor and version of the Operating System define which version of a Middleware (e.g. Tomcat, Apache) is available for installation. In turn, the Middleware version defines which libraries are available for the Application. On most Unix-based systems, shared libraries are at the root of strong links between software versions, based on the underlying ABIs required to run them.

If such strongly coupled systems are to be managed in an immutable manner, then the whole system is the immutable artifact. In the majority of organizations, this implies managing tens to thousands of artifacts, and rebuilding them from scratch on a regular basis. Such complexity is too much of a cost.

Enters decoupling technologies: Over the years, new technologies have surfaced, which allow to decouple system components and ease their management in an immutable manner.

Virtualization and IaaS

With the rise of virtualization in the early years of the 21st century, it became easier to decouple the hardware from the Operating System. You could now size virtual machines as precisely as desired, in terms of CPU, memory, or disk, without adding or replacing any physical device.

This unlocked access to a first level of automated immutability, using Virtual Machine images as the immutable artifact. Image generators (such as Hashicorp Packer) appeared, easing the generation of VM images.

Provided the whole target state —including the OS, middleware and application itself— is built into the VM image, an immutable workflow can be used to manage it. In this case, whenever a change is required, the whole image needs to be rebuilt and redeployed to new VMs.

Golden Images are a common approach to divergent templating

For a time, this deployment could not be easily automated, as physical nodes still needed to be manually picked before deploying the VM images. Infrastructure as a Service (IaaS), often referred to as “Cloud”, changed that.

IaaS provided APIs on top of virtualization, and the IaaS system (e.g. AWS EC2, OpenStack) would often pick the physical hypervisor node itself, making it possible to fully automate the deployment of new artifacts (such as VM images).

One important problem remained to achieve immutable infrastructure in most situations: the complexity of the artifact itself.

Containers and Orchestrators

When setting up applications on existing systems, several issues often arise, among which are: packaging, configuration, and dependencies.

For years, developers and systems engineers tried to solve the problem of application packaging and deployment using all kinds of package managers, from deb/rpm to homemade systems. This often failed, because these packages didn’t allow for multiple instantiation of the application, were not easy to configure, and were too tightly coupled to the rest of the system.

Docker containers provided a unified way of packaging applications, in the form of OCI images, a rather unified way of configuring them (using environment variables or mounted files, in the 12 factor app fashion).

But mainly, containers provided an abstraction level, a decoupling from the system. With containers, it doesn’t matter anymore which OS is running the container engine. Developers can now choose to run any version of Tomcat or Apache, on any node with a container engine. As a corollary, they can also run any combination of Middleware and Applications, regardless of the libraries provided on the underlying system.

Furthermore, containers were made to be managed in an immutable manner, using OCI images as immutable artifacts. Every time a container needs to be modified, it requires the creation of a new container from a new image.

The benefits are huge. With this decoupling of the Operating System from the Middleware and Applications, the monolithic immutable artifact that was previously managed as a VM image can now be broken down into many pieces: the application is now an immutable artifact, and so are the middleware components as well.

Even better: since all the components running on the machines are now immutable, the machines themselves have now become totally neutral; all they require is a container engine in order to run containers.

One thing can still make a node a snowflake: manually deployed containers, using tools such as Docker or Docker-compose.

What IaaS did for VMs, Container orchestrators now do for containers: they provide an API to orchestrate the dynamic deployment of containers on a cluster of nodes.

Container Orchestration is to Containers what IaaS is to Virtual Machines.

The nodes are thus totally neutral now. As a result, their templates are greatly simplified, and they can now easily be managed in an immutable way. This new paradigm opened the way to new Operating Systems specialized for container orchestration, such as CoreOS or RancherOS, whose life cycles are meant to be managed with an immutable workflow.

Immutability & Convergence

Now that we have a full Immutable System, does it solve the problem of convergence?

Immutable artifacts in themselves are not supposed to evolve, but their target state does evolve. In this regard, the situation with containers is similar to that of Golden Images for Virtual Machines: though the artifact is immutable, it can easily be used as a template leading to an unmaintained, divergent system.

There is thus still a need for convergence tools in the container world. However, this is not because the objects themselves drift from their target state. Rather, the target state evolves while the objects —supposedly— are stuck in their original state.

Where Configuration Management tools ensure a convergence of states for mutable systems such as VMs, packaging tools such as Helm and Helmfile can be used to periodically re-synchronize containers and other immutable objects with their target state.

Finally, congruence is also achievable, with tools such as Argo CD. Argo CD not only deploys immutable objects to a Kubernetes cluster, but also keeps them synchronized with their last known target state, ensuring a continuous management.

Conclusion

Containers and Container Orchestrators enable fully immutable workflows for infrastructure, middleware and applications alike:

Nodes can be managed as virtual machines, with immutable VM images deployed dynamically using IaaS;
Middleware and applications can be managed as containers, with immutable OCI images deployed dynamically using Container Orchestrators.

Are immutable systems always the best answer? As we’ve seen, the cost in artifact management and orchestration is far from negligible. Due to the transient nature that comes with their immutability, containers are better suited for stateless applications that easily scale on clusters of neutral nodes. For this reason, highly stateful applications with long life cycles, such as databases, are still better maintained as mutable systems most of the time.

Choosing an immutable vs mutable architecture depends a lot on an organization’s software architecture and culture, and is not a light choice to make. Is immutable infrastructure the solution to your automation problems? Contact us, we can help you find out!

Open Source, Standards, and Technical Debt

Raphaël Pinson — Wed, 03 Feb 2021 11:10:41 +0000

Twenty years ago, Camptocamp was a pioneer company in Open Source adoption. Nowadays, Open Source has become mainstream and the vast majority of the industry agrees on the many benefits of its practices. In fact, the Open Source model has become a de facto standard in some fields such as Web Frontend development.

Many companies make an increasing use of Open Source software in their infrastructure and development stacks, and there are countless proven reasons for doing so, such as standard formats or security by openness, to name just a couple.

In spite of these benefits, companies openly contributing —let alone Open Sourcing their own projects— are still somehow not very common, and most firms think of Open Source purely as a consumer’s benefit.

So why should you contribute to Open Source software?

For years, I used to think the best argument in favor of contributing to existing projects was maintenance and compatibility. If I fork a project and add functionality to it, there is a risk that my changes will become increasingly hard to maintain as time goes by. If the core developers of the program are aware of my changes and actively intend to go along with them, this risk will be greatly reduced.

So contributing my changes ensures they will stay compatible with the base code as time goes by. There might even be improvements to my code if more people encounter a similar need in the future, and decide to build on top of my changes.

Today, however, I believe the example I have just given is a specific case of a more general rule, which encompasses more pragmatic reasons to contribute code as Open Source. This more general context is linked to the concept of Technical Debt: the idea that technical decisions imply a hidden cost (a “debt”) that will have to be paid in the future in order to catch up with state-of-the-art technology.

So how do I minimize the debt?

Minimizing technical debt is a vast —and at times conflicting— subject. However, I think it is safe to assume that one way to reduce its risk is to stick to standards. The closer a project sticks to industry standards, the less likely it will have to be ported to another technological stack in the foreseeable future.

What if the standards don’t meet my needs?

When faced with a missing feature, most people’s reflex might be to start building a specific component to meet their use case. In the words of Strategy Theorist Simon Wardley, they’ll be shifting this component to the Genesis stage, making it more unpredictable —or even erratic—, less standard, and thus more prone to building up technical debt in time.

There is another way though. If my need is not met, and it is in fact a valid need (which is a very important question to ask in the first place), then other people might have this need in the future. When they do, someone, somewhere, will create a new standard for this need. When this new standard becomes enforced, then will my specific component’s debt become obvious.

So what if, instead of building a specific component to make up for the lack of standard, I set the new standard myself? Open Source lets you do just that! It gives you the opportunity to be the first one providing an open implementation to a generic need, and the chance to make it into the new standard. If that new standard catches on, you have not only solved your problem, but you also haven’t accumulated technical debt. In fact, you’re ahead of the other users, because you set the new standard.

Wait, we’re no FAANG!

Obviously, the majority of organizations can't afford to have engineers focusing on IETF RFCs or moving ISO standards to fit our needs.

However, a standard doesn't have to be that complicated. Let’s say I use this popular CLI tool, but I need to specify an option which doesn’t exist yet. I could hack something around the generation of its configuration file to produce the options I need. Or I could patch that tool and add a new flag for my needs, and contribute that change back to the project. Chances are, if I need this option, some else does too.

Now, every time someone has the need for that option, they’ll be using my new flag. I’ve contributed a new standard, and I haven’t made any technical debt on my side.

It’s not the size of the steps that matters, it’s really the direction in which you take them.

Where do I start?

Open Source is not just a philosophy. It encompasses licensing issues, technology standards, culture, and much more.

At Camptocamp, we’ve been committed to the Open Source approach for years.

This means we have a habit of solving problems in generic terms and building new standards.

It also means we have contacts in many Open Source communities, which allow us to brainstorm ideas and quickly contribute to projects, ensuring a fast feedback loop on our work.

When we implement Open Source software for our clients, we actively seek to limit technological debt. Because we believe in a world of standards, we don’t want our clients to feel entirely stuck with a technological stack in the future. Or even with us!

A Simple Auth Proxy for EKS

Raphaël Pinson — Wed, 11 Nov 2020 16:10:48 +0000

AWS EKS is a great option for a hosted Kubernetes cluster.

It is in particular easy to use for demos and training sessions.

However, EKS authentication is based off AWS IAM, which means users need an AWS account. Authenticating to EKS typically involves calling the aws eks get-token command in your .kube/config so as to retrieve an authentication token.

As we were setting up EKS for Kubernetes training, we needed a simple way for users without an AWS account to access the cluster, so we created a basic proxy service for the EKS get-token action.

camptocamp / aws-iam-authenticator-proxy

A simple HTTP proxy to share AWS IAM authentication

aws-iam-authenticator HTTP Proxy

Amazon Services require valid accounts to be used. This proxy allows external users to access an AWS EKS cluster without requiring access to AWS credentials.

Disclaimer: the proxy does not implement any form of authentication. You are responsible for implementing whatever security measure you wish to enforce in front of it.

Example usage

In order to give access to an AWS EKS cluster without distribution credentials you can start the proxy with the necessary credentials as well as the cluster ID. For example, using Docker:

$ docker run --rm -p 8080:8080 \
             -e AWS_ACCESS_KEY_ID=<AWS_ACCESS_KEY_ID> \
             -e AWS_SECRET_ACCESS_KEY=<AWS_SECRET_ACCESS_KEY> \
             -e EKS_CLUSTER_ID=<EKS_CLUSTER_ID> \
             -e PSK="mysecretstring" \
    camptocamp/aws-iam-authenticator-proxy:latest

You should then be able to retrieve authentication tokens for your user at http://localhost:8080.

If a PSK is passed, you will need to pass its value in the…

View on GitHub

Deploying with Docker

The proxy can be deployed using Docker, with AWS credentials, e.g.:

docker run --rm -p 8080:8080 \
             -e AWS_ACCESS_KEY_ID=<AWS_ACCESS_KEY_ID> \
             -e AWS_SECRET_ACCESS_KEY=<AWS_SECRET_ACCESS_KEY> \
             -e EKS_CLUSTER_ID=<EKS_CLUSTER_ID> \
             -e PSK="mysecretstring" \
    camptocamp/aws-iam-authenticator-proxy:latest

The rights on the cluster will depend on the user you chose to create the access key.

The PSK is optional, and allows to secure the proxy a little bit.

Once the proxy is started, you can access it at http://localhost:8080?psk=mysecretstring, so you can simply set your ~/.kube/config to use curl instead of aws:

users:
- name: <cluster_name>
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: curl
      args:
        - -s
        - "http://<your_ip>:8080/?psk=mysecretstring"

Deploying in EKS

Since you've got an EKS cluster in the first place, you might as well deploy the proxy in it.

The repository provides a Helm chart for that, in the k8s directory of the GitHub project.

You can simply instantiate the chart with the following values:

eks_cluster_id: "<EKS_CLUSTER_ID>"
psk: "mysecretstring"
aws:
  access_key_id: "<AWS_ACCESS_KEY_ID>"
  secret_access_key: "<AWS_SECRET_ACCESS_KEY>"

The AWS credentials will be stored in a Kubernetes secret and passed to the container.

Using

However, since we're in AWS, we can also use IAM roles for service accounts and bypass the access keys altogether. This is a much cleaner approach.

Here's how to do it, using Terraform to create the role and deploy the proxy.

First, create a role linked to OIDC:

module "iam_assumable_role_aws_iam_authenticator_proxy" {
  source                        = "terraform-aws-modules/iam/aws//modules/iam-assumable-role-with-oidc"
  version                       = "3.3.0"
  create_role                   = true
  number_of_role_policy_arns    = 0
  role_name                     = "aws-iam-authenticator-proxy"
  provider_url                  = replace(module.cluster.cluster_oidc_issuer_url, "https://", "")
  oidc_fully_qualified_subjects = ["system:serviceaccount:yournamespace:aws-iam-authenticator-proxy"]
}

replacing yournamespace with the Kubernetes namespace where you will be deploying the proxy.

Now we can configure the cluster to use map that role to the Kubernetes role we want (e.g. system::masters to make it cluster admin).

We'll create a random PSK and generate the Kubeconfig file to use curl with the proxy:

data "aws_vpc" "this" {
  id = var.vpc_id
}

data "aws_subnet_ids" "private" {
  vpc_id = data.aws_vpc.this.id

  tags = {
    "kubernetes.io/role/internal-elb" = "1"
  }
}

module "cluster" {
  source  = "terraform-aws-modules/eks/aws"
  version = "13.1.0"

  cluster_name    = var.cluster_name
  cluster_version = "1.18"

  subnets          = data.aws_subnet_ids.private.ids
  vpc_id           = var.vpc_id
  enable_irsa      = true
  map_roles        = [
    {
      rolearn  = module.iam_assumable_role_aws_iam_authenticator_proxy.this_iam_role_arn,
      username = module.iam_assumable_role_aws_iam_authenticator_proxy.this_iam_role_name,
      groups   = ["system:masters"]
    },
  ]

  worker_groups = [
    {
      instance_type        = "m5a.large"
      asg_desired_capacity = 2
      asg_max_size         = 3
    }
  ]

  kubeconfig_aws_authenticator_command = "curl"
  kubeconfig_aws_authenticator_command_args = [
    "-s",
    "https://${var.auth_url}/?psk=${random_password.auth_proxy_psk.result}",
  ]
}

resource "random_password" "auth_proxy_psk" {
  length  = 16
  special = false
}

Finally, we can deploy the proxy in Kubernetes using Helm:



resource "helm_release" "aws-iam-authenticator-proxy" {
  name              = "aws-iam-authenticator-proxy"
  chart             = "https://github.com/camptocamp/aws-iam-authenticator-proxy/tree/master/k8s"
  namespace         = "aws-iam-authenticator-proxy"
  dependency_update = true
  create_namespace  = tr

  values = [
    << EOT
eks_cluster_id: "${var.cluster_name}"
psk: "${random_password.auth_proxy_psk.result}"
serviceAccount:
  name: "aws-iam-authenticator-proxy"
  annotations:
    eks.amazonaws.com/role-arn: ${module.iam_assumable_role_aws_iam_authenticator_proxy.this_iam_role_arn}
EOT
  ]

  depends_on = [
    module.cluster,
  ]
}

You can add an Ingress or configure the Service to use an L4 LoadBalancer by tuning the Helm values.

Colored wrappers for kubectl

Raphaël Pinson — Tue, 06 Oct 2020 19:50:58 +0000

When using Kubernetes, kubectl is the command we use the most to visualize and debug objects.

However, it currently does not support colored output, though there is a feature request opened for this.

Let's see how we can add color support. I'll be using zsh with oh my zsh.

Edit: this feature was merged in oh my zsh, so it is now standard.

Zsh plugin

Let's make this extension into a zsh plugin called kubectl_color:

❯ mkdir -p ~/.oh-my-zsh/custom/plugins/kubectl_color
❯ touch ~/.oh-my-zsh/custom/plugins/kubectl_color/kubectl_color.plugin.zsh

Now we need to fill in this plugin.

JSON colorizing

Let's start with JSON, by adding an alias that colorizes JSON output using the infamous jq:

kj() {
  kubectl "$@" -o json | jq
}

compdef kj=kubectl

The compdef line ensures the kj function gets autocompleted just like kubectl.

Edit: I've added another wrapper for fx, which provides a dynamic way to parse JSON:

kjx() {
  kubectl "$@" -o json | fx
}

compdef kjx=kubectl

YAML colorizing

Just like for JSON, we can use yh to colorize YAML output:

ky() {
  kubectl "$@" -o yaml | yh
}

compdef ky=kubectl

Energize!

Our plugin is now ready, we only need to activate it in ~/.zshrc by adding it to the list of plugins, e.g.:

plugins=(git ruby kubectl kubectl_color)

and with fx:

Simple secret sharing with gopass and summon

Raphaël Pinson — Tue, 28 Jul 2020 16:34:57 +0000

Secrets are a fundamental, yet complex issue in software deployment.

Solutions such as KeepassX are simple to use, but quite impractical when it comes to automation.

More complex options like Hashicorp Vault are extremely powerful, but harder to set up and maintain.

Pass: a simple solution

When it comes to storing securely and sharing passwords in a team, it is hard to come up with a more simple and efficient solution than Git and GnuPG combined.

Pass is a shell script that does just that. Inside a Git repository, Pass stores passwords in individual files encrypted for all private GnuPG keys in the team. It features a CLI to manipulate passwords, add new entries, or search through existing passwords.

More features

However, Pass is quite limited in its features, so another project was born a few years later, to provide a new Go implementation of the Pass standard. Its name: simply Gopass.

Installing

Gopass is provided as binaries you can download from the releases page on GitHub.

Features

Here are some of the features that make Gopass a great tool.

Multiple mounts

While pass allows you to have a single Git repository with your passwords, Gopass lets you create multiple repositories called "mounts", which is very useful when you want to share different secrets with different people:



$ gopass mounts
gopass (/home/raphink/.password-store)
├── c2c (/home/raphink/.password-store-c2c)
├── perso (/home/raphink/.password-store-perso)
└── terraform (/home/raphink/.password-store-terraform)

Gopass uses a prefix to access secrets in mounts, so terraform/puppet/c2c actually refers to the secret stored in /home/raphink/.password-store-terraform/puppet/c2c.gpg.

Multiple users

Each Git repository can be set to encrypt passwords for multiple GnuPG keys.

The .gpg-id file at the root of each repository contains the list of public keys to use for encryption, and the .public-keys/ directory keeps a copy of each key, making it easy for collaborators to import them into their keyring before they can start encrypting passwords for the team.

Fuzzy search

Gopass helps you find entries when the key you gave it doesn't match an exact known path:



$ gopass jenkins
[ 0] c2c/c2c_aws/jenkins-c2c
[ 1] c2c/c2c_mgmtsrv/freeipa/c2c-jenkins-swarm
[ 2] c2c/c2c_mgmtsrv/freeipa/jenkins-test-users
[ 3] perso/Devel/jenkins-ci.org
[ 4] terraform/aws/jenkins-c2c

Found secrets - Please select an entry [0]:

Structured secrets

When decrypting a password, Gopass parses the content into two different parts: a password and a YAML document. For example, the content of a secret could look like this:



foo
---
key1: value1
another_key:
  bar: baz

Password

The first line of the content is the password. If this is all you're interested in, you can use gopass show --password to retrieve it:



$ gopass show --password perso/test
foo

Querying keys

When the second part of the content (lines 2 and following) is a valid YAML document, you can query these values by providing a key, for example:



$ gopass show perso/test key1
value1

Starting with Gopass 1.9.3, you can also query subkeys using either a dot or slash notation:



$ gopass show perso/test another_key.bar
baz
$ gopass show perso/test /another_key/bar
baz

This makes it extremely powerful to store several fields in the same secret.

TOTP

Gopass allows to store TOTP keys alongside passwords. For example, you can have the following secret, stored at terraform/service.io/api:



WPTmU`>b<Y31
---
password: 'WPTmU`>b<Y31'
totp: 'PIJ6AIHETAHSHOO7SHEI1AEK6IH1SOOCHATUOSH8XUAN0OOTH9XAHRUXO4AHJAEVI'
url: https://myservice.io
username: jdoe

In addition to retrieving each field with the corresponding key, you can also generate TOTP tokens with gopass totp:



$ gopass totp terraform/service.io/api
568000 lasts 18s    |------------==================|

Integrations

Gopass can be easily integrated into projects for deployments or CI/CD tasks.

Summon

The easiest way to integrate Gopass is probably to use Summon.

Summon is a tool which dynamically exposes environment variables with values retrieved from various secret stores. gopass is one of its possible providers.

Setup

Setting it up to use gopass is rather straightforward. We use a simple wrapper called summon-gopass, which needs to be in your PATH:



#!/bin/sh
gopass show $(echo "${@}"|tr : \ )

You can also simply make summon-gopass a symbolic link to your gopass binary, but subkeys won't work in this case.

Usage

Summon lets you provide a local secrets.yml file which defines which environment variables you wish to define, and how to find the values.

Here's a simple example of a secrets.yml file using the secret we defined earlier:



SERVICE_URL: !var terraform/service.io/api url
USER: !var terraform/service.io/api username
SERVICE_PASSWORD: !var terraform/service.io/api password

You can test this setup by running the following command in the directory containing secrets.yml:



$ summon env

The output should contain the 3 variables with the values stored in Gopass.

Exposing files

While the format above allows you to expose simple secrets as variables, it is not very practical when you need secrets exposed as files.

Summon covers this need however, using the file flag. For example:



SSH_KEY: !var:file terraform/service.io/ssh private_key

If terraform/service.io/ssh is a secret in Gopass whose private_key YAML field contains an SSH private key, then Summon will extract this secret, place it into a temporary file (in /dev/shm by default) and set the SSH_KEY variable with the path to the file. After the command returns, the temporary file will be delete.

You could then use such a secrets.yml file with:



summon sh -c 'ssh -i $SSH_KEY user@service'

Another useful example is to store a Kubernetes cluster configuration in Gopass, e.g.:



---
apiVersion: v1
clusters:
    - cluster:
        server: https://k8s.example.com
      name: k8s
contexts:
    - context:
        cluster: k8s
        namespace: default
        user: default-cluster-admin
      name: default-admin
current-context: default-admin
kind: Config
preferences: {}
users:
    - name: default-cluster-admin
      user:
        token: averylongtoken

With the following secrets.yml file:



KUBECONFIG: !var:file path/to/secret

You can then work on the Kubernetes cluster with kubectl using:



$ summon kubectl <some command>

Terraform integration

A simple way to pass variables to Terraform is to declare them and use summon to pass the values:



TF_VAR_var1: !var terraform/project1/secret1 field1

You can then run summon terraform to dynamically pass these secrets to Terraform.

Another possibility is to use Camptocamp's Terraform Pass Provider which lets you retrieve and set passwords in Gopass natively in Terraform:



provider "pass" {
  store_dir = "/srv/password-store"    # defaults to $PASSWORD_STORE_DIR
  refresh_store = false                # do not call `git pull`
}

# Store a value into the Gopass store
resource "pass_password" "test" {
  path = "secret/foo"
  password = "0123456789"
  data = {
    zip = "zap"
  }
}

# Retrieve password at another_secret/bar to be used in Terraform code
data "pass_password" "test" {
  path = "another_secret/bar"
}

The provider exposes the secret with the following properties:

path: path to the secret
password: secret password (first line of the content)
data: a structure (map) of the YAML data in the content
body: the content found on lines 2 and following, if it could not be parsed as YAML
full: the full content (all lines) of the secret

Hiera Integration

The most standard way to store secrets in Hiera is to use hiera-eyaml, which stores secret values encrypted inside YAML files, using either a PKCS7 key (default) or multiple GnuPG keys (when using hiera-eyaml-gpg).

If your passwords are already stored in Gopass, you might want to integrate that into Hiera instead.

The camptocamp/hiera-pass module provides two Hiera backends to retrieve keys either as full Gopass secrets, or as keys inside the secrets.

Decomissioning with Puppet: report & purge unmanaged resources

Raphaël Pinson — Thu, 23 Jul 2020 14:42:09 +0000

Puppet lets you manage resources explicitely. But did you know you can also dynamically purge unmanaged resources using Puppet?

Why?

A user in your organization just left, and you need to remove their account from all nodes. If you were managing their account with Puppet —whether with a user resource type or using an accounts module—, you need to make sure this user is absent:

user { 'jdoe':
  ensure => absent,
}

Great. Job done. Now, how long should this resource be kept in your code? One hour? One week? One year? What if an old node that was turned off wakes up months from now with this account activated?

To be honest, if a node turned off for months suddenly wakes up, you'll probably have more issues than just old users if your Puppet code base is quite active…
However, purging all unknown users would be a much easier approach than managing them explicitely!

How?

As explained in a previous post about managing files in Puppet, Puppet has the ability of purging unmanaged resources. I'll let you see the post for more explanations on how this works:

All the ways to manage files with Puppet

Raphaël Pinson for Camptocamp Infrastructure Solutions ・ Jun 8 '20

#puppet #cfgmgmt #tutorial #devops

What if I don't want to purge?

What if instead of purging, I'd just like Puppet to report the unmanaged resources but not do anything about them?

Luckily for us, noop works fine with the purge type, so you can use something like:

purge { 'user':
  noop   => true,
  unless => [
    ['uid', '<', '1000'],
    ['name', '==', 'nobody'],
  ],
}

This code will mark all users with a UID above 999 (except the nobody user) to be purged, but it won't do it. As a result, you'll get noop resources in your reports, for example in Puppetboard:

And then in the report, you'll see the unmanaged users:

Forcing purge

If you see users that should be purged, you can add again a user resource in your Puppet code to ensure their absence:

user { 'iperf':
  ensure => absent,
}

Another option is to make it a bit more dynamic. I've added an option in my accounts base class to use a dynamic fact to purge users on demand:

class osbase::accounts (
  Boolean $purge_users = str2bool($facts['purge_users']),
) {
  purge { 'user':
    noop   => !$purge_users,
    unless => [
      ['uid', '<', '1000'],
      ['name', '==', 'nobody'],
    ],
  }
}

The purge_users fact doesn't exist by default, so I can define it on the go when I need to purge users.
Now I can run puppet apply on a node and force purging the users with:

$ FACTER_purge_users=y puppet agent -t

And all unmanaged users will be removed from the node!

Do you have specific Puppet needs? Contact us, we can help you!

DEV Community: Raphaël Pinson

How to Automatically Issue Badges for Instruqt Labs

Issuing Credentials

isovalent / instruqt-go

A Go library for the Instruqt API

instruqt-go

Features

Installation

Example Usage

isovalent / credly-go

A Go library for the Credly API

credly-go

Features

Installation

Example Usage

Contributing

Running Tests

Overview

Pre-requisites

Setting Up the Environment

Setting Up the Basics

Implementing the Webhook Receiver

Step 4: The processWebhook() Function

Setting Up the Webhook on Instruqt

Testing the Code

Alternative testing: using Docker

Deploy the Function

Testing

Further Considerations

Streamlining Access to Embedded Instruqt Labs

isovalent / instruqt-go

A Go library for the Instruqt API

instruqt-go

Features

Installation

Example Usage

Why a Proxy

How to Pass User Data

Custom Parameters

Invites

Third Party Forms

A Note on Embed Tokens

Creating the Proxy

Initial Steps

Google Cloud Function Harnessing

Creating the Function

Implement getLabURL()

The getUser() Function

Testing the code

Alternative testing: using Docker

Hosting the Proxy on Google Cloud Functions

Create a Secret for the API Token

Deploy the Function

Further Considerations

Using the Proxy

Towards a Modular DevOps Stack

🌟 The Original Design

🐙 Argo CD Provider

🗺 The Plan for Modularization

💡 A Full Example

☸ The Cluster Module

🐙 The Argo CD Provider

🧩 Component Modules

🐙 The Argo CD Module

🥾 The Bootstrap Pattern

🚀 To Infinity

A 15-year Puppet Journey

How to allow dynamic Terraform Provider Configuration

Provider Initialization

Leveraging Pointers

Initialize the Clients

Client Pool Caching

Conclusion

March Cloud Native Romandie Meetup

Subjects

CloudBees CI

Exoscale SKS

Camptocamp DevOps Stack

Future Meetups

Immutability & loose coupling: a match made in heaven

Step 4: The `processWebhook()` Function

The `getUser()` Function