DEV Community: mikeyGlitz

Setting up Parseable with Kubernetes and Docker Desktop

mikeyGlitz — Fri, 09 May 2025 21:42:17 +0000

Setting up Parseable with Kubernetes and Docker Desktop

This guide will walk you through setting up Parseable, a modern log analytics platform, using Kubernetes on Docker Desktop. We'll cover the complete setup process including MinIO for storage, Parseable deployment using Helm, and configuring ingress for external access.

Prerequisites

Before we begin, ensure you have the following prerequisites installed and configured:

Docker Desktop with Kubernetes enabled
Ingress-NGINX controller deployed on your Kubernetes cluster
kubectl configured to work with your Docker Desktop Kubernetes cluster
Ansible configured to run playbooks

Setting up MinIO
Deploying Parseable using Helm
Configuring Ingress
Complete Ansible Playbook

Setting up MinIO

MinIO is an object storage system that Parseable uses for storing logs. Let's set it up first.

Deploying MinIO

We'll use Helm to deploy MinIO with some basic configuration. This setup creates a MinIO instance with a root user (minioadmin), configures a 10GB persistent volume for storage, and creates a bucket named "parseable". Additionally, we set up a service account with specific permissions - the account has access keys configured and is granted permissions to perform read, write and delete operations on objects within the parseable bucket, as well as list the bucket contents. This configuration ensures Parseable will have the necessary permissions to store and manage logs in MinIO.

- name: Deploy MinIO using Helm
    kubernetes.core.helm:
    release_name: minio
    release_namespace: minio
    chart_ref: minio
    chart_repo_url: https://charts.min.io
    create_namespace: true
    release_values:
        rootUser: minioadmin
        rootPassword: minioadmin
        persistence:
        size: 10Gi
        buckets:
          - name: parseable
            policy: none
            purge: false
        svcaccts:
          - accessKey: parseable
            secretKey: 'parseable'
            user: console
            policy:
            statements:
              - resources:
                  - arn:aws:s3:::parseable/*
                actions:
                  - s3:GetObject
                  - s3:PutObject
                  - s3:DeleteObject
              - resources:
                - arn:aws:s3:::parseable
              actions:
                - s3:ListBucket

Deploying Parseable

Now that we have MinIO set up, let's deploy Parseable using its Helm chart.

Installing Parseable

The following Ansible code block deploys Parseable using two tasks. The first task creates a Kubernetes Secret containing essential configuration parameters for Parseable, including MinIO connection details (like the service URL, access credentials, and bucket name), network settings, storage directories, and admin authentication credentials. All these values are base64 encoded as required for Kubernetes Secrets. The second task uses Helm to deploy the actual Parseable application, utilizing the official Helm chart from the Parseable repository and configuring it to use S3 (MinIO) as its storage backend.

- name: Deploy Parseable configuration
    kubernetes.core.k8s:
      state: present
      definition:
        apiVersion: v1
        kind: Secret
        metadata:
          name: parseable-env-secret
          namespace: parseable
        data:
          s3.url: "{{ 'https://minio.minio.svc.cluster.local:9000' | b64encode }}"
          s3.access.key: "{{ 'parseable' | b64encode }}"
          s3.secret.key: "{{ 'parseable' | b64encode }}"
          s3.bucket: "{{ 'parseable' | b64encode }}"
          addr: "{{ '0.0.0.0:8000 | b64encode }}"
          staging.dir: "{{ './staging' | b64encode }}"
          fs.dir: "{{ './data' | b64encode }}"
          username: "{{ 'parseable' | b64encode }}"
          password: "{{ 'parseable' | b64encode }}"
- name: Deploy Parseable helm chart
  kubernetes.core.helm:
    name: parseable
    chart_ref: parseable
    chart_repo_url: https://charts.parseable.com
    release_namespace: parseable
    values:
      parseable:
        store: s3-store

Configuring Ingress

To expose the Parseable ingest API endpoint for log ingestion, we'll create an Ingress resource that routes requests through the /logs path prefix. This approach both limits external access to just the ingest API and avoids CORS issues, while keeping the UI and other endpoints private within the cluster.

Creating Ingress Resource

- name: Deploy the Parseable ingress
  kubernetes.core.k8s:
  state: present
  definition:
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: parseable-ingress
      namespace: parseable
      annotations:
        kubernetes.io/ingress.class: nginx
        nginx.ingress.kubernetes.io/use-regex: "true"
        nginx.ingress.kubernetes.io/rewrite-target: /api/v1/ingest/$2
    spec:
      rules:
      - host: parseable.local
        http:
          paths:
          - path: /logs/api/v1/ingest(/|$)(.*)
            pathType: ImplementationSpecific
            backend:
              service:
                name: parseable
                port:
                  number: 80

Complete Ansible Playbook

For convenience, here's a complete Ansible playbook that combines all the tasks we've discussed. Create a file named deploy-parseable.yml with the following content:

---
- name: Deploy Parseable Stack
  hosts: localhost
  connection: local
  tasks:
    - name: Deploy MinIO using Helm
      kubernetes.core.helm:
        release_name: minio
        release_namespace: minio
        chart_ref: minio
        chart_repo_url: https://charts.min.io
        create_namespace: true
        release_values:
          rootUser: minioadmin
          rootPassword: minioadmin
          persistence:
            size: 10Gi
          buckets:
            - name: parseable
              policy: none
              purge: false
          svcaccts:
            - accessKey: parseable
              secretKey: 'parseable'
              user: console
              policy:
                statements:
                  - resources:
                      - arn:aws:s3:::parseable/*
                    actions:
                      - s3:GetObject
                      - s3:PutObject
                      - s3:DeleteObject
                  - resources:
                      - arn:aws:s3:::parseable
                    actions:
                      - s3:ListBucket

    - name: Deploy Parseable configuration
      kubernetes.core.k8s:
        state: present
        definition:
          apiVersion: v1
          kind: Secret
          metadata:
            name: parseable-env-secret
            namespace: parseable
          data:
            s3.url: "{{ 'https://minio.minio.svc.cluster.local:9000' | b64encode }}"
            s3.access.key: "{{ 'parseable' | b64encode }}"
            s3.secret.key: "{{ 'parseable' | b64encode }}"
            s3.bucket: "{{ 'parseable' | b64encode }}"
            addr: "{{ '0.0.0.0:8000' | b64encode }}"
            staging.dir: "{{ './staging' | b64encode }}"
            fs.dir: "{{ './data' | b64encode }}"
            username: "{{ 'parseable' | b64encode }}"
            password: "{{ 'parseable' | b64encode }}"

    - name: Deploy Parseable helm chart
      kubernetes.core.helm:
        name: parseable
        chart_ref: parseable
        chart_repo_url: https://charts.parseable.com
        release_namespace: parseable
        values:
          parseable:
            store: s3-store

    - name: Deploy the Parseable ingress
      kubernetes.core.k8s:
        state: present
        definition:
          apiVersion: networking.k8s.io/v1
          kind: Ingress
          metadata:
            name: parseable-ingress
            namespace: parseable
            annotations:
              kubernetes.io/ingress.class: nginx
              nginx.ingress.kubernetes.io/use-regex: "true"
              nginx.ingress.kubernetes.io/rewrite-target: /api/v1/ingest/$2
          spec:
            rules:
            - host: parseable.local
              http:
                paths:
                - path: /logs/api/v1/ingest(/|$)(.*)
                  pathType: ImplementationSpecific
                  backend:
                    service:
                      name: parseable
                      port:
                        number: 80

Run the complete playbook with:

ansible-playbook deploy-parseable.yml

Accessing Parseable

Once deployed, you can access the Parseable UI by setting up port forwarding from your local machine to the Parseable service in the cluster. Run the following command to forward port 8000 on your local machine to port 80 of the Parseable service:

kubectl port-forward -n parseable svc/parseable 8000:80

You can now send logs to Parseable by making POST requests to the ingress endpoint at host.docker.internal/logs/api/v1/ingest. Here's an example curl command that demonstrates how to send logs:

curl -X POST http://host.docker.internal/logs/api/v1/ingest \
  -H "Content-Type: application/json" \
  -H "X-P-Stream: my-logs" \
  -H "Authorization: Basic $(echo -n 'parseable:parseable' | base64)" \
  -d '{
    "timestamp": "2024-03-20T10:00:00Z",
    "level": "INFO",
    "message": "Application started",
    "service": "my-service",
    "environment": "development"
  }'

In this example:

X-P-Stream header specifies the log stream name
Authorization header contains the base64-encoded credentials (username:password)
The JSON payload contains structured log data with:
- timestamp: ISO 8601 formatted timestamp
- level: Log level (INFO, ERROR, etc.)
- message: The actual log message
- Additional fields for context (service, environment, etc.)

You can send multiple log entries in a single request by using an array:

curl -X POST http://host.docker.internal/logs/api/v1/ingest \
  -H "Content-Type: application/json" \
  -H "X-P-Stream: my-logs" \
  -H "Authorization: Basic $(echo -n 'parseable:parseable' | base64)" \
  -d '[
    {
      "timestamp": "2024-03-20T10:00:00Z",
      "level": "INFO",
      "message": "Application started",
      "service": "my-service"
    },
    {
      "timestamp": "2024-03-20T10:00:01Z",
      "level": "INFO",
      "message": "Database connection established",
      "service": "my-service"
    }
  ]'

After sending some sample logs, you can view them in the Parseable UI by accessing localhost:8000 in your browser:

Next Steps

Now that you have Parseable up and running, here are some recommended next steps to enhance your logging setup:

Configure Client-Side Log Shipping
- Implement pino.js in your frontend applications for structured logging
- Set up a custom transport to send logs directly to Parseable's ingest endpoint
- Configure log levels and formatting for optimal analysis
- Implement error tracking and performance monitoring
Integrate with Grafana
- Add Parseable as a data source in Grafana
- Create dashboards to visualize your logs
- Set up alerts based on log patterns and thresholds
- Configure log exploration and analysis workflows
Implement Authentication with OpenID Connect
- Set up Keycloak as your identity provider
- Configure Parseable to use OpenID Connect for authentication
- Define user roles and permissions
- Implement SSO (Single Sign-On) for your logging platform

Each of these steps will be covered in detail in upcoming articles. Would you like to proceed with any specific integration first?

Troubleshooting

If you encounter any issues during the setup, check the following:

Verify all pods are running:

kubectl get pods -A

Check MinIO logs:

kubectl logs -n minio -l app=minio

Check Parseable logs:

kubectl logs -l app.kubernetes.io/name=parseable

Conclusion

You now have a fully functional Parseable instance running on your local Kubernetes cluster. This setup provides a solid foundation for log analytics and monitoring in your development environment.

Bringing It All Together: Integrating GraphQL with Gin in Go

mikeyGlitz — Thu, 06 Jun 2024 02:05:29 +0000

In this phase of our journey, we delve into the realm of middleware integration with gin and the implementation of authentication middleware using gocloak. Building upon the groundwork laid in previous sections, we now unify our efforts by integrating middleware seamlessly into our GraphQL server. With gin, a powerful HTTP web framework for Go, we enhance our server's capabilities by incorporating middleware functions to preprocess requests. Leveraging gocloak, a Go module for interfacing with Keycloak, we secure our server with authentication middleware. This pivotal stage marks the convergence of all preceding elements, culminating in the creation of the server run function, which orchestrates the execution of our GraphQL API server. Let's explore how these components harmonize to elevate our server's functionality and security.

Implementing Authentication Middleware with Keycloak and Gocloak

The final middleware we need to add to our server is authentication. In this example, we'll use Keycloak as our identity provider. To interface with Keycloak in Go, we'll use the gocloak module. By leveraging gocloak, we can perform authentication against Keycloak using Gin middleware.

To create the middleware, we begin by specifying the header that we want to inspect from the request. Keycloak leverages the OpenID Connect protocol, so we expect the Authorization header to begin with the word "Bearer," followed by a space, and then the full token string. Below, we define the constant "Bearer " for this purpose:

const headerPrefix = "Bearer "

Next, we need to verify the token. To achieve this, we create a function that accepts a Gorm database pointer (*gorm.DB) and an HTTP request pointer (*http.Request). This function will extract the Authorization header from the request, validate the token using Keycloak, and return a user if a match is found in the database. If the calls do not complete successfully, the function will return an error.

func ValidateToken(db *gorm.DB, req *http.Request) (*model.User, error) {
    authToken := req.Header.Get("Authorization")
    authToken = strings.TrimPrefix(authToken, headerPrefix) // Strip the "Auth " from the bearer token
    keycloak := config.Config.Auth

    // Make call to keycloak authenticating the token
    client := gocloak.NewClient(keycloak.Endpoint)

    // Add certificate verification if a certificate path is set
    if len(keycloak.CertificatePath) > 0 {
        log.Infof("Reading certificate from %s...", keycloak.CertificatePath)
        cert, err := os.ReadFile(keycloak.CertificatePath)
        if err != nil {
            log.Errorf("[identity.cert] Unable to read certificate => %v", err)
            return nil, err
        }
        certPool := x509.NewCertPool()
        if ok := certPool.AppendCertsFromPEM(cert); !ok {
            log.Errorf("[identity.cert] Unable to add cert to pool => %v", err)
            return nil, err
        }
        restyClient := client.RestyClient()
        restyClient.SetTLSClientConfig(&tls.Config{RootCAs: certPool})
        log.Info("Imported certificate to keycloak client")
    }

    res, err := client.RetrospectToken(req.Context(), authToken, keycloak.ClientID, keycloak.ClientSecret, keycloak.RealmName)
    if err != nil {
        log.Errorf("unable to validate access token => %v", err)
        return nil, err
    }
    log.Debugf("[auth] Access Token => %v", *res)
    if !*res.Active {
        err = errors.New("session is not active")
        log.Errorf("session is not active => %v", err)
        return nil, err
    }
    // fetch userinfo and query the database for the user
    info, err := client.GetUserInfo(req.Context(), authToken, keycloak.RealmName)
    if err != nil {
        log.Errorf("unable to fetch user info => %v", err)
        return nil, err
    }

    // add the user to the database if there is no current entry for the user
    var user model.User
    if err = db.FirstOrCreate(&user, model.User{
        Username: *info.PreferredUsername,
        Name:     fmt.Sprintf("%s %s", *info.GivenName, *info.FamilyName),
    }).Error; err != nil {
        log.Errorf("unable to save user to database => %v", err)
        return nil, err
    }

    log.Debug(user)

    return &user, nil
}

The AuthenticationMiddleware function is designed to integrate authentication into a Gin web server. This function takes a Gorm database pointer (*gorm.DB) as an argument and returns a Gin handler function. Inside the handler, the middleware calls the ValidateToken function, passing it the database pointer and the current HTTP request. If the token validation fails, an error is logged, and the request is aborted with an HTTP status of 403 (Forbidden). If the token is successfully validated, the user information is added to the request context, allowing downstream handlers to access it. Finally, the middleware calls c.Next() to pass control to the next handler in the chain.

func AuthenticationMiddleware(db *gorm.DB) gin.HandlerFunc {
    return func(c *gin.Context) {
        user, err := ValidateToken(db, c.Request)
        if err != nil {
            log.Errorf("unable to authenticate token => %v", err)
            err = c.AbortWithError(http.StatusForbidden, err)
            log.Debug(err)
            return
        }

        c.Request = c.Request.WithContext(context.WithValue(c.Request.Context(), userKey, user))
        c.Next()
    }
}

Finally, the ForUser function is designed to retrieve the authenticated user from the request context within a Gin web server. This function takes a context (ctx) as an argument and attempts to extract the user information stored in the context using a predefined key (userKey). It utilizes the ctx.Value method to access the value associated with userKey and performs a type assertion to convert it to a *model.User. If the user information is not found or the type assertion fails, the function returns nil. This utility function allows other parts of the application to conveniently access the authenticated user from the context, facilitating user-specific operations and data handling.

func ForUser(ctx context.Context) *model.User {
    user, _ := ctx.Value(userKey).(*model.User)
    return user
}

Finalizing Server Setup: The Run Function

With all the middleware components established throughout the series, it's time to bring everything together and finalize the server setup. The Run function acts as the glue, orchestrating the integration of various middleware components and starting the Gin server. This function typically initializes the Gin router, applies the middleware layers in the desired order, and defines the routes or endpoints for handling incoming requests. It encapsulates the server configuration and provides a unified entry point for launching the web server. By consolidating the middleware setup and server initialization logic into a single function, we ensure consistency, maintainability, and ease of management for the entire server application.

The Run function sets up the server configuration, defines routes, applies middleware, and starts the server. It initializes the server with Gin's default middleware, sets up endpoints for actuator, GraphQL playground, and GraphQL itself. The middleware stack includes services, data loader middleware, and authentication middleware to handle various aspects of request processing and security. Finally, it starts the server to listen on the configured host and port, logging the endpoint for reference.

func Run(db *gorm.DB) {
    config := config.Config
    endpoint := fmt.Sprintf("%s:%d", config.Service.Host, config.Service.Port)
    r := gin.Default()

    r.GET("/actuator/*endpoint", handlers.ActuatorHandler(db))
    r.Use(middleware.Services(db, index.IndexConnection))
    r.Use(middleware.DataloaderMiddleware())
    r.GET(config.Service.PlaygroundPath, handlers.PlaygroundHandler())
    r.Use(gin.Recovery())

    secured := r.Group(config.Service.Path)
    secured.Use(middleware.AuthenticationMiddleware(db))
    secured.POST("/", handlers.GraphqlHandler())

    log.Infof("Running @ http://%s", endpoint)
    log.Fatal(r.Run(endpoint))
}

The Run function serves as the centerpiece of our server logic, orchestrating the integration of GraphQL with Gin in Go. Encapsulated within the pkg/server package, it represents the culmination of our efforts across various modules and middleware layers. At the bottom of our application entrypoint, housed in cmd/main.go, we invoke server.Run to kickstart the server and bring our GraphQL-powered application to life.

Finishing Touches: Revisiting GraphQL

This GraphqlHandler function serves as the entry point for GraphQL requests in our server. It initializes a config struct with the resolver functions provided by our graph package. Additionally, it configures directives, such as validation, to be used during query execution. Finally, it creates a handler using handler.NewDefaultServer, passing in the executable schema generated by gqlgen based on our schema and resolvers. This handler is then returned as a Gin middleware function, allowing it to process GraphQL requests coming to our server.

func GraphqlHandler() gin.HandlerFunc {
    config := generated.Config{Resolvers: &graph.Resolver{}}
    // Add directives
    config.Directives.Validate = directives.Validate

    h := handler.NewDefaultServer(generated.NewExecutableSchema(config))
    return func(c *gin.Context) { h.ServeHTTP(c.Writer, c.Request) }
}

As we put the finishing touches on our GraphQL server, let's revisit one of our resolvers to demonstrate how we can seamlessly integrate middleware into our GraphQL operations. Middleware plays a crucial role in intercepting and augmenting requests before they reach our resolvers, allowing us to perform additional tasks such as authentication, logging, or data manipulation. By integrating middleware into our resolver functions, we can enhance the functionality and security of our GraphQL API without cluttering our resolver logic. Let's dive into the details of how middleware can be seamlessly incorporated into our GraphQL server architecture.

In this Go code snippet, we revisit the User resolver previously implemented in our GraphQL server. This resolver function, named Pantries, is responsible for fetching a list of pantries associated with a particular user. Within the function, we access the services layer through the context, leveraging a middleware function to retrieve the necessary service. Once obtained, we call the FetchPantriesByAuthor method from the PantryService to retrieve the pantries associated with the user. The function accepts optional parameters such as order, pagination details (startAt and size), and returns a PantryList along with any potential errors encountered during the process. This resolver exemplifies how middleware can seamlessly integrate with resolver functions to enhance the functionality of our GraphQL server.

func (r *userResolver) Pantries(ctx context.Context, obj *model.User, order *model.SearchOrder, startAt *int, size *int) (*model.PantryList, error) {
    services := middleware.ForServices(ctx)
    return services.PantryService.FetchPantriesByAuthor(order, &obj.ID, startAt, size)
}

Conclusion: Bringing it All Together

In conclusion, this article series has provided a comprehensive guide to building a robust GraphQL API server in Go. We began by setting up gqlgen for GraphQL integration, customized it to fit Go project conventions, and defined our GraphQL schema with resolvers. We abstracted our data model using services, integrated them using middleware, and implemented schema-level validation. Additionally, we optimized data retrieval with dataloaders, ensuring efficient query execution. Finally, we tied everything together with authentication middleware and a run function encapsulated in the server package. By following these steps, we've laid a solid foundation for creating powerful GraphQL APIs in Go, ready to handle various use cases and scale with ease.

References

Keycloak golang webservices - https://mikebolshakov.medium.com/keycloak-with-go-web-services-why-not-f806c0bc820a
Opinionated graphql server with go - https://dev.to/cmelgarejo/creating-an-opinionated-graphql-server-with-go-part-1-3g3l

Building Robust Applications in Go: Integrating Envconfig, Gorm, and OpenSearch

mikeyGlitz — Wed, 24 May 2023 16:28:25 +0000

Welcome to the second entry in our multi-part blog series on building a powerful application with Go! In our previous post, we laid the foundation by creating a Go-based project skeleton using GitLab, Docker, and Go-based linting and build tools. Now, we're ready to delve deeper into the development process and explore three crucial topics: creating an application configuration with envconfig, integrating a data persistence layer with Gorm, and indexing data with OpenSearch.

Efficiently managing application configuration, persisting data, and enabling powerful search capabilities are vital aspects of building robust and scalable applications. In this article, we'll guide you through the process of implementing these features in your Go application, equipping you with the necessary tools to develop high-performing and feature-rich software.

Specifically, we'll cover the following topics:

Creating an Application Configuration with envconfig: We'll explore the envconfig package to establish a flexible and centralized configuration system for our application. By leveraging environment variables, we'll make our application easily configurable across different deployment environments.
Integrating a Data Persistence Layer with Gorm: Gorm, a popular ORM library for Go, provides powerful abstractions for interacting with databases. We'll investigate how to integrate Gorm into our application, enabling seamless data persistence operations. You'll learn about defining models, performing CRUD operations, and establishing relationships between entities.
Indexing Data with OpenSearch: OpenSearch, a high-performance search engine, allows us to efficiently index and search through large datasets. We'll integrate OpenSearch into our application and demonstrate how to leverage its indexing capabilities for quick and accurate search operations. You'll discover techniques for indexing data and executing various types of searches.

By the end of this article, you'll have a solid understanding of application configuration management, data persistence with Gorm, and data indexing with OpenSearch in your Go-based projects. Armed with these essential skills, you'll be well-equipped to develop robust, scalable, and search-enabled applications using Go.

Let's jump right in and explore the powerful combination of envconfig, Gorm, and OpenSearch to enhance your Go applications!

Creating an Application Configuration with envconfig

One of the primary aspects I focus on is configuring my applications. I rely on configuration to store essential details for the various systems that the application integrates with, such as databases, identity provider services, indexing services, and listener ports. The API's target environment is a Kubernetes cluster. To provide configuration values to Kubernetes, I make use of ConfigMaps or Secrets to supply environment variables.

To extract values from the system environment, I utilize envconfig, a Go package. Envconfig facilitates mapping system environment variables to a Go struct. These Go structs are exposed through a config package, enabling other parts of the application to access them.

Below is an example code block that demonstrates a configuration model using struct tags:

package config

type DatabaseConfiguration struct {
  Host      string  `required:"true"`
  Port      int     `required:"true"`
  Username  string
  Password  string
  Name      string
}

type OpensearchConfiguration struct {
  Username        string
  Password        string
  CertificatePath string    `split_words:"true"`
  Addresses       []string  `required:"true"`
}

type Configuration struct {
  Database    DatabaseConfiguration
  Opensearch  OpensearchConfiguration
}

Once the configuration model is established, I expose it through a package-level function called GetConfiguration. It is expected to run once during the package setup.

package config

var Config Configuration

func GetConfiguration() error {
  var dbConfiguration DatabaseConfiguration
  if err := envconfig.Process("db", &dbConfiguration); err != nil {
    log.Errorf("unable to read database configuration => %v", err)
  }

  var indexConfiguration DatabaseConfiguration
  if err := envconfig.Process("os", &indexConfiguration); err != nil {
    log.Errorf("unable to read opensearch configuration => %v", err)
  }

  Config = Configuration {
    Database:   dbConfiguration
    Opensearch: indexConfiguration
  }
}

In the GetConfiguration function, I retrieve the configuration values for the database and OpenSearch. These values are populated into their respective structs. If any errors occur during the process, appropriate error logging is performed. The resulting configurations are assigned to the Config variable within the Configuration struct.

Integrating a Data Persistence Layer with Gorm

After successfully configuring the application, it's time to delve into integrating the data layer. For this purpose, I will utilize gorm, a powerful SQL ORM that facilitates rapid development of the data layer using model structs.

Similar to the usage of struct tags in envconfig, gorm also leverages structs with struct tags to create a data model. To begin, we establish a Base struct that serves as a foundation for other models. The Base struct allows for overriding ID creation using UUID and sets default values for CreatedAt and UpdatedAt.
The BaseSoftDelete struct serves as a mechanism for implementing soft deletion. It extends the Base struct and introduces the DeletedAt field, which utilizes the gorm.DeletedAt type for handling soft deletes. By applying a struct tag, the DeletedAt field is indexed for optimized querying.

package model

type Base struct {
  ID        string    `json:"id" gorm:"primaryKey,type:uuid"`
  CreatedAt time.Time `json:"createdAt"`
  UpdatedAt time.Time `json:updatedAt"`
}

type BaseSoftDelete {
  Base
  DeletedAt gorm.DeletedAt  `json:"deletedAt" gorm"index"`
}

func (b *Base) BeforeCreate(txn *gorm.DB) error {
  b.ID = uuid.NewString()
  return nil
}

In addition, the Base struct contains a BeforeCreate function, which serves as a pre-create hook for any struct that embeds the Base struct. This hook is invoked automatically before the struct is saved into the database, allowing us to set the ID to a new UUID. By generating a new UUID, we ensure uniqueness and maintain consistency when persisting data.

By utilizing the Base and BaseSoftDelete structs along with the BeforeCreate function, we establish a solid foundation for managing soft deletion and automated UUID generation within our data models. These features contribute to the overall robustness and reliability of our application's data layer.

With the base types created, we go on to create the rest of the model.

package model

type Measurement struct {
  BaseSoftDelete
  Unit string
  Amount float32
}

type Ingredient struct {
  BaseSoftDelete
  Name string
  Measurement Measurement
}

type Recipe struct {
  BaseSoftDelete
  Ingredients []Ingredient
  Steps       []string
}

type User struct {
  BaseSoftDelete
  Name    string
  Email   string
  Recipes []Recipe
}

These model structs define the data structure for various entities. For instance, the Measurement struct represents a measurement with its associated unit and amount. Similarly, the Ingredient struct represents an ingredient with its name and corresponding measurement. The Recipe struct captures a recipe, including its ingredients and steps. Lastly, the User struct represents a user with their name, email, and associated recipes.

With these model definitions, we establish a solid foundation for building our data layer using gorm. These structs provide a structured representation of our data entities, enabling efficient storage, retrieval, and manipulation of data within the application.

Once we have created the data models, the next step is to define the database connection. To accomplish this, we can implement an initialization function called InitDB. This function will facilitate the establishment of a database connection by leveraging the values from the config package we defined earlier.

package database

func InitDB() (*gorm.DB, error) {
  dbc := config.Config.Database
  databaseConnection := fmt.Sprintf("host=%s user=%s password=%s dbname=%s port =%d", dbc.Host, dbc.Username, dbc.Password, dbc.Name, dbc.Port)
  db, err := gorm.Open(postgres.Open(databaseConnection), &gorm.Config{})
  if err != nil {
    log.Errorf("[database.connection] => %v", err)
    return nil, err
  }
  return db, nil
}

In the InitDB function, we extract the database configuration values from the config package. These values include the host, username, password, database name, and port. Using these values, we construct a connection string for the PostgreSQL database.

Next, we use the gorm.Open function to establish a connection to the database. We pass in the PostgreSQL driver (postgres.Open) along with the constructed connection string. Additionally, we provide an empty &gorm.Config{} as the second argument to indicate the default configuration for the gorm instance.

If the connection is successfully established, the db object is returned, allowing us to interact with the database in our application. However, if an error occurs during the connection process, an error message is logged.

By implementing the InitDB function, we create a reusable and efficient method for initializing the database connection within our application's data layer. This sets the stage for seamless integration with the data models we defined earlier.

Indexing Data with OpenSearch

With a basic data layer established, it's time to explore the next step: making the data searchable. In this regard, we will leverage OpenSearch to index our data. OpenSearch, a fork of Elasticsearch created by Amazon Web Services, provides a powerful search engine that enhances data retrieval capabilities. While OpenSearch inherits many features from Elasticsearch, it stands out as a fully open-source solution that offers more features for free.

To kickstart our OpenSearch implementation, we first need to model the OpenSearch connection. Our goal is to implement three primary operations in OpenSearch: inserting a new record, updating an existing record, deleting a record, and retrieving search results.

package index

type OpenSearchConnection interface {
  Insert(index string, payload interface{}) error
  Delete(index string, id string) error
  Update(index string, id string, updates interface{}) error
}

To kickstart our OpenSearch implementation, we first need to model the OpenSearch connection. Our goal is to implement four three operations in OpenSearch: inserting a new record, updating an existing record, deleting a record, and retrieving search results.

Implementing the OpenSearch Client

The Client struct implements the OpenSearchConnection interface and serves as a container for the OpenSearch client object. Additionally, we introduce a handleBodyError function to handle error messages received from OpenSearch.

type Client struct {
  DB *opensearch.Client
}

func handleBodyError(status string, responseBytes []byte) error {
  res := gjson.GetManyBytes(responseBytes, "error.type", "error.reason")
  return fmt.Errorf(
    "[ %s ] %s: %s",
    status,
    res[0].String(),
    res[1].String(),
  )
}

func (idx *Client) Insert(index string, payload interface{}) error {
  log.Debugf("[opensearch.insert] inserting record into index %s:\n%v", index, payload)
  res, err := idx.DB.Index(index, opensearchutil.NewJSONReader(&payload))
  if err != nil {
    log.Errorf("[opensearch.insert] unable to insert document into OpenSearch => %v", err)
    return err
  }
  defer res.Body.Close()
  responseBytes, err := io.ReadAll(res.Body)
  if err != nil {
    log.Errorf("[opensearch.insert] unable to parse JSON response => %v", err)
    return err
  }
  if res.IsError() {
    err := handleBodyError(res.Status(), responseBytes)
    log.Errorf("[opensearch.insert] received error response from OpenSearch => %v", err)
    return err
  }
  inserted := gjson.GetBytes(responseBytes, "result")
  log.Debugf("[opensearch.insert] insertion result => %s", inserted.String())

  return nil
}

// ... (rest of the functions)

The provided code defines the Client struct, which implements the OpenSearchConnection interface. It includes methods for inserting, updating, deleting, and searching records in OpenSearch. The handleBodyError function is used to handle error messages received from OpenSearch.

In the Insert method, a record is inserted into the specified index using the OpenSearch client. The response is processed, and any errors or relevant information are logged accordingly.

The Update and Delete methods follow a similar pattern, leveraging the OpenSearch client to perform the respective operations and handling the responses appropriately.

The Search method relies on a couple of additional helpers:
A SearchResponse struct to store the JSON structure of the search result,
and a function parseSearchResults to parse the OpenSearch response into
usable search results.

type SearchResponse struct {
 Hits     int                       // The number of results returned from the search
 Results  []map[string]gjson.Result // An array of results
 LastItem string                    // The ID of the last item in the list
}

func parseSearchResult(responseBytes []byte, isGeneric bool) (*SearchResponse, error) {
 res := gjson.GetManyBytes(responseBytes, "hits.total.value", "hits.hits")
 hits, err := strconv.Atoi(res[0].String())
 if err != nil {
  return nil, err
 }
 results := res[1].Array()
 mapped := make([]map[string]gjson.Result, 0)
 if len(results) > 0 {
  mapped = funk.Map(results, func(elem gjson.Result) map[string]gjson.Result {
   source := elem.Map()["_source"]
   if !isGeneric {
    return source.Map()
   }

   index := elem.Map()["_index"]
   sourceFields := source.Map()
   document := make(map[string]gjson.Result)
   document["index"] = index
   for key, element := range sourceFields {
    document[key] = element
   }
   return document
  }).([]map[string]gjson.Result)
  log.Debugf("Received results => %v", mapped)
 }

 response := &SearchResponse{
  Hits:    hits,
  Results: mapped,
 }

 if len(results) > 0 {
  lastElem := funk.Last(results).(gjson.Result)
  lastId := gjson.Get(lastElem.Raw, "id").String()
  response.LastItem = lastId
 }

 return response, nil
}

func (idx *Client) Search(index string, payload *map[string]interface{}, order string, startAt int, size int) (*SearchResponse, error) {
 sort := []map[string]interface{}{{
  "updatedAt": map[string]interface{}{"order": order},
 }}

 pageSize := 10
 if size != 0 {
  pageSize = size
 }

 query := &map[string]interface{}{
  "match_all": map[string]interface{}{},
 }

 reqBody := map[string]interface{}{
  "query": query,
  "sort":  sort,
  "size":  pageSize,
  "from":  startAt,
 }

 res, err := idx.DB.Search(idx.DB.Search.WithIndex(index), idx.DB.Search.WithBody(opensearchutil.NewJSONReader(&reqBody)))
 if err != nil {
  log.Errorf("[opensearch.search] encountered error while attempting to search:\n%v", err)
  return nil, err
 }
 defer res.Body.Close()
 bodyBytes, err := io.ReadAll(res.Body)
 if err != nil {
  log.Errorf("[opensearch.search] unable to read in response body:\n%v", err)
  return nil, err
 }
 if res.IsError() {
  err := handleBodyError(res.Status(), bodyBytes)
  log.Errorf("[opensearch.search] OpenSearch responded with error:\n%v", err)
  return nil, err
 }
 // If we're searching in all indices, we want to change the shape of the response
 genericResult := index == "_all"
 parsedResult, err := parseSearchResult(bodyBytes, genericResult)
 if err != nil {
  log.Errorf("[opensearch.search] unable to parse search response:\n%v", err)
  return nil, err
 }
 return parsedResult, nil
}

By encapsulating the OpenSearch functionality within the Client struct and implementing the OpenSearchConnection interface, we ensure a modular and extensible design for interacting with OpenSearch in our application.

Connecting to OpenSearch

Much like we saw in the previous section on how to establish a connection to our data persistance layer using Gorm, we'll now examine how to establish a connection to OpenSearch and leverage its capabilities within our application. We'll begin by introducing the InitializeClient() function, which plays a crucial role in initializing the OpenSearch connection. This function retrieves configuration values from the environment and sets up the necessary components for communication with OpenSearch. Let's dive into the details.

var IndexConnection OpensearchConnection

func InitializeClient() error {
    esConfig := config.Config.Opensearch

    config := opensearch.Config{
        Addresses: esConfig.Addresses,
        Username:  esConfig.Username,
        Password:  esConfig.Password,
    }

 if len(esConfig.CertificatePath) > 0 {
  log.Infof("Reading certificate from %s...", esConfig.CertificatePath)
  esCert, err := certificateFileReader(esConfig.CertificatePath)
  if err != nil {
   log.Errorf("[opensearch.initialization] Unable to read in CA certificate file at %s => %v", esConfig.CertificatePath, err)
   return err
  }
  esCa := x509.NewCertPool()
  if ok := esCa.AppendCertsFromPEM(esCert); !ok {
   err := errors.New("unable to add certificate to certificate pool")
   log.Errorf("[opensearch.initialization] Encountered an error adding certificate to pool => %v", err)
   return err
  }

  config.Transport = &http.Transport{
   TLSClientConfig: &tls.Config{RootCAs: esCa},
  }
  log.Info("Successfully loaded certificate")
 }

    client, err := opensearch.NewClient(config)
    if err != nil {
        log.Errorf("[opensearch.initialization] Unable to create elasticsearch connection => %v", err)
        return err
    }

    IndexConnection = &Client{DB: client}

    return nil
}

In the given code block, the InitializeClient() function demonstrates how to establish a connection to OpenSearch. It starts by retrieving the OpenSearch configuration from the environment settings. The configuration includes addresses, username, and password required for the connection.

To ensure secure communication, the function supports certificate-based authentication. If a certificate path is provided in the configuration, the function reads the certificate file and adds it to the certificate pool.

Once the necessary configurations are set, the function creates an OpenSearch client using the specified settings. The created client is assigned to the DB field of the Client struct, which is then assigned to the IndexConnection variable. This package-level variable allows other parts of the application to access the OpenSearch client conveniently.

By utilizing the InitializeClient() function, we can establish a connection to OpenSearch and prepare the client for subsequent database operations.

Synching with the Database

Having set up the OpenSearchConnection interface, we can now leverage gorm hooks to synchronize database operations with OpenSearch. This synchronization ensures that when CRUD operations are performed on the data model, the corresponding changes are also reflected in OpenSearch.

We will begin by integrating hooks into our data models. Gorm offers the following hooks to handle specific events:

AfterCreate
AfterUpdate
AfterDelete

For example, let's consider the AfterDelete hook implementation below:

func (i *Ingredient) AfterDelete(txn *gorm.DB) error {
  client := index.IndexConnection
  err := client.Delete("ingredients", i.ID)
  if err != nil {
    log.Errorf("Unable to delete ingredient from OpenSearch: %v", err)
  }
}

In this code snippet, the AfterDelete hook is used to synchronize the deletion of an ingredient from the OpenSearch index. The hook retrieves the OpenSearch connection from index.IndexConnection and calls the Delete method to remove the corresponding ingredient using its ID. If any error occurs during the deletion process, an error message is logged.

By leveraging Gorm hooks, we can seamlessly integrate OpenSearch with our data models and ensure that database operations trigger synchronized actions in OpenSearch, maintaining data consistency between the two systems.

Summary

In this article, we have covered the essential topics that form the foundation of developing robust applications in Go. We started by creating a Go-based project skeleton using GitLab, Docker, and Go-based linting and build tools, ensuring a standardized and efficient development process. We then delved into creating a flexible application configuration using envconfig, enabling us to easily manage different deployment environments.

Next, we explored Gorm, a powerful ORM library, and integrated it into our application to provide seamless data persistence capabilities. With Gorm, we were able to interact with databases effortlessly, abstracting away the complexities of database operations.

Lastly, we discussed OpenSearch and learned how to index and search data efficiently, enhancing the search capabilities of our applications. By integrating OpenSearch, we empowered our applications with robust search functionality, enabling users to find relevant information quickly.

Building upon the knowledge gained from this post, we are now ready to take the next step in our journey. In the upcoming article, we will implement a web server using Gin-Gonic, a fast and flexible HTTP framework for Go. With Gin-Gonic, we will be able to create a powerful and scalable API layer, seamlessly integrating our previous topics into a comprehensive application architecture.

By combining the concepts of project structure, application configuration, data persistence with Gorm, and search capabilities with OpenSearch, we have established a solid foundation for developing sophisticated applications in Go. Join us in the next post as we continue to leverage the strengths of Go's ecosystem and build powerful applications with Gin-Gonic as our web server framework.

References

gorm documentation - https://gorm.io
gormigrate - https://github.com/go-gormigrate/gormigrate
Creating an Opinionated GraphQL server with Go - Part 3 - https://dev.to/cmelgarejo/creating-an-opinionated-graphql-server-with-go-part-3-3aoi
The Go client for Elasticsearch: Working with data - https://www.elastic.co/blog/the-go-client-for-elasticsearch-working-with-data

OpenSearch Snapshots with Min.io

mikeyGlitz — Mon, 22 May 2023 17:53:13 +0000

Retaining data is a critical responsibility for any business that stores and manages customer information. Data loss or corruption can arise from various factors, including hardware failures and human errors. The risks associated with data loss can be mitigated by implementing regular backups. OpenSearch, similar to many other databases, manages backups through the use of snapshots. By leveraging snapshots, OpenSearch can create secure point-in-time copies of data.

In our setup, we will utilize Min.io as an interface for OpenSearch to connect to storage. Additionally, we will establish a snapshot repository to facilitate backups.

Installing Min.io

Min.io is a file storage server that exposes files as HTTP URLs using the REST protocol. The service is cloud-native and can be deployed on various service providers, such as Docker, Kubernetes, OpenShift, EKS, and GCP. It can be scaled to ensure high availability and fault tolerance, which is crucial for OpenSearch to have reliable access to its snapshots and backups. Additionally, Min.io supports the Amazon AWS S3 access protocols, allowing for secure file storage with fine-grained access controls.

OpenSearch has native support for taking snapshots leveragingthe S3 protocol.

We will be deploying Min.io onto Kubernetes using the Helm chart.
The Helm chart will be deployed using an Ansible playbook.
To avoid the tutorial becoming to lengthy, high-availablilty configurations
will be left out. Min.io will be a standalone single-node cluster.

- name: Deploy Min.io
  kubernetes.core.heml:
    release_name: storage
    release_namespace: opensearch
    chart_ref: minio
    chart_repo_url: https://charts.min.io
    wait: true
    values:
      root_user: "{{ minio_user }}"
      root_password: "{{ minio_password }}"
      replicas: 1
      mode: standalone
      persistance:
        enabled: false
      resources:
        requests:
          memory: 512mi
      buckets:
        - name: opensearch
          policy: none
          purge: false
      svcaccts:
        - accessKey: opensearch
          secretKey: "{{ opensearch_secret_key }}"
          user: console
          policy:
            statements:
              - resources:
                  - arn:aws:::opensearch/*
                actions:
                  - s3:GetObject
                  - s3:PutObject
                  - s3:DeleteObject
              - resources:
                  - arn:aws:::opensearch
                actions:
                  - s3:ListBucket

The Ansible playbook creates a single-node Min.io server, which is suitable for local development purposes such as a test lab environment. The Helm chart provisions a bucket called opensearch. This opensearch bucket will serve as the storage location for our OpenSearch snapshots.

The Helm chart also deploys a service account called opensearch. This opensearch service account will be utilized by the OpenSearch service to log into Min.io using the generated access key and secret key. Min.io will leverage the opensearch service account to control access to the opensearch bucket, allowing only the actions specified in the policy to be performed by the service account.

The opensearch service account has the ability to utilize ListBucket to discover the opensearch bucket. Additionally, it can create and update objects on the bucket using PutObject, retrieve objects from the bucket using GetObject, and delete items from the bucket using DeleteObject.

In the next section we will be setting up a Helm chart through Ansible to provision the OpenSearch cluster.

Installing OpenSearch

Before we can set up our OpenSearch cluster, we need to store our Min.io credentials into a secret so that they may be consumed by the Helm chart and provided in the OpenSearch configuration when the OpenSearch cluster gets provisioned.

- name: Deploy OpenSearch Minio Secrets
  kubernetes.core.k8s:
    state: present
    definition:
      apiVersion: v1
      kind: Secret
      metadata:
        name: opensearch-storage-secrets
        namespace: opensearch
      data:
        s3.client.default.access_key: "{{ 'opensearch' | b64encode }}"
        s3.client.default.secret_key: "{{ opensearch_secret_key | b64encode}}

With the secret set up, we deploy the OpenSearch cluster:

- name: Deploy OpenSearch Cluster
  kubernetes.core.helm:
    release_name: index
    release_namespace: opensearch
    chart_ref: opensearch
    chart_repo_url: https://opensearch-project.github.io/helm-charts
    values:
      replicas: 1
      minimumMasterNodes: 0
      config:
        opensearch.yml: |
          s3.client.default:
            endpoint: "http://storage-minio.opensearch:9000
            protocol: http
            path_style_access: true
      keystore:
        - secretName: opensearch-storage-secrets
      plugins:
        enabled: true
        installList: ["repository-s3"]

⚠️ Sections of the configuration have been omitted.
The ommision is to highlight the configurations which
are specific to configuring OpenSearch to work with Min.io
For a full configuration reference, please consult
The Helm chart values
and the opensearch.yml values.

The Ansible play mentioned above provisions a single-node OpenSearch cluster. The opensearch.yml configuration section corresponds to the primary configuration file for OpenSearch, also known as opensearch.yml. The s3.client.default configuration sets up the default S3 client, which will be utilized to establish a connection with Min.io.

The keystore section connects to the Kubernetes Secret that was created earlier. The keystore details include the access key and secret key, which are used to pass credentials to Min.io.

The plugins section instructs the OpenSearch Helm chart to install the repository-s3 plugin

Setting up the Snapshot Repository

With OpenSearch deployed, the configuration of the snapshot repository can be initiated. An OpenSearch snapshot repository serves as a mechanism to store and manage backups (snapshots) of an OpenSearch cluster's data and configuration. Snapshots are crucial for capturing the data stored within OpenSearch indices, enabling point-in-time recovery and restoration in the event of system failures or data loss scenarios.

To set up a snapshot repository, we will have to port-forward our OpenSearch Dashboards service.

kubectl port-forward -n opensearch services/index-ui-opensearch-dashboards 5601:5601

OpenSearch Dashboards will be accessible at https://localhost:5601.

OpenSearch 2.x introduces a snapshot management dashboard which can be accessible through the hamburger menu at the top right and then click snapshot management.
Click on Repositories from the sidebar menu.
There will be a Create repository button which can be used to create a new snapshot repository.
We will create a repository with the following information:

Repository Name: minio
Repository Type: Custom configuration

Under Custom configuration we will enter the following JSON string:

{
    "type": "s3",
    "settings": {
        "bucket": "opensearch",
        "base_path": "snapshots"
    }
}

Click Add. The snapshot repository will be created.

Now that the snapshot repository is created, you should be able to create snapshots.

For the case of our example, I've created a snapshot of the .opendistro_security index.

With the snapshot created, the snapshot will be visible from the Min.io console

Improving the Deployment for Production Readiness

The installation procedure described in the previous sections outlined how to set up Min.io and OpenSearch for snapshot backups using the S3 plugin. To make the installation production-ready, several enhancements can be implemented:

Enable network-based persistence volume (e.g., NFS) for the Min.io deployment.
Scale up Min.io to a multi-node cluster configuration.
Generate server keys for Min.io to encrypt objects at rest.
Enable SSL configuration for Min.io.
Configure a snapshot policy in OpenSearch to capture snapshots at regular intervals.
Install service monitors to regularly send usage metrics to a data store (e.g., Prometheus).

References

Connecting OpenSearch to Keycloak

mikeyGlitz — Tue, 14 Feb 2023 14:40:52 +0000

Having developed multiple web applications, search is an important
capability for a lot of the projects I've worked on. Earlier in my
career, search capabilities were provided by using simple query searches

i.e.:

SELECT * from users where name like '%term%';

As you could imagine, this is not the most accurate search and can get
very complicated when it comes to attempting to search through multiple
fields.

A database product, ElasticSearch was developed based on the Lucene index
which enables functionality such as fuzzy searches, and ranks search
results based on partial matches.

In 2021, OpenSearch was introduced as a result of ElasticSearch no
longer using the Apache license. OpenSearch is a fork of ElasticSearch
7.10 - the last version of ElasticSearch to use the Apache 2.0 license.
OpenSearch is backed by the Amazon Web Services and the OpenSearch community.

As a result of the AWS backing, my organizations have shifted to using
OpenSearch at work. With the exposure at work, I've begun to leverage
OpenSearch in my personal projects.

This tutorial demonstrates how to deploy an OpenSearch cluster with
OpenID authentication using Keycloak on a Kubernetes cluster.
Deployments are shown using Ansible tasks.

Managing Certificates

Generally speaking it is good practice to secure communication with
your database. Before we can deploy OpenSearch, we would first have
to create certificates. cert-manager is a helpful tool which can be deployed
onto Kubernetes to create and manage certificates.
With our applications being disbursed across several different
namespaces, creating certificates will take place across multiple
stages:

Deploy Cert Manager
Deploy Trust Manager
Deploy the root certificate issuer
Generate a root CA using the root issuer
Create a cluster-issuer which will enable us to deploy certs in other namespaces

trust-manager will
enable us to share the root CA chain globally across all the different
namespaces so that we can use the public certificates to verify any
certificate which is derived from the root CA.

The following Ansible snippet details how to preform these steps.

- name: Deploy cert-manager
  kubernetes.core.helm:
    release_name: cert-manager
    release_namespace: cert-manager
    create_namespace: true
    wait: true
    chart_ref: cert-manager
    chart_repo_url: https://charts.jetstack.io
    values:
      installCRDs: true
- name: Deploy trust-manager
  kubernetes.core.helm:
    release_name: cert-manager-trust
    release_namespace: cert-manager
    create_namespace: true
    wait: true
    chart_ref: cert-manager-trust
    chart_repo_url: https://charts.jetstack.io
    values:
      installCRDs: true
- name: Create Root Issuer
  kubernetes.core.k8s:
    state: present
    definition:
      apiVersion: cert-manager.io/v1
      kind: Issuer
      metadata:
        namespace: cert-manager
        name: root-issuer
      spec:
        # Example uses self-signed
        # It is advisable to utilize a different kind of Issuer
        # for production
      selfSigned: {}
- name: Create Root CA Certs
  kubernetes.core.k8s:
    state: present
    definition:
      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        namespace: cert-manager
        name: root-ca
      spec:
        isCA: true
        # Omitted multiple values.
        # https://cert-manager.io/docs/usage/certificate/
        # for full Certificate spec
        secretName: root-ca
        privateLey:
          algorithm: ECDSA
          size: 256
        issuerRef:
          name: root-issuer
          kind: Issuer
          group: cert-manager.io
- name: Create Cluster Issuer
  kubernetes.core.k8s:
    state: present
    definition:
      apiVersion: cert-manager.io/v1
      kind: ClusterIssuer
      metadata:
        name: cluster-issuer
        namespace: cert-manager
      spec:
        ca:
          secretName: root-ca
- name: Create Root CA Chain bundle
  kubernetes.core.k8s:
    state: present
    definition:
      apiVersion: trust.cert-manager.io/v1alpha1
      kind: Bundle
      metadata:
        name: root-bundle
        namespace: cert-manager
      spec:
        sources:
          - secret:
              name: root-ca
              key: ca.crt
        target:
          configMap:
            key: root.ca.crt

Deploying Keycloak

Before we can authenticate OpenSearch against Keycloak, we'll need to
install Keycloak. The following Ansible snippet demonstrates how to
deploy Keycloak onto a Kubernetes cluster using the codecentric helm chart.

- name: Deploy Keycloak
  kubernetes.core.helm:
    wait: true
    release_name: keycloak
    release_namespace: keycloak
    chart_ref: keycloak
    chart_repo_url: https://codecentric.github.io/helm-charts
    values:
      extraEnv: |
        - name: KEYCLOAK_USER
          value: {{ keycloak_user }}
        - name: KEYCLOAK_PASSWORD
          value: {{ keycloak_password }}
        - name: PROXY_ADDRESS_FORWARDING
          value: "true"
- name: Provision admin role
  community.general.keycloak_role:
    auth_keycloak_url: "{{ keycloak_url }}"
    auth_realm: master
    auth_username: "{{ keycloak_user }}"
    auth_password: "{{ keycloak_password }}"
    validate_certs: false
    realm: "{{ realm_name }}"
    name: admin
    description: >-
      The admin role enables administrator privileges for any
      user which assigned to this role.
      The admin role will also map to the admin role in OpenSearch
- name: Provision readall role
  community.general.keycloak_role:
    auth_keycloak_url: "{{ keycloak_url }}"
    auth_realm: master
    auth_username: "{{ keycloak_user }}"
    auth_password: "{{ keycloak_password }}"
    validate_certs: false
    realm: "{{ realm_name }}"
    name: admin
    description: >-
      The readall role enables read-only privileges.
      readall will have read-only privileges in OpenSearch

Sections pertaining to the ingress and associated rules were left out
to keep the code snippet small. The above snippet provisions two
roles on our Keycloak IDp which will map directly to OpenSearch roles
with the same names.

A client will also need to be prepared so that OpenSearch may
authenticate with Keycloak as an OpenID backend. The below Ansible
snippet demonstrates how to set up the client using the
community.general.keylcloak_client plugin.

- name: Deploy OpenSearch Client
  community.general.keycloak_client:
    auth_keycloak_url: "{{ keycloak_url }}"
    auth_realm: master
    validate_certs: false
    auth_username: "{{ keycloak_user }}"
    auth_password: "{{ keycloak_password }}"
    client_id: opensearch-dashboards
    secret: "{{ os_client_secret }}"
    realm: "{{ realm_name }}"
    enabled: true
    direct_access_grants_enabled: true
    authorization_services_enabled: true
    service_accounts_enabled: true
    redirect_uris:
      # localhost:5601 is used as the OpenSearch-Dashboards URL
      - https://localhost:5601/*
    protocol_mappers:
      - name: opensearch-audience
        protocol: openid-connect
        protocolMapper: oidc-audience-mapper
        config:
          id.token.claim: "true"
          access.token.claim: "true"
          included.client.audience: opensearch-dashboards
      - name: opensearch-role-mapper
        protocol: openid-connect
        protocolMapper: oidc-usermodel-realm-role-mapper
        config: {
          "multivalued": "true",
          "userinfo.token.claim": "true",
          "id.token.claim": "true",
          "access.token.claim": "true",
          "claim.name": "roles",
          "jsonType.label": "String"
        }

direct_access_grants_enabled enables Keycloak direct access grants
which allows the client to authenticate users by directly using
username/password combos.

service_accounts_enabled is for future use which will enable the
Keycloak client to use service accounts (i.e. to authenticate applications).

authorization_services_enabled enables fine-grained authorization
support for a client.

In the protocol_mappers section, two protocol mappers are created.
The protocol mappers are utilized to map different items to fields on
the JWT which is returned once a user logs in. For OpenSearch
OpenID authentication, OpenSearch looks for a key to identify the
roles associated with a user account to map to an OpenSearch role
which will determine which functions that the user is allowed to access.

I'm not exactly sure if the audience-mapper is needed, but I left it
in there since I've used it for other authentication purposes
(i.e. express-passport)

Deploying OpenSearch

There are two different ways to deploy OpenSearch to Kubernetes:
The OpenSearch Operator or through using the OpenSearch Helm Chart.
This guide will provide the steps for a helm-based deployment.

There are two main configuration files which connect OpenSearch to an
OpenID provider, in this case, Keycloak: opensearch-security/config.yml
and opensearch-dashboards.yml.
opensearch-security/config.yml connects the OpenSearch security plugin
and opensearch_dashboards.yml connects the OpenSearch Dashboards UI.

opensearch-security/config.yml

_meta:
  type: "config"
  config_version: 2
config:
  dynamic:
    authz: {}
    authc:
      basic_internal_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: intern

      openid_auth_domain:
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: openid
          challenge: false
          config:
            openid_connect_idp:
              enable_ssl: true
              verify_hostnames: false
              pemtrustedcas_filepath: /usr/share/opensearch/config/root-ca/root.ca.crt
            subject_key: preferred_username
            roles_key: roles
            openid_connect_url: "{{ keycloak_url }}/auth/realms/pantry/.well-known/openid-configuration"
        authentication_backend:
          type: noop

_meta contains the metadata configuration which is boilerplate
designations for the file as a configuration file. config contains
the configuration that the OpenSearch Security plugin will use to
configure how authentication and authorization will be managed in
OpenSearch.

authc contains authentication backends.
The authentication backends which are provided by the configuration
snippet are basic, and openid. The basic authentication is used for the
built-in users (i.e. admin, kibanaserver).
The openid backend is used to connect to Keycloak.

openid_connect_url points to the OpenID provider URL. OpenSearch
expects the URL to point to a .well-known/openid-configuration endpoint
which is used to fetch the metadata configuration.

roles_key points to the field on the JWT which is issued by the
OpenID provider where the realm roles are set. The roles are used to
map the OpenID realm roles to OpenSearch roles for fine-grained
access control.

subject_key points to the field on the JWT which is issued by the
OpenID provider where the username is located. This field is usually
username, preferred_username, or email.

openid_connect_idp contains the details which are used to provide
TLS/SSL values to the OpenID server. enable_ssl tells OpenSearch
that the OpenID connection is over SSL. verify_hostnames instructs
OpenSearch whether or not to verify the HTTP hostname against the
hostname provided by the certificate. pemtrustedcas_filepath
contains the file path to the root Certificate Authority certificate file
which is used to verify the certificate. Certificate verification prevents
man-in-the-middle attacks or certificate impersonation.

Based on the OpenSearch documentation, setting the authentication_backend
to noop is required because JSON web-tokens already contain the
required information to verify the request.

opensearch_dashboards.yml

server:
  name: dashboards
  host: 0.0.0.0
  ssl:
    enabled: true
    key: /usr/share/dashboards/certs/tls.key
    certificate: /usr/share/dashboards/certs/tls.crt
opensearch_security:
  auth.type: openid
  openid:
    connect_url: https://{{ domain_name }}/auth/realms/pantry/.well-known/openid-configuration
    base_redirect_url: https://localhost:5601
    client_id: opensearch-dashboards
    client_secret: {{ os_client_secret }}
    scope: openid profile email
    header: Authorization
    verify_hostnames: false
    root_ca: /usr/share/dashboards/root-ca/root.ca.crt
    trust_dynamic_headers: "true"
opensearch:
  requestHeadersAllowlist: ["Authorization", "security_tenant"]
  hosts: [ "opensearch-cluster-master" ]
  username: "kibanaserver"
  password: "kibanaserver"
  ssl:
    certificateAuthorities: /usr/share/dashboards/certs/ca.crt

client_id contains the OpenID client ID which OpenSearch Dashboards
will use to authenticate with Keycloak.
client_secret contains the client secret which OpenSearch Dashboards
will use to authenticate with Keycloak.
root_ca contains the root CA path for OpenSearch Dashboards to verify
the OpenID provider (Keycloak) certificate.
verify_hostnames instructs OpenSearch whether or not to verify the
HTTP hostname against the hostname provided by the certificate.
scope contains the scopes which are used to determine the identity
from the token issued by the OpenID provider.
auth.type instructs OpenSearch Dashboards to use OpenID for user logins.

Refer to the OpenSearch and the OpenSearch Dashboards for all the supported values for OpenSearch and OpenSearch Dashboards deployment.

The configuration in opensearch-security/config.yml will be added to
the OpenSearch helm chart like so:

- name: Deploy OpenSearch
  kubernetes.core.helm:
    release_name: opensearch
    release_namespace: opensearch
    chart_ref: opensearch
    chart_repo_url: https://opensearch-project.github.io/helm-charts
    values:
      replicas: 1
      minimumMasterNodes: 0
      securityConfig:
        enabled: true
        config:
          dataComplete: false
          data:
            config.yml: |-
              {{ opensearch_security_config }}

I've omitted values for opensearch.yml, volumes, and volume mounts.
Volumes and volume mounts will be required for security configuration
to mount the SSL certificates.

The configuration in opensearch_dashboards.yml can be added to the
opensearch-dashboards release like so:

- name: Deploy OpenSearch Dashboards
  kubernetes.core.helm:
    release_name: opensearch-dashboards
    release_namespace: opensearch
    chart_ref: opensearch-dashboards
    chart_repo_url: https://opensearch-project.github.io/helm-charts
    values:
      config:
        opensearch_dashboards.yml: |-
        {{ opensearch_dashboards_config }}

Once the deployments are complete, Dashboards should be accessible
through Kubernetes port forwarding

kubectl port-forward services/opensearch-dashboards 5601:5601 -n opensearch

The backends are visible through the OpenSearch Security page.

References

Keycloak OpenSearch - https://github.com/cht42/opensearch-keycloak
OpenID Backend Documentation - https://opensearch.org/docs/latest/security/authentication-backends/openid-connect/
Configuring OpenSearch 2.x With OpenID - https://www.rushworth.us/lisa/?p=9370
OpenSearch Dashboards helm chart - https://github.com/opensearch-project/helm-charts/blob/main/charts/opensearch-dashboards/values.yaml
OpenSearch helm chart - https://github.com/opensearch-project/helm-charts/blob/main/charts/opensearch/values.yaml

Go API Project Set-Up

mikeyGlitz — Fri, 23 Dec 2022 20:18:49 +0000

Have you ever wondered what goes into creating a production-ready
workflow? Have you ever considered what kind of conventions to follow
when you're beginning a new project?

Recently I had the pleasure to experiment with producing a Go based
web server to serve an API. Here are some lessons I've learned along
the way.

Conventions

As a general practice for working with any language, it is best to adhere
to conventions which are commonly recognized in the language's community
as much as possible. Before a single line of code is written, I set up
my project with the baseline tools that I think I'll need during the
development process.

Project Structure

In the case of a Go-based project my starting point is the project
layout which is documented in the golang-standards Project Layout.

My project layout adheres to the following structure:

Path	Description
`cmd`	Contains the `main` package which is the primary entry point for my application
`pkg`	Contains public packages which are intended to be utilized by other projects (i.e. shared struct declarations)
`test`	Integration tests (i.e. testcontainers)
`internal`	Any code which not intended to be utilized outside of the application or module package

While the go-lang standards layout recommends additional folders like
api and web for the purposes of serving a web API, since this
project will leverage GraphQL to provide API functionality, I won't
have separate folders for JSON/OpenAPI specifications nor will I have
a separate folder for hosting endpoints which will later be explained
in a future gqlgen section.

Also, the convention for which unit tests are written in go with the
test package. Unit test files need to be siblings to the implementation
files in their respective hierarchies. Due to this sibling
relationship, I leverage the test folder specifically for integration
testing since the integration tests don't necessarily have a dependency
on the internal state of the packages themselves.

Since I develop using GO111MODULE=on, I do not have need for a vendor folder.

go.mod is a file which contains the dependency list for all the 3rd
party go modules I'll be using to develop my project.

tools.go is a file which contains all the CLI tools that my project depends on.

To set up go.mod I use the following command and answer all the prompts:

go mod init

Enforcing Style

Generally when working on a project, I prefer to enforce a recognized style-guide.

For Java, I use the Google style-guide with checkstyle.
For Kotlin, I use the Pinterest style-guide with ktlint.
For Python, I use flake8 to enforce styles.
For Typescript/Javascript I use eslint with the AirBnB style guide.

For Go, I use golangci-lint.

I add the following line to my tools.go file under import:

_ "github.com/golangci/golangci-lint/cmd/golangci-lint"

Mocks

Unit tests are leveraged to test individual units of code. As such it
is not recommended for a developer to scaffold entire dependencies for
the sake of testing a single object. Due to the way Go's specific
implementations work, I've learned over time to declare interfaces
for a lot of the structs that I use in Go. Interfaces not only define
a contract for which struct-based implementations should adhere,
but they also provide a mechanism for which struct methods can be
mocked. While I've experimented with the mock package in
testify, I've come to prefer the mock functionality which is
provided by mockgen.

Mockgen is a utility which attaches to the go generate command.
Mockgen will generate mocked structures to emulate behavior for
interfaces in your code so that you can short circuit behavior to
assist in unit testing.

To generate mocks, I add a line to the top of the file under the
package declaration to run a generate script I.E.:

//go:generate go run github.com/golang/mock/mockgen@v1.6.0 -destination=./mocks/mock_index.go -package=mocks gitlab.com/pantry-organize/pantry-api/internal/index OpensearchConnection

The generate line has the following sections:

go run - tells go to run the script which is provided in the next argument
github.com/golang/mock/mockgen@1.6.0 - the module which you want go to run
-destination=[relative-path] - Where you want mockgen to produce the files relative to the running directory
-package=[package-name] - The package name you want to create the mocked implementation in
[package-path] - The package that you want to mock
[interface-name] - The interface that you want to mock

I add the following line to my tools.go file under import:

_ "github.com/golang/mock/mockgen"

Test Reporting

Since my project is hosted on GitLab, I have to make some tool
integration considerations when it comes to test reporting.
The kind of test reporting we're interested in reporting to GitLab are
test reports for which tests passed and which tests failed as well as
reports for which lines of code are covered by tests, also known as coverage.

The go test command does not support test reporting out of the box.
To generate a test report, we have to use
gotestsum to generate a JUnit report.
We add gotestsum to our tools.go file:

_ "github.com/gotestyourself/gotestsum"
```



To report coverage to GitLab we have to use a tool which can convert
the coverage report from `go test` to Cobertura.
We'll pull in [gocover-cobertura](https://github.com/t-yuki/gocover-cobertura) to generate our coverage report.

We add the following line to our `tools.go` file:



```go
_ "github.com/t-yuki/gocover-cobertura"
```



### Makefile

Every language ecosystem that I've worked with has had some mechanism
for which to run scripts. I leverage scripting as glue for running
various build tasks.

In the case of Java, there's usually a maven plugin (i.e. checkstyle,
spring, shadow) which I can use to accomplish a particular build phase,
or leverage Gradle for the ability to run a custom build script.

For Node-based projects, you can specify a build script using the
`scripts` block in `package.json`.

For Go, the general convention is to leverage good ol' Makefile.

Makefile is a tool which can be used to compile source files. My first
experience using make was for C/C++ projects in university. Make can
also be used to run command-line scripts which is what we'll be using
in the case of go.

Generally speaking my Makefiles consist of the build steps that I'm
familiar with in the case of other languages:

- `clean` - Removes files which were either compiled binaries, generated code, or otherwise not tracked in version control
- `lint` - Scans an analyzes code for style guide adherence or syntactic correctness
- `test` - Preforms unit tests
- `integration` - Preforms all tests: unit and integration
- `report` - Prepare test reports (pass/fail and test coverage)
- `build` - Compiles source code into a single binary

In the case of Go, for a majority of the language's history there was
no concept of generics in the language. A stop-gap measure to create
generic-like functionality in Go was to leverage tools to auto-generate code.
In Go projects I usually create a `gen` task in `Makefile` which is
used for the express case of running `go generate ./...`.

Here is an example Makefile:



```Makefile
clean:
    @rm -rf ./internal/dataloaders/*_gen.go
    @rm -rf ./internal/graph/model/models_gen.go
    @rm -rf ./internal/graph/generated
    @rm ./pantry-api

gen:
    @go generate ./...

lint: gen
    @go run github.com/golangci/golangci-lint/cmd/golangci-lint run

test: gen
    @go test -short ./...

integration: lint
    @go test ./... -coverprofile=coverage.txt -covermode count
    @go run gotest.tools/gotestsum --junitfile report.xml --format testname
    @go run github.com/t-yuki/gocover-cobertura < coverage.txt > coverage.xml

report: integration
    @go tool cover -html coverage.txt -o coverage.html

build: gen
    @go build -ldflags "$(LDFLAGS)" -o pantry-api ./cmd/pantry-api
```



## Containerization

As a developer, you generally want to take your user experience into
consideration. When the application you're developing is a web server,
one of your users will be the operations person or the site administrator
who will ensure that your application runs and functions properly in
its deployment environment.

While a lot of legacy applications run directly on the host environment,
the use of containerization is more prevelant than ever. With a
container, it is possible to the deployment time and the installation
and configuration of your application. With a container, your
application deploys inside a self-contained environment (a lightweight
VM) and usually contains the minimal pre-configuration (more on this
in subsequent sections) which is required to run.

[Docker](docker.com) is the most widely used container engine to date.
For this example, we will be deploying our application using Docker
containers. To create a Docker container you must first create a
`Docker` file. When creating a Dockerfile, there are certain
considerations which may be taken into account as part of your build
process. You can either copy your application package or binary
into your container pre-built, or you can create a build step within
your container context and copy the binary from the build stage in
`Dockerfile` to a destination runtime environment.

As a general rule of thumb, I prefer to build my applications within
the `docker build` process. Organizaing `Dockerfile` into a `build` stage
and a `ship` stage provides the added benefit of multi-architecture builds.

With the increasing popularity of ARM-based architectures (as seen in
the Apple Silicone Macs, Raspberry Pi, and AWS Graviton instances),
supporting both Intel and ARM platforms may be a consideration for
increased application portability as well as cost savings.

In my example, I will demonstrate releasing a multi-stage Docker image
which is compatible with `docker buildx` for releasing a multi-
architecture image. This example will contain a `build` stage for
compiling a Go application and a `ship` stage for the actual runtime
environment once the container starts



```Dockerfile
# TARGETPLATFORM specifies the build/runtime environment i.e. ARM/AMD64
# This argument defaults to linux/amd64
FROM --platform=${TARGETPLATFORM:-linux/amd64} golang:1.19-alpine3.16 as build

ARG TARGETPLATFORM
ARG TARGETOS
ARG TARGETARCH

# Enable golang modules
ENV CGO_ENABLED=1
ENV GO111MODULE=on

# LDFLAGS are used to set values for signing your go binary
ENV LDFLAGS=

# Install required C packages in Alpine
RUN apk update \
    && apk add --no-cache gcc g++ libstdc++ make ca-certificates binutils-gold git
RUN update-ca-certificates

# Create a temporary /build folder and use it to compile the runtime binary
RUN mkdir /build
COPY . /build
WORKDIR /build
RUN export GOOS=${TARGETOS} && \
    GOARCH=${TARGETARCH} && \
    make test && \
    make build

# Set up the ship build context for the final deployment
FROM --platform=${TARGETPLATFORM:-linux/amd64} alpine:3.14 as ship

# Create a folder from which the application will run
RUN mkdir -p /home/app
# Copy the runtime binary from the build stage
COPY --from=build /build/pantry-api /home/app/server
# Add execute permission to the runtime binary
RUN chmod +x /home/app/server

# Docker best practice: NEVER EVER RUN YOUR CONTAINER AS A ROOT USER
# Here we create an application user which is a system account for the
# express purpose of running our application
RUN adduser -S appuser
RUN addgroup -S appuser && adduser appuser appuser
# Assign ownership of /home/app and all of its contents to the appuser account
RUN chown -R appuser:appuser /home/app

WORKDIR /home/app

# Anything beyond this statement will be executed as appuser
USER appuser

# Expose 8080 because that is the port which the web server is listening on
EXPOSE 8080

# This is the container start-up command which is run when the container is started
CMD ["./server"]
```



## CI/CD Pipeline

Pipelines enable automated builds of your application to kick off
whenever an update is pushed to version control. There are several
pipeline providers which are available at multiple price points:
Bitbucket Pipelines, GitHub Actions, Travis, Circle CI, just to
name a few.

Ever since the Microsoft Acquisition of GitHub, my preference has
leaned more towards using GitLab as my version control host.
GitLab offers the same functionality as GitHub as well as a self-hosted
option if you don't feel comfortable with hosting your code on
the Internet. GitLab offers a better (IMO) experience with its CI/CD pipelines.

To set up CI/CD pipelines in GitLab, you'll need to add a `.gitlab-ci.yml`
file. `.gitlab-ci.yml` specifies the build phases under the `stages` block.
The project builds occur under two stages: `test` and `release`.



```yaml
stages:
  - test
  - release
```



The next block in `.gitlab-ci.yml` is the services block. Since our
tests use [testcontainers](https://golang.testcontainers.org) package and we're pushing a
docker container onto [Dockerhub](https://hub.docker.com), we will need to specify a `services`
block next. Services will enable our pipeline to leverage Docker-in-Docker DinD.



```yaml
services:
  - name: docker:dind
    command:
      - "--tls=false"
```



Test is a stage which triggers the `make test` command which will run the go code
generators, lint the code, run the unit tests, and run the integration tests. 

The next block is the `sast` block which will pull in GitLab's standard SAST template.
The SAST template will scan the Go code for vulnerabilities as well as preform static
analysis against the code.



```yaml
sast:
  stage: test
include:
  - template: Security/SAST.gitlab-ci.yml
```



The next block is `go-test` which will be used to explicitly run the go tests.



```yaml
go-test:
  variables:
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ""
    DOCKER_DRIVER: overlay2
    CGO_ENABLED: "1"
    GO111MODULE: "on"
  image: golang:1.19-alpine3.16
  stage: test
  before_script:
    - apk update && apk add --no-cache ca-certificates make binutils-gold gcc g++ libstdc++ git
    - update-ca-certificates
  script:
    - make integration
    - make build
  artifacts:
    reports:
      junit: report.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml
```



The `artifacts` block specifies paths for GitLab to capture artifacts.
These artifacts come in the form of test reports. Our project creates
a pass/fail report which uses the JUnit test report format, `report.xml`,
and a `coverage.xml` file which is consumed by cobertura to upload a test
coverage report to GitLab.

The next stage is the release stage. Release steps encompass everything which
is needed to release the application. Release in this context refers to publishing
a release to the GitLab repository and publishing the Docker image to Docker Hub.

This block creates the release tag in the GitLab repository



```yaml
release_job:
  stage: release
  image: registry.gitlab.com/gitlab-org/release-cli:latest
  rules:
    - if: $CI_COMMIT_TAG
      when: never
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  script:
    - echo "running release_job for $TAG"
  release:
    tag_name: 'v0.$CI_PIPELINE_IID'
    description: '$CI_COMMIT_MESSAGE'
    ref: '$CI_COMMIT_SHA'
```



This block runs the docker image build and pushes it out to Docker Hub



```yaml
docker-build:
  # Use the official docker image.
  image: docker:latest
  stage: release
  before_script:
    - echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER --password-stdin
  # Default branch leaves tag empty (= latest tag)
  # All other branches are tagged with the escaped branch name (commit ref slug)
  script:
    - |
      if [[ "$CI_COMMIT_BRANCH" == "$CI_DEFAULT_BRANCH" ]]; then
        tag=""
        echo "Running on default branch '$CI_DEFAULT_BRANCH': tag = 'latest'"
      else
        tag=":$CI_COMMIT_REF_SLUG"
        echo "Running on branch '$CI_COMMIT_BRANCH': tag = $tag"
      fi
    - docker buildx create --use
    - >
        docker buildx build --push 
        --platform linux/arm/v7,linux/arm64/v8,linux/amd64 
        --build-arg CI_COMMIT_SHA=${CI_COMMIT_SHA} 
        --build-arg CI_COMMIT_AUTHOR="${CI_COMMIT_AUTHOR}"
        --build-arg CI_COMMIT_TIMESTAMP="${CI_COMMIT_TIMESTAMP}" 
        --build-arg CI_REPOSITORY_URL="${CI_REPOSITORY_URL}" 
        --build-arg CI_COMMIT_BRANCH=${CI_COMMIT_BRANCH} 
        --build-arg CI_JOB_STARTED_AT="${CI_JOB_STARTED_AT}" 
        --tag "$CI_REGISTRY_IMAGE${tag}" .
```



This block pushes a readme out to the docker repo in Docker Hub



```yaml
readme:
  stage: release
  image:
    name: chko/docker-pushrm
    entrypoint: ["/bin/sh", "-c", "/docker-pushrm"]
  script: "/bin/true"
  variables:
    DOCKER_USER: $CI_REGISTRY_USER
    DOCKER_PASS: $CI_REGISTRY_PASSWORD
    PUSHRM_TARGET: docker.io/$CI_REGISTRY_IMAGE
    PUSHRM_FILE: $CI_PROJECT_DIR/README.md
```



In the next post we'll be going over how to build out the service

## References

- Release CI/CD Examples - https://docs.gitlab.com/ee/user/project/releases/release_cicd_examples.html#create-a-release-when-a-commit-is-merged-to-the-default-branch)
- Go Project Layout - https://github.com/golang-standards/project-layout
- Go test coverage visualization - https://docs.gitlab.com/ee/ci/testing/test_coverage_visualization.html#go-example
- Go unit test results - https://docs.gitlab.com/ee/ci/testing/unit_test_report_examples.html#go
- gotestsum - https://github.com/gotestyourself/gotestsum
- golangci lint - https://golangci-lint.run/
- gocover-cobertura - https://github.com/t-yuki/gocover-cobertura
- multiarch builds with Docker - https://www.docker.com/blog/multi-arch-build-what-about-gitlab-ci/

Setting up a GitLab Runner

mikeyGlitz — Fri, 23 Dec 2022 16:04:01 +0000

Recently I've become aware of new disruption which is occurring in computing hardware.
ARM based architectures are exploding in adoption and becoming more mainstream.
This change in CPU architectures brings advantages in energy efficiency and when it comes to
cloud computing, cost reduction.

Slowly over the course of the past few weeks I've been adapting my build pipelines to support
multi architecture builds. The thought was that I wanted to continue supporting Intel-based CPU
architecture, but I wanted to add support for ARM and ARM64 architectures.
What I quickly began to find out is that supporting multi-architecture Docker builds comes at a
pretty significant cost in supporting the CI pipelines. Suddenly I was receiving emails from
GitLab informing me that my builds were timing out after running for over an hour.

Pipeline Runners

Since GitLab is an open-source platform, GitLab enables users to provide their own compute resources
to run builds in addition to the shared runners which are already provisioned for GitLab.com.
With a hosted runner, I can allocate more resources than I would have at my disposal using one of
the many shared runners. Since I already had a Kubernetes cluster running locally, I built an Ansible
playbook so that I could deploy the runner to my local cluster using the
GitLab Runner Helm Chart

Getting Your Runner Token

When you create a repository, GitLab provides you with a token to register runners to your project.
This token ensures that only your CI jobs run on your runners. You can access your token from
Settings > CI/CD > Runners.

Setting Ansible Variables

With the token in mind, the other information which is required by the Helm chart is the URL for
your GitLab sever. Bear in mind that GitLab can be a hosted service which you keep on your internal
network, it may also be cloud hosted, but for the sake of this tutorial, I will be using https://gitlab.com
as my GitLab URL.

At the beginning of our playbook.yml I set the following vars block:

  vars:
    gitlab_token: # Token to connect to the GitLab server -- this is provided when you run the playbook
    gitlab_server: https://gitlab.com

Securing Your Token

While the Helm chart does have a variable to pass in the token, as a general rule of thumb I prefer to store
sensitive information in Kubernetes Secrets.

We create a secret in Kubernetes using the following Ansible task:

  - name: Deploy Runner connection Token
    kubernetes.core.k8s:
      state: present
      definition:
        apiVersion: v1
        kind: Secret
        metadata:
          name: gitlab-runner-token
          namespace: gitlab
        data:
          runner-registration-token: "{{ gitlab_token | b64encode }}"
          runner-token:

b64encode is an in-line function within ansible which base64 encodes the values passed into it.
Secrets in Kubernetes can never be stored in plaintext and are required to be base64 encoded.

Deploying the Helm Release

With our runner registration token deployed in our Kubernetes environment, we'll be able to install
the Helm release into our gitlab namespace.

  - name: Deploy GitLab Runner
    kubernetes.core.helm:
      release_name: ci-runner
      chart_ref: gitlab-runner
      chart_repo_url: https://charts.gitlab.io
      release_namespace: gitlab
      release_values:
        gitlabUrl: "{{ gitlab_server }}"
        rbac:
          create: true
        runners:
          secret: gitlab-runner-token
          config: |
            [[runners]]
              [runners.kubernetes]
                privileged=true
                image="docker:20-dind"
              [[runners.kubernetes.volumes.host_path]]
                name="docker"
                read_only=true
                mount_path="/var/run/docker.sock"
                host_path="/var/run/docker.sock"

This Helm configuration indicates to use the gitlab_server variable set in the vars section.
The runner will also be configured to register using the secret gitlab-runner-token which contains
the GitLab runner token set in the Ansible variable gitlab_token which we will pass in as a command-line
option when we run ansible-playbook.

A section of particular note is the configuration provided by the config section of the release_values.
The configuration provided in config instructs the gitlab runner to run as a privileged pod in Kubernetes.
The elevated privileges are required to allow the pod to run as Docker-in-Docker (DinD).
I use dind to run integration tests with testcontainers as well as the multi-arch Docker builds mentioned in the
introduction.

The Ansible play is detailed in the full snippet below:

  vars:
    gitlab_token: # Token to connect to the Gitlab server
    gitlab_server: https://gitlab.com
  tasks:
    - name: Ensure namespace exists
      kubernetes.core.k8s:
        state: present
        definition:
          apiVersion: v1
          kind: Namespace
          metadata:
            name: gitlab
    - name: Deploy Runner conneciton Token
      kubernetes.core.k8s:
        state: present
        definition:
          apiVersion: v1
          kind: Secret
          metadata:
            name: pantry-runner-token
            namespace: gitlab
          data:
            runner-registration-token: "{{ gitlab_token | b64encode }}"
            runner-token: ""
    - name: Deploy GitLab
      kubernetes.core.helm:
        release_name: pantry
        chart_ref: gitlab-runner
        chart_repo_url: https://charts.gitlab.io
        release_namespace: gitlab
        release_values:
          gitlabUrl: "{{ gitlab_server }}"
          rbac:
            create: true
          runners:
            secret: pantry-runner-token
            config: |
              [[runners]]
                [runners.kubernetes]
                  privileged=true
                  image="docker:20-dind"
                [[runners.kubernetes.volumes.host_path]]
                  name="docker"
                  mount_path="/var/run/docker.sock"
                  host_path="/var/run/docker.sock"
                  read_only=true

In my version of the Ansible playbook, I have it hard-coded to run on localhost since I only
want to deploy the Helm Release to my local Kubernetes cluster. I deploy the playbook with the
following command:

ansible-playbook -e gitlab_token="<your-gitlab-token-here>" gitlab.yml

Project Configuration

To run jobs using your registered runners, the most reliable way is to disable shared runners.
In Settings > CI/CD > Runners you may find the disable toggle:

I'm able to run my CI/CD pipeline with the dind service in .gitlab-ci.yml.

services:
  - name: docker:dind
    command:
      - "--tls=false"

Setting --tls=false is a fix for Testcontainers. Newer versions of Docker starting with 20 induce
a delayed start-up time which will cause Testcontainers to fail to initialize.

For my job which runs Testcontainers, I'm able to run using my local runner in the following configuration.
The provided example is for Golang

go-test:
  variables:
    DOCKER_TLS_CERTDIR: ""
    DOCKER_DRIVER: overlay2
    CGO_ENABLED: "1"
    GO111MODULE: on
  image: golang:1.19-buster
  stage: test
  before_script:
    - apt-get update
    - DEBIAN_FRONTEND=noninteractive apt-get install make git ca-certificates
    - update-ca-certificates
  script:
    - make integration
    - make build

The Testcontainers instructions mention to add a variable DOCKER_HOST: tcp://docker:2375 to the variables section.
I omitted this variable due to leaving it in caused the Testcontainers tests to fail with a TCP timeout due to being unable
to find the docker socket. I believe this has largely to do with the docker socket on my local runners being provided by the default
path /var/run/docker.sock instead of being accessible through TCP socket.

I set up my multi-arch Docker build with the following CI job:

docker-build:
  # Use the official docker image.
  image: docker:latest
  stage: release
  before_script:
    - echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER --password-stdin
  # Default branch leaves tag empty (= latest tag)
  # All other branches are tagged with the escaped branch name (commit ref slug)
  script:
    - |
      if [[ "$CI_COMMIT_BRANCH" == "$CI_DEFAULT_BRANCH" ]]; then
        tag=""
        echo "Running on default branch '$CI_DEFAULT_BRANCH': tag = 'latest'"
      else
        tag=":$CI_COMMIT_REF_SLUG"
        echo "Running on branch '$CI_COMMIT_BRANCH': tag = $tag"
      fi
    - docker buildx create --use
    - >
        docker buildx build --push 
        --platform linux/arm/v7,linux/arm64/v8,linux/amd64 
        --tag ${CI_REGISTRY_IMAGE}${tag}

Introducing Goctuator: a Go-Based Actuator Module

mikeyGlitz — Wed, 23 Nov 2022 03:55:58 +0000

As a developer, I'm constantly deploying and updating production-ready web servers. Some of the most important features that I look for in a web server are the metrics and the server's health. Having developed a number of Java web servers, I've come to appreciate the Spring Boot Actuator extensions for Spring Web applications.

In the context of a Spring application, an actuator is a server extension which will provide information about the server. These metrics could be items like the health and stability of connections that the application depends on (i.e. the database), or system resource utilization metrics. When I deploy applications onto a deployment environment such as a Kubernetes cluster or into the Cloud (EKS, ECS, Fargate), actuators are consumed by the cloud services to monitor the service health.

I wasn't able to find much to go on as far as modules to implement this feature in Golang. I decided to roll my own.

What is Goctuator?

Goctuator is short for Go-Actuator. This module provides spring-boot like actuator functionality to Go-based web servers. Goctuator provides the following endpoints

/env - Displays the current runtime environment variables
/info - Used to convey information about the package being published such as author, version number, go-lang runtime that the project was built with, CPU architecture that the application was built for, Git revision hash, and the repo URL
/health - This endpoint displays the overall system health as well as the health of any dependencies (i.e. database connection status)
/metrics - This endpoint displays system resource utilization
/treaddump - This endpoint displays a snapshot of the current thread and leverages pprof.lookup to gather information about the goroutines in use

Installation

Installing Goctuator is fairly simple. Use go get to retrieve the module and add it to your project

go get gitlab.com/mikeyGlitz/gohealth

After the package is installed you could add it to your project using a HTTP handler

HTTP package

handler := actuator.GetHandler(&actuator.Config{})
http.Handle("/actuator", handler)

Gin handler

handler := actuator.GetHandler(&actuator.Config{})
r := gin.Default()
r.Get("/actuator", func(ctx *gin.Context) {
    handle(ctx.Writer, ctx.Request)
}

Customization

The Goctuator module may be customized in one of two ways:

You can disable an endpoint
You can add custom health checkers
You can tailor the /info endpoint

Customizing available endpoints

The actuator.Configuration structure supports a slice Endpoints which is used to configure which endpoints are enabled as actuators:

config := &actuator.Configuration {
  Endpoints: []actuator.Endpoint {
    actuator.ENV,
    actuator.INFO,
  }
}

With this configuration, only the /info and /env endpoints would be enabled

Customizing Health Checkers

The health checkers may also be customized. By default the module provides endpoints which simulate a generic ping response as well as an endpoint which provides disk usage statistics for the application. These health checkers are located in health.PingChecker and health.DiskStatusChecker.

You may also write your own Health Checker by implementing the HealthChecker interface:

type ComponentDetails struct {
  Status HealthStatus `json:status` // Can either be UP or DOWN
  // Details - a struct which can be converted into a JSON
  // object which will provide additional details about the
  // service which health is being checked
  Details interface{} `json:details`
}

type HealthCheckResult struct {
  ComponentDetails
  Service string
}

type HealthChecker interface {
  CheckHealth() HealthCheckResult
}

When the /health endpoint is requested, the response will resemble the following JSON object:

{
  "status": "UP",
  "components": {
    "diskspace": {
      "status": "UP",
      "details": {
        "all": "",
        "used": "",
        "free": "",
        "available": ""
      }
    },
    "ping": {
      "status": "UP"
    }
  }
}

Customizing information returned by `/info`

The information returned by the /info endpoint is set at compile-time using ldflags

go build -ldflags="-X gitlab.com/mikeyGlitz/gohealth/pkg/info.AppName=${appName} <other flags>"

The /info endpoint utilizes the following variables:

Variable	Description
AppName	The name of your application
AppDescription	A brief description of what your application does
AppVersion	An application version i.e. v1.0.0
Git Variables
CommitID	The SHA1 of the commit
CommitTime	A timestamp of when the commit took place
BuildTime	A timestamp of when the application build took place
RepositoryUrl	A URL of where the Git repository is located
Branch	The branch name the build was executed on
Application Environment
OS	Operating system the application runs - darwin, linux
RuntimeVersion	The version of go the application was built for
Arch	The CPU architecture the application was built for - arm64

For more information about Goctuator and how to use it, please consult the Readme.

Proactive Kubernetes Monitoring with Alerting

mikeyGlitz — Wed, 15 Jun 2022 01:11:47 +0000

Last year I had written about the importance of service monitoring and observability.
While service monitoring is important, service monitoring requires active involvement from application maintainers. When supporting applications and services which service real users, it's important to become aware of problems when they first happen.
The longer your service is down, experiencing errors or performance issues, or otherwise buggy, the more negative an experience you provide for your users.
Alerting provides application developers a passive mechanism for becoming aware of problems involving their applications.
Instead of being informed by users that a problem is occurring in an application, developers receive automated alarms which inform them a problem is occurring.

Kubernetes Alerting

Alerting isn't natively provided in Kubernetes. Fortunately there are a few open source applications which can satisfy the alerting needs. As mentioned in my previous post, Linkerd provides many monitoring tools out-of-the box, but some customizations need to be made in order to provide alerting.

Prometheus is used to provide metrics monitoring (Memory usage, CPU usage, Network usage, etc.)
Alertmanager is used to create and manage alarms using Prometheus metrics
Grafana A visualization application which displays graphs and charts based on data received from metrics. Grafana also provides alerting.
Prometheus Operator a project which expresses Prometheus services, rules, and alarms as Kubernetes Custom Resource Definitions (CRDs). Prometheus Operator is a Kubernetes-idiomatic way of declaring Prometheus services.

Alerting Channels

Before going through the process of setting up alarms, it is important do decide how you want to be notified.
For this tutorial, I'll be setting up email alerts.
Simple Mail Transfer Protocol (SMTP) is the service which allows applications, in this case our monitoring services, to send emails to the people supporting the applications. By default, most Internet Service Providers (ISPs), Email Providers, and Cloud Providers block the port that SMTP sends emails on.

Setting up a SMTP Relay

A SMTP Relay is a service which can be used as a proxy for SMTP traffic when you want to send emails external to your Local Area Network (LAN). I've researched a few SMTP relays and ultimately decided on Dynu.
Dynu won me over through providing easy-to-use documentation on how to set up their SMTP relay service. Also at $9/year, the service is not unaffordable.

After opening an account with Dynu, my next step was to deploy the SMTP service onto Kubernetes.
Since I've started working with Kubernetes, I've discovered the convenience of working with Helm charts. ArtifuctHub is now my go-to for finding quick and easy helm charts to install onto Kubernetes.
I decided to deploy SMTP using the bokysan postfix chart.
I started out with these initial values.yml for the helm chart:

config:
  general:
    TZ: America/New_York
    LOG_FORMAT: json
    RELAYHOST: "{{ relay_host }}"
    ALLOWED_SENDER_DOMAINS: "cluster.local {{ domain_name }}"
secret:
  RELAYHOST_USERNAME: "{{ relay_username }}"
  RELAYHOST_PASSWORD: "{{ relay_password }}"

I installed the helm release using Ansible, but you can install with the following helm commands:

helm repo add docker-postfix https://bokysan.github.io/docker-postfix/
helm install --values values.yml --namespace=mail --create-namespace sender docker-postfix/mail

Forking the Postfix Base Image

Prometheus Alertmanager is built on Golang. A limitation of the SMTP integration in Alertmanager is that TLS is required for remote connections.
I attempted to use the certs.create value from the helm chart values, but the resulting certificate was not accepted by Alertmanager.
As a result, I had to create a new image from bokysan's Postfix image.

The new image detects for the following files to be mounted in /mnt/certs:

tls.crt
tls.key
ca.crt

If the files are present, they are copied into /etc/ssl and the postfix configuration at /etc/postfix/main.cf is updated to enable the SSL settings.

This set up is useful if using cert-manager to set up certificates using Kubernetes Secrets.

⚠️ When using the Postfix image, please note that the signing algorithm for your certificates needs to be RSA

I'm still able to use Bokysan's Helm chart with the following values:

image:
  repository: mikeyglitz/postfix
  tag: latest
extraVolumes:
  - name: tls-cert
    secret:
      secretName: mail-tls
extraVolumeMounts:
  - name: tls-cert
    mountPath: /mnt/certs
    readOnly: true
config:
  general:
    TZ: America/New_York
    LOG_FORMAT: json
    RELAYHOST: "{{ relay_host }}"
    ALLOWED_SENDER_DOMAINS: "cluster.local {{ domain_name }}"
secret:
  RELAYHOST_USERNAME: "{{ relay_username }}"
  RELAYHOST_PASSWORD: "{{ relay_password }}"

Installing Prometheus Operator

Prometheus Operator is a suite of applications leveraging the Operator Pattern for managing Prometheus applications in a way that is idiomatic for Kubernetes.
Configurations, Prometheus instances, and Alertmanager instances are managed using Custom Resource Definitions.

Before we can create the Prometheus Stack, we need to create a namespace where the prometheus resources will live

apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
  annotations:
    linkerd.io/inject: enabled

Installation of the Prometheus Operator is handled using the Prometheus Operator Helm Chart.
The Prometheus Operator Helm Chart installs the entire Prometheus Stack:

Prometheus
Alertmanager
Grafana
Prometheus-Operator
Prometheus NodeExporter

The values.yml for the chart is quite extensive and will take up a lot of space in the subsequent sections. I've broken down the values that go into the helm chart so that they can be better comprehended.
With Linkerd, additional customizations need to be made to ensure that the Linkerd dashboards work by installing the correct Prometheus rules. This can be accomplished with the following values.yaml

Prometheus Operator Values

Since I'm using Linkerd as my service mesh, and have created the monitoring namespace where I will be installing the Helm Chart using the linkerd.io/inject annotation, I will need to set the values for the Prometheus Operator so that the webhook is skipped from linkerd injection.

# Configure Prometheus -- we need to skip Linkerd injection or
# the operator will not install
prometheusOperator:
  admissionWebhooks:
    patch:
      podAnnotations:
        linkerd.io/inject: disabled
    certManager:
      enabled: "true"
      issuerRef:
        name: monitoring-issuer
        kind: Issuer

This configuration also utilizes cert-manager to inject certificates into the Prometheus Webhook. The certificates generated by the monitoring-issuer would be signed by our Root CA.

Prometheus Values

The default prometheus configuration must be modified to include custom scrapers so that Prometheus can export metrics to Linkerd. Without this configuration, the Linkerd Dashboard will not function properly.

prometheus:
  prometheusSpec:
    evaluationInterval: 10s
    scrapeInterval: 10s
    scrapeTimeout: 10s
    resources:
      requests:
        memory: 4Gi
    additionalScrapeConfigs:
    - job_name: 'prometheus'
      static_configs:
      - targets: ['localhost:9090']

    - job_name: 'grafana'
      kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: ['monitoring']
      relabel_configs:
      - source_labels:
        - __meta_kubernetes_pod_container_name
        action: keep
        regex: ^grafana$
    #  Required for: https://grafana.com/grafana/dashboards/315
    - job_name: 'kubernetes-nodes-cadvisor'
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
      metric_relabel_configs:
      - source_labels: [__name__]
        regex: '(container|machine)_(cpu|memory|network|fs)_(.+)'
        action: keep
      - source_labels: [__name__]
        regex: 'container_memory_failures_total' # unneeded large metric
        action: drop

    - job_name: 'linkerd-controller'
      kubernetes_sd_configs:
      - role: pod
        namespaces:
          names:
          - 'linkerd'
          - 'monitoring'
      relabel_configs:
      - source_labels:
        - __meta_kubernetes_pod_container_port_name
        action: keep
        regex: admin-http
      - source_labels: [__meta_kubernetes_pod_container_name]
        action: replace
        target_label: component

    - job_name: 'linkerd-service-mirror'
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels:
        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
        - __meta_kubernetes_pod_container_port_name
        action: keep
        regex: linkerd-service-mirror;admin-http$
      - source_labels: [__meta_kubernetes_pod_container_name]
        action: replace
        target_label: component

    - job_name: 'linkerd-proxy'
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels:
        - __meta_kubernetes_pod_container_name
        - __meta_kubernetes_pod_container_port_name
        - __meta_kubernetes_pod_label_linkerd_io_control_plane_ns
        action: keep
        regex: ^linkerd-proxy;linkerd-admin;linkerd$
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      # special case k8s' "job" label, to not interfere with prometheus' "job"
      # label
      # __meta_kubernetes_pod_label_linkerd_io_proxy_job=foo =>
      # k8s_job=foo
      - source_labels: [__meta_kubernetes_pod_label_linkerd_io_proxy_job]
        action: replace
        target_label: k8s_job
      # drop __meta_kubernetes_pod_label_linkerd_io_proxy_job
      - action: labeldrop
        regex: __meta_kubernetes_pod_label_linkerd_io_proxy_job
      # __meta_kubernetes_pod_label_linkerd_io_proxy_deployment=foo =>
      # deployment=foo
      - action: labelmap
        regex: __meta_kubernetes_pod_label_linkerd_io_proxy_(.+)
      # drop all labels that we just made copies of in the previous labelmap
      - action: labeldrop
        regex: __meta_kubernetes_pod_label_linkerd_io_proxy_(.+)
      # __meta_kubernetes_pod_label_linkerd_io_foo=bar =>
      # foo=bar
      - action: labelmap
        regex: __meta_kubernetes_pod_label_linkerd_io_(.+)
      # Copy all pod labels to tmp labels
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
        replacement: __tmp_pod_label_$1
      # Take `linkerd_io_` prefixed labels and copy them without the prefix
      - action: labelmap
        regex: __tmp_pod_label_linkerd_io_(.+)
        replacement:  __tmp_pod_label_$1
      # Drop the `linkerd_io_` originals
      - action: labeldrop
        regex: __tmp_pod_label_linkerd_io_(.+)
      # Copy tmp labels into real labels
      - action: labelmap
        regex: __tmp_pod_label_(.+)

Grafana Options

ℹ️ At the time of writing, using an external instance of Grafana
is only supported by an upcoming release of Linkerd 2.12.
Linkerd 2.12 can only be found on the edge branch, not the stable.
Subsequent sections will cover how to adjust the Linkerd install
to support an external Grafana instance in more detail.

Grafana needs to be updated to pre-install the Linkerd dashboard.
As per the Monitoring and Observability post, Loki also needs to be added as a data source for Grafana.

To support Grafana's Alerting Functionality Alertmanager can be added as a data source so that Alertmanager rules will appear in grafana.

ℹ️ At the time of writing, Alertmanager is only supported in Grafana plugins alpha.
Grafana plugins alpha can only be added in the grafana configuration via the plugins.enable_alpha option in grafana.ini.

# Grafana options -- pre-install Linkerd dashboards
# and configure datasources
grafana:
  grafana.ini:
    server:
      root_url: '%(protocol)s://%(domain)s:/grafana/'
    auth:
      disable_login_form: false
    auth.anonymous:
      enabled: true
      org_role: Editor
    auth.basic:
      enabled: true
    analytics:
      check_for_updates: false
    panels:
      disable_sanitize_html: true
    log:
      mode: console
    log.console:
      format: text
      level: info
    plugins:
      enable_alpha: true
    smtp:
      enabled: true
      from_address: grafana@<domain>
      host: sender-mail.mail-sender:587
      skip_verify: true
  dashboardProviders:
    dashboardproviders.yaml:
      apiVersion: 1
      providers:
      - name: 'default'
        orgId: 1
        folder: ''
        type: file
        disableDeletion: false
        editable: true
        options:
          path: /var/lib/grafana/dashboards/default

  dashboards:
    default:
      # Logging dashboard - https://grafana.com/grafana/dashboards/7752
      logging:
        gnetId: 7752
        revision: 5
        datasource: prometheus
      # all these charts are hosted at https://grafana.com/grafana/dashboards/{id}
      top-line:
        gnetId: 15474
        revision: 3
        datasource: prometheus
      health:
        gnetId: 15486
        revision: 2
        datasource: prometheus
      kubernetes:
        gnetId: 15479
        revision: 2
        datasource: prometheus
      namespace:
        gnetId: 15478
        revision: 2
        datasource: prometheus
      deployment:
        gnetId: 15475
        revision: 5
        datasource: prometheus
      pod:
        gnetId: 15477
        revision: 2
        datasource: prometheus
      service:
        gnetId: 15480
        revision: 2
        datasource: prometheus
      route:
        gnetId: 15481
        revision: 2
        datasource: prometheus
      authority:
        gnetId: 15482
        revision: 2
        datasource: prometheus
      cronjob:
        gnetId: 15483
        revision: 2
        datasource: prometheus
      job:
        gnetId: 15487
        revision: 2
        datasource: prometheus
      daemonset:
        gnetId: 15484
        revision: 2
        datasource: prometheus
      replicaset:
        gnetId: 15491
        revision: 2
        datasource: prometheus
      statefulset:
        gnetId: 15493
        revision: 2
        datasource: prometheus
      replicationcontroller:
        gnetId: 15492
        revision: 2
        datasource: prometheus
      prometheus:
        gnetId: 15489
        revision: 2
        datasource: prometheus
      prometheus-benchmark:
        gnetId: 15490
        revision: 2
        datasource: prometheus
      multicluster:
        gnetId: 15488
        revision: 2
        datasource: prometheus
  additionalDataSources:
  - name: alertmanager
    type: alertmanager
    url: http://metrics-kube-prometheus-st-alertmanager:9093
    access: proxy
    orgId: 1
    jsonData:
      implementation: prometheus
  - name: loki   
    type: loki
    access: proxy
    default: false
    editable: true
    url: http://loki:3100
    maximumLines: "300"
    orgId: 1
    jsonData:
      manageAlerts: true
      alertmanagerUid: alertmanager

Configuring Alertmanager

The final piece of the puzzle is configuring Alertmanager.
Alertmanager manages Prometheus alerts and is responsible for forwarding messages to the various alert receivers. Majority of the values below were extracted from the default configuration that the Alertmanager helm chart creates.

The notable differences are the root route. By default the root route doesn't allow a matcher. If we attached a receiver to the root route, we will be constantly notified. The constant notifications would trigger spam blockers to block our Alertmanager emails.

Under the root route, we create a nested routes field. The routes field is set up so that based on the matchers criteria we will receive an alert via email. In this case our alert will trigger whenever there is an alert where its severity is critical.

alertmanager:
  config:
    global:
      resolve_timeout: 5m
    route:
      group_by: ['job']
      group_wait: 15s
      group_interval: 5m
      repeat_interval: 12h
      receiver: 'null'
      routes:
        - group_by: ['alertname']
          group_wait: 15s
          group_interval: 10s
          repeat_interval: 12h
          matchers:
            - severity="critical"
          receiver: email
    receivers:
      - name: 'null'
      - name: email
        email_configs:
        - to: <recipient@mail.com>
          from: alertmanager@haus.net
          smarthost: sender-mail.mail-sender:587 
          require_tls: true
          tls_config:
            insecure_skip_verify: true
    templates:
      - '/etc/alertmanager/config/*.tmpl'
  alertmanagerSpec:
    externalUrl: https://monitoring.haus.net/alarms
    logFormat: json
    alertmanagerConfigNamespaceSelector:
      matchLabels:
        alertmanagerconfig: enabled
    alertmanagerConfigSelector:
      matchLabels:
        role: alertmanager-config

Installing the chart

The chart may be installed with the following command:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install --values values.yml --namespace monitoring metrics prometheus-community/prometheus-stack

Configuring Linkerd

In my previous post, I had leveraged kustomize to install Grafana with an additional data source.
This time around I'll be installing the linkerd-viz helm chart

First I begin by installing the edge repo. The current stable branch of Linkerd does not support bringing your own Grafana instance to linkerd

source: https://linkerd.io/2.11/tasks/grafana/
These notes apply only to recent Linkerd Edge releases and the upcoming Linkerd 2.12 stable release, which have stripped off the embedded Grafana instance, recommending users to install it separately as explained below.

helm repo add linkerd-edge https://helm.linkerd.io/edge

Based on the values.yml from the helm chart, I can set the update the grafana settings with the grafanaUrl parameter. I also set up Jaeger so that I can visualize application tracing.

values.yml

jaegerUrl: jaeger.linkerd-jaeger:16686
prometheusUrl: http://metrics-kube-prometheus-st-prometheus.monitoring:9090
grafanaUrl: metrics-grafana.monitoring:80
# Since we're bringing our own Prometheus and Grafana instances,
# we have to disable the embedded Prometheus and Grafana instances
prometheus:
  enabled: false
grafana:
  enabled: false

Install the linkerd-viz helm chart with the following command:

helm install --values values.yml --namespace linkerd-viz --create-namespace linkerd-viz linkerd-edge/linkerd-viz

Once the install completes, you will be able to access Grafana through Linkerd.

Alerts and Monitoring with Logging Operator

I use the Banzaicloud Logging Operator to provide idiomatic configurable logging from applications in my Kubernetes cluster to Grafana Loki. Logging Operator can provide Service Monitors and alarms for Logging instances with the following configuration:

apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  name: my-logger
  namespace: my-namespace
spec:
  fluentd:
    metrics:
      serviceMonitor: true
      prometheusRules: true
  fluentbit:
    metrics:
      serviceMonitor: true
      prometheusRules: true

The serviceMonitor tag sets up a Prometheus Service monitor.
The prometheusRules set up the Prometheus rules for alerting on certain thresholds.

The default alerting rules trigger alerts when:

Prometheus cannot access the Fluentd node

Fluentd buffers are quickly filling up

Traffic to Fluentd is increasing at a high rate

The number of Fluent Bit or Fluentd errors or retries is high

Fluentd buffers are over 90% full

Next Steps

Once everything has been set up, we should have a foundation for receiving alarms from our Kubernetes cluster.
From this point, we could enhance the alarms by creating additional rules, such as FluentD rules with the Logging Operator.
Perhaps you would like additional Alertmanager configurations to send alarms to messaging integrations such as Slack.

In my next post, I'll demonstrate how to set up an Alertmanager configuration to work with Discord/Guilded WebHooks.

References

Developing a NextJS app on OpenFaaS

mikeyGlitz — Wed, 10 Nov 2021 19:26:03 +0000

Prerequisites

The instructions covered in this article require the following tools to be installed before proceeding.

faas-cli - https://github.com/openfaas/faas-cli
Terraform - https://terraform.io
kubectl - https://kubernetes.io/docs/tasks/tools/
Docker - https://www.docker.com/get-started
Helm - https://helm.sh

Personally, I prefer to leverage VS Code Remote Containers to create portable development environments. Below you'll find the devcontainer.json and the Dockerfile to put inside your project's .devcontainer folder.

{
"name": "<appname>",
    "build": {
        "dockerfile": "Dockerfile",
        // Update 'VARIANT' to pick an Alpine version: 3.11, 3.12, 3.13, 3.14
        "args": {
            "VARIANT": "3.14",
            "DOCKER_GID": "1001",
            "NODE_VERSION": "14"
        }
    },

    // Set *default* container specific settings.json values on container create. 
    "settings": {},
    "mounts": [
        "source=/var/run/docker.sock,target=/var/run/docker.sock,type=bind",
        "source=${env:HOME}${env:USERPROFILE}/.kube,target=/home/vscode/.kube,type=bind"
    ],

    // Add the IDs of extensions you want installed when the container is created.
    // Note that some extensions may not work in Alpine Linux. See https://aka.ms/vscode-remote/linux.
    "extensions": [
        "ms-kubernetes-tools.vscode-kubernetes-tools",
        "ms-azuretools.vscode-docker"
    ],

    // Use 'forwardPorts' to make a list of ports inside the container available locally.
    // "forwardPorts": [],

    // Use 'postCreateCommand' to run commands after the container is created.
    // "postCreateCommand": "uname -a",

    // Replace when using a ptrace-based debugger like C++, Go, and Rust
    // "runArgs": [ "--init", "--cap-add=SYS_PTRACE", "--security-opt", "seccomp=unconfined" ],
    "runArgs": ["--init"],

    // Comment out connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.
    "remoteUser": "vscode"
}

Creating the OpenFaaS Deployment

The first step in deploying an application to OpenFaas is to deploy the OpenFaaS platform to Kubernetes. I use Helm and Terraform to create the OpenFaaS deployment.
OpenFaaS provides a helm chart

provider "kubernetes" {
  config_context = "docker-desktop"
  config_path = "~/.kube/config"
}
provider "helm" {
  kubernetes {
    config_context = "docker-desktop"
    config_path = "~/.kube/config"
  }
}

variable "openfaas_password" {
  type = string
  description = "OpenFaaS admin password"
}

resource "kubernetes_namespace" "ns_openfaas_fn" {
  metadata {
    name = "openfaas-fn"
  }
}

resource "kubernetes_namespace" "ns_openfaas" {
  metadata {
    name = "openfaas"
  }
}

resource "kubernetes_secret" "sec_openfaas_creds" {
  metadata {
    name = "basic-auth"
    namespace = "openfaas"
  }
  data = {
    "basic-auth-user: "admin",
    "basic-auth-password": var.openfaas_password
  }
}

resource "helm_release" "rel_openfaas" {
  name = "openfaas"
  namespace = "openfaas"
  chart = "openfaas"
  repository = "https://openfaas.github.io/faas-netes/"

  set {
    name = "functionNamespace"
    value = "openfaas-fn"
  }
  set {
    name = "generateBasicAuth"
    value = "false"
  }
  set {
    name = "basic_auth"
    value = "true"
  }
  set {
    name = "serviceType"
    value = "ClusterIP"
  }
  set {
    name = "ingressOperator.create"
    value = "true"
  }
}

The terraform script can be deployed with the following commands:

terraform init
terraform plan -var openfaas_password='<openfaas_password>' --out out.plan
terraform apply out.plan

The terraform script performs the following operations:

Creates the openfaas namespace
Creates the openfaas-fn namespace
Creates a Kubernetes secret with the basic-auth credentials
Deploys the OpenFaaS helm template
- Creates the OpenFaaS stack
- Disables the generation of a randomized admin password -- instead preferring the basic-auth secret we created earlier
- Deploys the OpenFaaS ingress operator which enables us to ingress our functions using a Custom Resource Definition (CRD)

Initializing the NextJS function

To create the function which will serve NextJS once we deploy it to OpenFaaS, the Docker template will need to be created.

faas-cli template store pull dockerfile
faas-cli new <appname> --lang dockerfile

The dockerfile template is created in a new folder which will be named the same value that was used for <appname> in the snippet above.

Next, the NextJS app will be initialized

npx create-next-app tmp-<appname> --ts # ts is optional. I like Typescript
mv tmp-<appname>/* <appname>/* # Relocate all files into the openfaas function folder
rm -rf tmp-<appname> # temporary folder is no longer needed

We have the basis for our NextJS OpenFaas function. The container template files need to be tweaked to work properly.

Update .dockerignore to exclude all unnecessary files from the Docker build

node_modules
.next
__tests__
coverage
docs

Update Dockerfile to properly build the NextJS application into an OpenFaaS function

# This template was adapted from the original node-express template
# https://github.com/openfaas-incubator/node10-express-template
FROM openfaas/of-watchdog:0.8.2 as watchdog

FROM node:14-alpine as ship

COPY --from=watchdog /fwatchdog /usr/bin/fwatchdog
RUN chmod +x /usr/bin/fwatchdog

RUN addgroup -S app && adduser -S -g app app

ENV NPM_CONFIG_LOGLEVEL warn

RUN mkdir -p /home/app

WORKDIR /home/app

RUN yarn

COPY . /home/app/

# Build the server
# remove the dev dependencies
RUN yarn && yarn build \
    && npm prune --production
RUN chown -R app:app /home/app && chmod 777 /tmp

USER app

ENV cgi_headers="true"
ENV fprocess="yarn start"
ENV mode="http"
ENV upstream_url="http://127.0.0.1:3000"

ENV exec_timeout="10s"
ENV write_timeout="15s"
ENV read_timeout="15s"

EXPOSE 8080

HEALTHCHECK --interval=3s CMD [ -e /tmp/.lock ] || exit 1

CMD ["fwatchdog"]

With all the configuration completed, you should be able to deploy the function to OpenFaaS

faas-cli login # Prompt for username and password
faas-cli up -f <appname>.yml # Deploy he function

References

OpenFaaS Helm Chart - https://github.com/openfaas/faas-netes/tree/master/chart/openfaas
OpenFaaS Ingress - https://github.com/openfaas/ingress-operator

Deploying a React app using Min.io

mikeyGlitz — Sun, 11 Jul 2021 15:55:23 +0000

In previous posts I had written about how to set up a Kubernetes cluster on self-hosted hardware with the purpose of hosting applications on a Local Area Network (LAN) to use as an intranet of sorts. Today I'll cover how to deploy client-side applications onto Kubernetes cluster which was provisioned.

Servers to Cloud Platforms

First, let's talk about how the deployments would be handled on a cloud platform. During the .com days, a server would have to be provisioned to host web applications (i.e. Apache/Nginx). The static website would then have to be copied into a static HTML folder (i.e. /var/www/html) in order to be served to clients via the web server.
The process of web hosting was improved with the rise of containerization. Instead of having to set up and manage a web server directly on your server hardware, now you can pull a pre-configured image and mount your static web content onto a container which would drastically improve website deployment times.
The rise of cloud platforms furthers the improvement by abstracting away the deployment environment from the developer entirely so that more focus can be placed on the website assets themselves instead of provisioning and configuring servers.
Cloud platforms implement hosted storage using a standard called Object-Based-Storage. Object-based-storage utilizes web endpoints to control and manage assets uploaded to a server. Site content can be managed and served directly using the HTTP protocol. AWS S3 is a perfect example of how object-based-storage works.

Minio

Minio is a self-hosted service which provides object-based-storage using the AWS S3 protocol; meaning that the endpoints which Minio provides are interchangeable with S3. Minio can be used as a gateway to hosted object-based-storage services which reside on multiple cloud platforms (i.e. Google Cloud Platform (GCP), Amazon Web Services (AWS), Microsoft Azure), but for the sake of this deployment, Minio will be used as a frontend for a mounted volume on our Kubernetes cluster.

Deploying the Services

Before we can deploy a static website to our Kubernetes cluster, we will first have to provision a Minio server.
The Minio documentation utilizes a Minio Operator and a kubectl krew plugin to provision servers. Utilizing these tools will be covered in a later production release document. For the sake of this tutorial, deploying Minio will be handled with the Minio helm chart

Utilizing Terraform, the Minio server can be deployed to Helm with the following snippet:

provider "helm" {}

resource "helm_release" "rel_minio" {
  name       = "files"
  chart      = "minio"
  repository = "https://charts.bitnami.com/bitnami"
}

The helm chart can be deployed with the following commands:

terraform plan
terraform apply

Once the helm deployment has been completed, the service will be available from the Kubernetes cluster. In order to interact with the services, the credentials will have to be read in from the Kubernetes secrets which are generated by the helm chart. The following commands retrieve the secrets from Kubernetes and store them in environment variables:

export minio_access_key=$(kubectl get secret files-minio --namespace econovizer -o=jsonpath='{.data.access-key}' | base64 --decode)
export minio_secret_key=$(kubectl get secret files-minio --namespace econovizer -o=jsonpath='{.data.secret-key}' | base64 --decode)

We then have to port-forward the Kubernetes service in order to access it.

kubectl port-forward services/files-minio 9000:9000

Configuring the Minio Client

Minio provides a CLI called mc which can be utilized to interact with the Minio server. We have to call the mc tool with the $minio_secret_key and the $minio_access_key environment variables we created earlier.

mc alias set local http://127.0.0.1:9000 $minio_access_key $minio_secret_key

With the client configured we can now create a bucket for hosting our static site.

mc mb local/static

Before assets can be served from the bucket, the bucket needs to be configured for public asses.

mc policy set download local/static

Creating the React Application

With the hosting environment established, we can now create our static website. The easiest way to set up the static website is using Create React App.

npx create-react-app my-app

This command will create a React application with the name my-app in the current folder. We need to change into the my-app folder -- cd my-app. Build the project with the command npm run build. The build command creates a new folder build.

With the build folder created, we can deploy the build to our bucket with the mc command

mc cp -r build/* local/static/
mc ls local/static # list the files which were just uploaded to the local/static bucket

Ideally from this point, you would be able to access the static site from http://localhost:9000/static, however Minio has a limitation which prevents it from serving up files unless they were referenced directly.
http://localhost:9000/static will return an XML document containing a ListBucketResult instead of index.html. http://localhost:9000/static/index.html will return the desired web page. Since the URL would end with index.html, React would be looking for a web root and fail to load.

Fortunately, the issue could be fixed with a proxy application: s3www.

To simplify the deployment of s3www, I created a Terraform template which deploys the following resources to Kubernetes:

s3www pod deployment
s3www Kubernetes service
Ingress which proxies the s3www service

This file can be run with the following commands:

terraform init
terraform plan -var "acces_key=$minio_access_key" -var "secret_key=$minio_secret_key" -var 'namespace=my-namespace' -out deployment.plan
terraform apply deployment.plan

Once Terraform is complete, the React application will be available from your Kubernetes cluster via Ingress. (i.e. http://host.docker.internal/)

References

Homelab: Cluster Architecture

mikeyGlitz — Mon, 14 Jun 2021 18:19:41 +0000

For a lot of people, 2020 was a rough year would be an understatement. I was fortunate enough to come across a video and some mentorship which motivated me to try and come out of lockdown better than I was before.

I had decided that it was time to pick up a new skill. At the time there was a goal at work to transition our infrastructure from using AWS Elastic Container Service (ECS) to an internal managed Kubernetes-based platform. Having only dabbled in Kubernetes I wanted to know more. I set out to build a home-lab using Kubernetes (k3s). It took me about 6 months to reach a point where I was comfortable with what I had built.

Deciding What to Build

I started out the process by thinking about what I wanted to use the home lab for. I was originally inspired by a blog post I had read about setting up a raspberry pi home lab which set up DNS, OpenLDAP, and NextCloud. I knew similarly that I wanted to have a centralized system for identity management and a place to store files. As a developer who sometimes works on personal projects, I also wanted a local network playground for any personal services I build.

I spent about 2 months pouring over Helm charts and experimenting with Kubernetes for Docker Desktop using my Windows laptop to get as close to an ideal configuration as I could. I ran out of resources and eventually created a 3-node cluster using embedded devices.

I ultimately came up with the following design:

Architectural Explanation

The architecture utilizes K3S as a lightweight alternative to Kubernetes. Due to the decision to leverage low-cost embedded devices, there was an inherit limitation on resources which would be available. A kubernetes distribution which utilized resources efficient would be paramount. Other factors which lead into the selection of K3S is the distribution provided the following features out-of-the-box:

Proxying services externally with ServiceLB
A one-click installer using k3sup

File Storage

nfs-client-provisioner implements portable networks storage across the Kubernetes cluster using the Network File System (NFS) Protocol. NFS allows for folders to be mounted via TCP protocols as network file shares. nfs-client-provisioner takes things a step further by auto-provisioning NFS volumes whenever a Kubernetes Persistent Volume Claim (PVC) is declared. The PVC will be designated as an NFS client if the nfs storage class is used.
The idea here is that I wanted the Docker containers running in the Kubernetes Pods to write directly to a USB Drive which is mounted onto the Kubernetes master node. nfs-client-provisioner creates a sub-folder in the NFS root with the name of the PVC. Deleted PVCs are designated archived-<pvc-name>.

Networking

By default K3S utilizes traefik as an ingress controller for the Kubernetes cluster. Having used Traefik before, I found setting up a traffic intercept using a project like Oauth-proxy not that straightforward with Traefik. I chose Nginx due to its widespread support, ease of configuration through annotations, its ability to provide proxy for TCP and UDP services, and being able to set up an authentication proxy using Kubernetes annotations on ingress objects.

Oauth2-proxy is a project which provides an authentication and authorization intercept to services where you want to protect pages. Oauth2-proxy leverages the OAUTH2 protocol to delegate authentication. Oauth2-proxy also allows custom configuration for identity management services such as Keycloak.

Identity Management

Having experienced the issue of identity management in past projects (I've literally published 5 web apps which became some iteration of user-profile applications), I found Keycloak to be of particular use when it comes to managing users and federating accounts. Keycloak is an open-sourced enterprise service which manages identity, authentication, authorization, and account federation which is part of the JBoss project and backed by RedHat. Since I wanted to use the Keycloak helm chart the Keycloak service runs using a Postgres backend.

Protecting Application Secrets

Nobody likes a data breach. One of the most common ways sensitive application secrets can be exposed is through use of environment variables, or through data templates. Hashicorp produced an awesome service called Vault which allows you to managed and protect application secrets. Vault is easier to adapt to Kubernetes when using the Banzaicloud Vault Operator. Vault Operator enables the creation of a multi-tenant vault service as well as injection of secrets into Kubernetes Pods using annotations and a service web hook.

NextCloud and Keycloak use Vault Operator to inject a randomly generated database password so that each service could connect to its backend database. Oauth2-proxy containers utilize Vault Operator in order to inject Keycloak client IDs and client secrets into the Ouath2-proxy containers.

Certificates

Along similar lines to the application security discussed in the previous section, unsecure connections are another particular pitfall for sensitive data leak. In Kubernetes, certificates can be managed and generated dynamically using the certificate-manager service.

The home cluster utilizes a self-signed ECDSA certificate to serve as the root certificate. Certificate manager dynamically generates certificates from the root certificate when pods or ingresses request a certificate upon creation.

Monitoring and Observability

The first level of monitoring in the home cluster occurs inside of the service mesh. A service mesh is a reliable service which proxies traffic internally inside of the Kubernetes cluster to provide mTLS security, monitoring, and additional reliability. While Istio is the most popular Kubernetes service mesh, LinkerD was selected due to its ease of installation as well as its ease of use.

Hand-in-hand with the service mesh is the Logging Operator. Logging Operator provides real-time log streaming using FluentD and FluentBit to stream log events to a 3rd party service (Grafana Loki, ELK, etc.). FluentD/FluentBit is injected to a monitored service via side-car injection where the container logs are captured by the fluent agent and streamed to the configured destination service.

Application Hosting

Due to the resource constraints of running on embedded devices, special considerations had to be taken into account for hosting my personal projects. I wanted as little resource overhead as I could possibly require. In the cloud world the term serverless exists for which developers can be concerned about the software they developed, but don't have to consciously consider the infrastructure that it runs on. Services are composed of multiple single-purpose functions which execute and terminate on-demand.

A similar functionality exists for Kubernetes clusters. Originally I had investigated Kubeless as a platform for running serverless applications; however, I ultimately decided on OpenFaaS due to the availability and community support.

Additional Services

Though not in current use, I wanted to be able to index data in my personal projects as a method of allowing data to be searched. I've worked with ElasticSearch and it is my preferred index. Fortunately Elastic also provides Elastic Cloud for Kubernetes (ECK).

ECK enables the deployment of a multi-tenant elastic environment which can be fully expressed using Kubernetes Custom Resource Definitions (CRDs). ECK is fairly easy to set up and can pass all configuration options available to any of the Elastic services (Elasticsearch, Kibana, Beats) directly to the Kubernetes pods.

References

I published the cluster set-up into an Ansible playbook on GitHub

NFS client provisioner - https://artifacthub.io/packages/helm/rimusz/nfs-client-provisioner
Keycloak Helm Chart - https://github.com/codecentric/helm-charts/tree/master/charts/keycloak
NextCloud Helm Chart - https://nextcloud.github.io/helm/
Vault Operator Helm Chart - https://github.com/banzaicloud/bank-vaults
Logging Operator Helm Chart - https://github.com/banzaicloud/logging-operator
MariaDB Helm Chart - https://github.com/bitnami/charts/tree/master/bitnami/mariadb
Postgres Helm Chart - https://github.com/bitnami/charts/tree/master/bitnami/postgresql
ECK User Guide - https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-deploy-eck.html
OpenFaaS Helm Chart - https://github.com/openfaas/faas-netes
Oauth2-proxy Helm chart - https://artifacthub.io/packages/helm/oauth2-proxy/oauth2-proxy
Oauth2-proxy - https://oauth2-proxy.github.io/oauth2-proxy/
cert-manager - https://artifacthub.io/packages/helm/cert-manager/cert-manager
Kubeless - https://kubeless.io
OwnCloud Helm Chart - https://github.com/bitnami/charts/tree/master/bitnami/owncloud
k3sup documentation - https://github.com/alexellis/k3sup
ingress-nginx documentation - https://kubernetes.github.io/ingress-nginx/deploy/#using-helm

So there it is! I've finally got a reliable application host set up using clustered container orchestration. Let me know if there's any questions, or if there's a particular area where I could go further in detail.

DEV Community: mikeyGlitz

Setting up Parseable with Kubernetes and Docker Desktop

Setting up Parseable with Kubernetes and Docker Desktop

Prerequisites

Table of Contents

Setting up MinIO

Deploying MinIO

Deploying Parseable

Installing Parseable

Configuring Ingress

Creating Ingress Resource

Complete Ansible Playbook

Accessing Parseable

Next Steps

Troubleshooting

Conclusion

Bringing It All Together: Integrating GraphQL with Gin in Go

Implementing Authentication Middleware with Keycloak and Gocloak

Finalizing Server Setup: The Run Function

Finishing Touches: Revisiting GraphQL

Conclusion: Bringing it All Together

References

Building Robust Applications in Go: Integrating Envconfig, Gorm, and OpenSearch

Creating an Application Configuration with envconfig

Integrating a Data Persistence Layer with Gorm

Indexing Data with OpenSearch

Implementing the OpenSearch Client

Connecting to OpenSearch

Synching with the Database

Summary

References

OpenSearch Snapshots with Min.io

Installing Min.io

Installing OpenSearch

Setting up the Snapshot Repository

Improving the Deployment for Production Readiness

References

Connecting OpenSearch to Keycloak

Managing Certificates

Deploying Keycloak

Deploying OpenSearch

References

Go API Project Set-Up

Conventions

Project Structure

Enforcing Style

Mocks

Test Reporting

Setting up a GitLab Runner

Pipeline Runners

Getting Your Runner Token

Setting Ansible Variables

Securing Your Token

Deploying the Helm Release

Project Configuration

Introducing Goctuator: a Go-Based Actuator Module

What is Goctuator?

Installation

HTTP package

Gin handler

Customization

Customizing available endpoints

Customizing Health Checkers

Customizing information returned by /info

Proactive Kubernetes Monitoring with Alerting

Kubernetes Alerting

Alerting Channels

Setting up a SMTP Relay

Forking the Postfix Base Image

Installing Prometheus Operator

Prometheus Operator Values

Prometheus Values

Grafana Options

Configuring Alertmanager

Installing the chart

Configuring Linkerd

Alerts and Monitoring with Logging Operator

Next Steps

References

Developing a NextJS app on OpenFaaS

Customizing information returned by `/info`