Custom Resource Definitions (CRDs)

Custom Resource Definitions (CRDs): Extending Kubernetes Beyond its Limits

Introduction

Kubernetes, the leading container orchestration platform, offers a robust set of built-in resources like Pods, Services, Deployments, and Namespaces to manage containerized applications. While these resources cater to a wide range of common use cases, there are situations where you need to define and manage application-specific resources tailored to your unique requirements. This is where Custom Resource Definitions (CRDs) come into play, allowing you to extend Kubernetes' API with your own custom resources, effectively tailoring the platform to your specific needs.

CRDs provide a declarative way to define new kinds of Kubernetes resources, much like you define built-in resources. They are a powerful mechanism for building platform-as-a-service (PaaS) solutions on top of Kubernetes, managing complex applications with domain-specific abstractions, and integrating with external systems seamlessly. In essence, CRDs turn Kubernetes into a truly extensible and adaptable platform, making it suitable for an even wider range of workloads.

Prerequisites

Before diving into the details of CRDs, you should have a solid understanding of the following Kubernetes concepts:

• Kubernetes API: Familiarity with the Kubernetes API server and how it interacts with resources.
• kubectl: Proficiency in using kubectl to interact with your Kubernetes cluster.
• YAML: Understanding of YAML syntax, as CRDs are defined using YAML.
• Kubernetes Controllers (Optional but Recommended): While you can create CRDs without controllers, controllers provide the logic to manage the lifecycle of custom resources and react to changes. A strong understanding of Kubernetes controllers will enable you to create truly powerful and dynamic custom resources.
• Basic Programming Knowledge (If writing a Controller): If you are building a controller to manage your custom resources, you will need programming knowledge in a language like Go, Python, or Java.

Advantages of Using CRDs

• Extensibility: CRDs enable you to extend the Kubernetes API with custom resources that represent application-specific concepts, allowing you to model your application in a way that aligns with your domain.
• Declarative Management: You can manage custom resources declaratively, just like built-in Kubernetes resources, using YAML manifests and kubectl. This simplifies configuration management and promotes infrastructure-as-code principles.
• Integration with Kubernetes Tooling: Custom resources created with CRDs seamlessly integrate with the Kubernetes ecosystem. You can use kubectl to create, update, delete, and monitor your custom resources. You can also leverage Kubernetes features like RBAC (Role-Based Access Control) to control access to these resources.
• No Code Required (for basic usage): While a controller is usually necessary for more complex lifecycle management, you can define and create custom resources without writing any code, making them easy to introduce to an existing infrastructure.
• Simplified API Extension: Compared to extending the Kubernetes API using API aggregation, CRDs are simpler to implement and manage, requiring less in-depth knowledge of Kubernetes internals.
• Namespaced or Cluster-Scoped: CRDs can be configured to be either namespaced or cluster-scoped, allowing you to control the scope of your custom resources. Namespaced CRDs are available only within a specific namespace, while cluster-scoped CRDs are available across the entire cluster.

Disadvantages of Using CRDs

• Limited Validation: CRDs offer validation mechanisms through OpenAPI v3 schemas, but they might not be as comprehensive as those provided by built-in resources.
• No Built-in Controller Logic: CRDs only define the structure of the custom resource. You need to create a custom controller to manage the lifecycle of these resources and implement any desired business logic.
• Potential for Complexity: Designing and implementing a CRD and its associated controller can be complex, especially for sophisticated applications with complex lifecycle management requirements.
• Overhead of Controller Development: Developing a controller to manage your custom resources introduces the overhead of writing, testing, and maintaining the controller code.
• Limited Versioning Support: While CRDs support versioning of the resource schema, managing complex migrations between different versions can be challenging.

Features of CRDs

• Schema Validation: CRDs allow you to define an OpenAPI v3 schema to validate the structure and content of your custom resources. This schema can specify data types, required fields, and validation rules.
• Storage Versions: CRDs support storage versions, allowing you to evolve your resource schema without breaking compatibility with existing resources.
• Namespaces: CRDs can be scoped to a namespace, making them specific to a particular application or team, or can be cluster-scoped.
• Subresources: CRDs can define subresources like /status and /scale, which are used to expose specific aspects of the resource's state or behavior. The /status subresource is commonly used to report the current state of a custom resource, while the /scale subresource is used to manage the number of replicas for a resource.
• Custom Naming: You have full control over the name and group of your Custom Resource Definition. This allows you to define resources that align well with the semantics of your domain.
• Webhook Integration: CRDs support webhooks for advanced validation and mutation. You can define webhooks that are called when a custom resource is created, updated, or deleted, allowing you to enforce custom business logic and policies.

Example CRD: Database

Let's create a simple CRD for managing database instances. We'll call it Database and define some basic properties like name, size, and engine.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.example.com  # {plural}.{group}
spec:
  group: example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                name:
                  type: string
                size:
                  type: integer
                  minimum: 1
                engine:
                  type: string
                  enum: [ "postgres", "mysql" ]
  scope: Namespaced
  names:
    plural: databases
    singular: database
    kind: Database
    shortNames:
    - db

Explanation:

• apiVersion: apiextensions.k8s.io/v1: Specifies the API version for CRDs.
• kind: CustomResourceDefinition: Defines this as a CRD.
• metadata.name: databases.example.com: The fully qualified name of the CRD, composed of the plural form of the resource and the group.
• spec.group: example.com: The API group for this CRD.
• spec.versions: Defines the API versions supported by the CRD. storage: true indicates that this version is used for storing the resource in etcd.
• spec.versions[0].schema.openAPIV3Schema: Defines the schema for validating custom resources. This example enforces string types and valid enums, and sets a minimum on the size property.
• spec.scope: Namespaced: Specifies that this CRD is namespaced.
• spec.names: Defines the names used to refer to the custom resource. plural is the plural form, singular is the singular form, kind is the type name, and shortNames provides convenient aliases.

Applying the CRD:

kubectl apply -f database-crd.yaml

After applying the CRD, you can create instances of the Database resource like this:

apiVersion: example.com/v1
kind: Database
metadata:
  name: my-postgres-db
spec:
  name: my-postgres-instance
  size: 10
  engine: postgres

Applying a Custom Resource:

kubectl apply -f my-database.yaml

Managing Custom Resources:

You can now interact with your custom resource using kubectl:

kubectl get databases
kubectl describe database my-postgres-db
kubectl edit database my-postgres-db
kubectl delete database my-postgres-db

The Importance of a Controller

While defining a CRD allows you to create and store custom resources, a controller is crucial for actually managing them. The controller observes the state of the custom resources, compares it to the desired state defined in the resource's specification, and takes actions to reconcile the difference. In the Database example, a controller would be responsible for provisioning the database instance, monitoring its health, handling backups, and performing other management tasks.

Building a controller typically involves using the Kubernetes client libraries for your programming language of choice (e.g., client-go for Go, kubernetes for Python) to interact with the Kubernetes API and implement the reconciliation logic. Frameworks like Operator SDK and Kubebuilder can significantly simplify the development of Kubernetes controllers.

Conclusion

CRDs provide a powerful and flexible mechanism for extending Kubernetes beyond its built-in capabilities. They allow you to define custom resources that represent application-specific concepts, enabling you to manage complex applications with domain-specific abstractions. While CRDs require careful design and implementation, particularly when developing a corresponding controller, they offer a compelling way to tailor Kubernetes to your unique needs and build platform-as-a-service solutions. With the right understanding and tools, CRDs can significantly enhance your ability to manage and orchestrate complex applications on Kubernetes. They are a cornerstone of the operator pattern and a critical component in many modern cloud-native architectures.