Creating CDK Pipeline Constructs to Improve Developer Experience

Projects with an integrated CDK Pipeline can be cumbersome (at times) when you need to deploy into a developer-friendly/sandbox account WITHOUT the pipeline. In part 2 of my CDK Pipeline series we're going to turn part 1's code into CDK Constructs that will allow you to easily deploy code with or without a pipeline.

If you missed Part 1 you can find it here: CDK Pipelines Crash Course

This is an intermediate level post.

If you're new to CDK... you might be getting a bit ahead of yourself. Maybe start with my freeCodeCamp CDK Crash Course instead? 😉

The code for this is located at https://github.com/martzcodes/blog-pipeline

Make Constructs (not war)

What we have so far is about 100 lines of code to create a Pipeline and CodeBuild project, but none of it is reusable. Let's fix that. Our goal is to create re-usable Constructs that could be installed from an open or inner-source library. One of the CDK Best Practices is to use properties to pass configuration in from the top-level of the app, so we're going to make sure we're doing that.

Pipeline Stack

The Pipeline Stack is already pretty well-defined as its own stack, but it has a bunch of things hard-coded in it. We'll replace all the const declarations with a property interface and move it to lib/central-pipeline.ts

export interface CentralPipelineProps extends StackProps {
  owner: string;
  repo: string;
  branch: string;
  secretArn: string;
}

Pull Request CodeBuild Construct

From our Automated PR Checks section in Part 1 we can take the code and with very little modification. We're going to pull this out into lib/central-pr.ts.

To make this more re-usable we're going to define the Construct properties as:

export interface CentralPullRequestBuildProps {
  branch: string;
  buildCommands: string[];
  owner: string;
  projectName?: string;
  repo: string;
}

This enables us to spin up multiple CodeBuild Projects (multiple PR checks) AND run different tasks.

In the CentralPipeline we can now update the properties to include a pullRequestProjects property that can include these commands:

export interface CentralPipelineProps extends StackProps {
  // ...
  pullRequestProjects: string[][];
}

Then we can create as many CodeBuild Projects as needed using:

    props.pullRequestProjects.forEach((projectCommands) => {
      new CentralPullRequestBuild(this, "UnitTests", {
        owner,
        repo,
        branch,
        buildCommands: projectCommands,
      });
    });

☠️Note on creating multiple CodeBuild Projects. CodeBuild has something called Batches available in their build spec. I could not find a single example of these working anywhere on the internet and I myself spent ~10 hours trying to get them to work in various configurations and couldn't do it. If you get these working, please let me know either via a comment or on twitter @martzcodes 🙏

Setting the Stage

The Stage is the interesting problem. It needs a reference to the Stack that it's under and it needs project specific code under it. 🤔

You could abstract this out... OR you can create a generator function for the contents of the stage (and this will come in handy later)! I created a simple stage generator in lib/stage-generator.ts and it looks like this:

export const stageGenerator = (scope: Construct, id: string) => {
    new QueueStack(scope, id, {});
}

This enables us to define a centrally-provided stage like this lib/central-stage.ts:

export class CentralStage extends Stage {
  constructor(scope: Construct, id: string, props: CentralStageProps) {
    super(scope, id, props);

    props.stageGenerator(this, props.stackId, { env: props.env });
  }
}

...where we pass in the generator via properties and we can do that from the bin/app level!

This way nothing in these three lib/central-* files is project-specific and they could all be put in an npm library for use in multiple projects.

In lib/central-pipeline.ts we can add the stage and stageGenerator props:

export interface CentralPipelineProps extends StackProps {
  // ...
  stages: {
    name: string;
    env: Environment;
  }[];
  stageGenerator: (scope: Construct, id: string, props: StackProps) => void;
}

and loop over the stages to add them all to the pipeline:

export class CentralPipeline extends Stack {
  constructor(scope: Construct, id: string, props: CentralPipelineProps) {
    super(scope, id, props);
    // ...
    props.stages.forEach((deploymentStage) => {
      const stage = new CentralStage(this, deploymentStage.name, {
        stackId: "QueueStack",
        stageGenerator: props.stageGenerator,
        env: deploymentStage.env,
      });
      pipeline.addStage(stage);
    });
}

At this point we haven't changed anything in our stack... so there's nothing to re-deploy.

Improving Developer Experience (DevEx)

It's a reasonable expectation that devs might want to deploy code themselves WITHOUT a pipeline.

For example, at my work we tend to have ~4 AWS Accounts per team: Sandbox, Dev, QA, Prod. Devs have access to deploy to Sandbox and Dev, while only CI/CD can deploy to QA/Prod.

If I'm working on a project I don't want to deploy a CodePipeline just to get a project up and running in a sandbox account.

By tweaking these constructs a little we can use an environment variable to tell CDK whether or not to create the pipeline or just to create the contents of the stage generator.

I'll define two npm scripts in package.json:

• npm run deploy which runs npx cdk deploy '*' and deploys just the output of the stage generator
• npm run deploy:pipeline which runs USE_PIPELINE=true npx cdk deploy '*' and deploys the full pipeline

To accomplish this, I've modified bin/blog-pipeline.ts to check for the USE_PIPELINE environment variable:

const usePipeline = !!process.env.USE_PIPELINE;

new CentralPipeline(app, `My${usePipeline ? "Pipeline" : "Stack"}`, {
  usePipeline,
  stages: [
    {
      name: "Main",
      env,
    },
    // {
    //   name: "FakeProd",
    //   env: {
    //     account: '12345',
    //     region: 'us-east-1',
    //   },
    // },
  ],
  stageGenerator: stageGenerator,
  // ...
});

...and in the CentralPipeline Stack I use the usePipeline property to conditionally create the pipeline (or not!):

export class CentralPipeline extends Stack {
  constructor(scope: Construct, id: string, props: CentralPipelineProps) {
    super(scope, id, props);
    const { usePipeline = false, owner, repo, branch, secretArn } = props;
    if (!usePipeline) {
      // we use scope here to put the stage contents on the top level
      props.stageGenerator(scope, "QueueStack", {});

      // the return is important here because it prevents the pipeline code from synthesizing
      return;
    }
    // ... pipeline code below
  }
}

🚀 Defining it in the CentralPipeline means this is a feature for EVERY PROJECT that uses these constructs!

👀 One Note: You could deploy the target stage using wildcard'ed deploys similar to the documentation here. It would be similar to `npx cdk deploy cdk deploy "//"`... but it's less plug and play that way and is more sensitive to renaming.* 👀

Recap

In this part 2 of this series we took the code from part 1 and turned that into a set of Constructs that could be published as part of an npm library. A benefit of having the construct within an inner-sourced library is you can easily update multiple project's CI/CD pipelines simply by publishing an update.

Want to add accelerate metrics as part of all your deployments? Update the library to include a metric step, publish a new version and on their next commit with dependency updates the pipeline will self-mutate themselves to pull in the new feature!

To improve Developer Experience, we also added in a check to conditionally deploy the pipeline... so when devs just want to deploy their stack they can do it without commenting out a bunch of code or figuring out which wildcard flavor they need for their cdk deploy command.