Concourse Pipeline to Build a Docker Image Automatically on Git Commit

In this tutorial we will build a ci pipeline using concourse to build and push a image to dockerhub automatically, whenever a new git commit is made to the master branch.

Our Project Setup

Our Directory Tree:

$ find .
./Dockerfile
./ci
./ci/pipeline.yml
./README.md
./docker-tunnel

The project used in this example is not important, but you can check it out at https://github.com/ruanbekker/docker-remote-tunnel

Our Pipeline

A visual to see how the pipeline will look like in concourse:

Our pipeline definition will consist of 3 resources, github repo, dockerhub image and a slack resource to inform use whether a build has completed.

Then we are specifying that the job should be triggered on a git commit for the master branch, build and push to our dockerhub repo.

Our pipeline definition ci/pipeline.yml:

resources:
- name: git-repo
  type: git
  source:
    uri: [e-mail]:ruanbekker/docker-remote-tunnel.git
    branch: master
    private_key: ((github_private_key))

- name: docker-remote-tunnel-image
  type: docker-image
  source:
    repository: ruanbekker/docker-remote-tunnel
    tag: test
    username: ((dockerhub_user))
    password: ((dockerhub_password))

- name: slack-alert
  type: slack-notification
  source:
    url: ((slack_notification_url))

resource_types:
  - name: slack-notification
    type: docker-image
    source:
      repository: cfcommunity/slack-notification-resource
      tag: v1.3.0

jobs:
- name: build-cached-image
  plan:
  - get: git-repo
    trigger: true
  - task: build-cached-image-workspace
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          repository: rbekker87/build-tools

      outputs:
      - name: workspace
      inputs:
      - name: git-repo

      run:
        path: /bin/sh
        args:
        - -c
        - |
          output_dir=workspace

          cat << EOF > "${output_dir}/Dockerfile"
          FROM alpine

          ADD git-repo /tmp/git-repo
          RUN mv /tmp/git-repo/docker-tunnel /usr/bin/docker-tunnel
          RUN apk --no-cache add screen docker openssl openssh-client apache2-utils
          RUN /usr/bin/docker-tunnel -h
          RUN rm -rf /tmp/git-repo
          EOF

          cp -R ./git-repo "${output_dir}/git-repo"

  - put: docker-remote-tunnel-image
    params:
      build: workspace

    on_failure:
      put: slack-alert
      params:
        channel: '#system_events'
        username: 'concourse'
        icon_emoji: ':concourse:'
        silent: true
        text: |
            *$BUILD_PIPELINE_NAME/$BUILD_JOB_NAME* ($BUILD_NAME) FAILED to build image
            https://ci.domain.com/teams/$BUILD_TEAM_NAME/pipelines/$BUILD_PIPELINE_NAME/jobs/$BUILD_JOB_NAME/builds/$BUILD_NAME
    on_success:
      put: slack-alert
      params:
        channel: '#system_events'
        username: 'concourse'
        icon_emoji: ':concourse:'
        silent: true
        text: |
            *$BUILD_PIPELINE_NAME/$BUILD_JOB_NAME* ($BUILD_NAME) SUCCESS - Image has been published
            https://ci.domain.com/teams/$BUILD_TEAM_NAME/pipelines/$BUILD_PIPELINE_NAME/jobs/$BUILD_JOB_NAME/builds/$BUILD_NAME

- name: test
  plan:
  - get: docker-remote-tunnel-image
    passed: [build-cached-image]
    trigger: true
  - get: git-repo
    passed: [build-cached-image]
  - task: run-tests
    image: docker-remote-tunnel-image
    config:
      platform: linux
      inputs:
      - name: git-repo
      run:
        dir: git-repo
        path: sh
        args:
        - /usr/bin/docker-tunnel
        - --help

    on_failure:
      put: slack-alert
      params:
        channel: '#system_events'
        username: 'concourse'
        icon_emoji: ':concourse:'
        silent: true
        text: |
            *$BUILD_PIPELINE_NAME/$BUILD_JOB_NAME* ($BUILD_NAME) FAILED - Testing image failure
            https://ci.domain.com/teams/$BUILD_TEAM_NAME/pipelines/$BUILD_PIPELINE_NAME/jobs/$BUILD_JOB_NAME/builds/$BUILD_NAME
    on_success:
      put: slack-alert
      params:
        channel: '#system_events'
        username: 'concourse'
        icon_emoji: ':concourse:'
        silent: true
        text: |
            *$BUILD_PIPELINE_NAME/$BUILD_JOB_NAME* ($BUILD_NAME) SUCCESS - Testing image Succeeded
            https://ci.domain.com/teams/$BUILD_TEAM_NAME/pipelines/$BUILD_PIPELINE_NAME/jobs/$BUILD_JOB_NAME/builds/$BUILD_NAME

Note that our secret information is templatized and saved in our local credentials.yml which should never be stored in version control:

slack_notification_url: https://api.slack.com/aaa/bbb/ccc
dockerhub_user: myuser
dockerhub_password: mypasswd
github_private_key: |-
        -----BEGIN RSA PRIVATE KEY-----
        some-secret-data
        -----END RSA PRIVATE KEY------

Set the Pipeline:

Now that we have our pipeline definition, credentials and application code (stored in version control), go ahead and set the pipeline, which will save the pipeline configuration in concourse:

# pipeline name: my-docker-app-pipeline
$ fly -t scw sp -n main -c pipeline.yml -p my-docker-app-pipeline -l credentials.yml

Now the pipeline is saved on concourse but in a paused state, go ahead and unpause the pipeline:

$ fly -t scw up -p my-docker-app-pipeline

Test your Pipeline

Make a commit to master and head over to concourse and look at it go:

Thanks for reading, make sure to check out my other posts on #concourse

Comments

[Tim Downey]

Nice post! I recently had to do this and having more resources like this available would have been helpful. 🙌

Have you played around with this new OCI registry image resource type/build task at all?

• github.com/vito/oci-build-task
• github.com/concourse/registry-imag...

I haven't, but I think the goal of eliminating the Docker daemon (and eventually privileged containers as well) requirement is a noble one.

Running Docker within Concourse's Garden containers can be rough sometimes. 😔

[Ruan Bekker]

Hey Tim,

I have not played around with them, but thanks for mentioning them, those look really useful. Think I will start to use them right away.

Totally agree about moving away from the docker daemon is a great one!

Thanks so much for the comment.