DEV Community: Taylor Silva

Introduction to Concourse Task Inputs & Outputs

Taylor Silva — Fri, 26 Jun 2020 18:26:42 +0000

Understanding how task inputs and outputs work in Concourse can be a little confusing initially. This post will walk you through a few example pipelines to show you how inputs and outputs work within a single Concourse job. By the end you should understand how inputs and outputs work within the context of a single job.

This was originally posted on the Concourse blog.

Let's define some jargon first.

step: A step is a container running code within the context of a Concourse job. A step may have inputs and/or outputs, or neither.
Job plan: A list of steps that a job will execute when triggered.
Inputs and Outputs: These are directories. Within Concourse they're generically referred to as artifacts. These artifacts are mounted in a step's container under a directory with some name. You, as a writer of Concourse pipelines, have control over what the name of your artifacts will be. If you're coming from the Docker world, artifact is synonymous with volumes.

To run the pipelines in the following examples yourself you can get your own Concourse running locally by following the Quick Start guide. Then use fly set-pipeline to see the pipeline in action.

Concourse pipelines contain a lot of information. Within each pipeline YAML there are comments to help bring specific lines to your attention.

Example One - Two Tasks

This pipeline will show us how to create outputs and pass outputs as inputs to the next step(s) in a job plan.

This pipeline has two tasks. The first task outputs a file with the date. The second task reads and prints the contents of the file from the first task.

---
jobs:
- name: a-job
  plan:
  - task: create-one-output
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: {repository: alpine}
      outputs:
        # Concourse will make an empty dir with this name
        # and save the contents for later steps
        - name: the-output
      run:
        path: /bin/sh
        args:
          - -cx
          - |
            ls -lah
            date > ./the-output/file
  - task: read-ouput-from-previous-step
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: {repository: alpine}
      # You must explicitly name the inputs you expect
      # this task to have.
      # If you don't then outputs from previous steps
      # will not appear in th step's container.
      # The name must match the output from the previous step.
      # Try removing or renaming the input to see what happens!
      inputs:
        - name: the-output
      run:
        path: /bin/sh
        args:
          - -cx
          - |
            ls -lah
            cat ./the-output/file

Here's a visual graphic of what happens when the above job is executed.

Example Two - Two tasks with the same output, who wins?

This example is to satisfy the curiosity cat inside all of us! Never do this in real life because you're definitely going to hurt yourself!

There are two jobs in this pipeline. The first job has two steps; both steps will produce an artifact named the-output in parallel. If you run the writing-to-the-same-output-in-parallel job multiple times you'll see the file in the-output folder changes depending on which of the parallel tasks finished last. Here's a visualization of the first job.

The second job is a serial version of the first job. In this job the second task always wins because it's the last task that outputs the-output, so only file2 will be in the-output directory in the last step in the job plan.

This pipeline illustrates that you could accidentally overwrite the output from a previous step if you're not careful with the names of your outputs.

---
jobs:
- name: writing-to-the-same-output-in-parallel
  plan:
  # running two tasks that output in parallel?!?
  # who will win??
  - in_parallel:
    - task: create-the-output
      config:
        platform: linux
        image_resource:
          type: registry-image
          source: {repository: busybox}
        outputs:
          - name: the-output
        run:
          path: /bin/sh
          args:
            - -cx
            - |
              ls -lah
              date > ./the-output/file1
    - task: also-create-the-output
      config:
        platform: linux
        image_resource:
          type: registry-image
          source: {repository: busybox}
        outputs:
          - name: the-output
        run:
          path: /bin/sh
          args:
            - -cx
            - |
              ls -lah
              date > ./the-output/file2
  # run this job multiple times to see which
  # previous task wins each time
  - task: read-ouput-from-previous-step
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: {repository: busybox}
      inputs:
        - name: the-output
      run:
        path: /bin/sh
        args:
          - -cx
          - |
            ls -lah ./the-output
            echo "Get ready to error!"
            cat ./the-output/file1 ./the-output/file2

- name: writing-to-the-same-output-serially
  plan:
  - task: create-one-output
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: {repository: busybox}
      outputs:
        - name: the-output
      run:
        path: /bin/sh
        args:
          - -cx
          - |
            ls -lah
            date > ./the-output/file1
  - task: create-another-output
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: {repository: busybox}
      outputs:
        - name: the-output
      run:
        path: /bin/sh
        args:
          - -cx
          - |
            ls -lah
            date > ./the-output/file2
  - task: read-ouput-from-previous-step
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: {repository: busybox}
      inputs:
        - name: the-output
      run:
        path: /bin/sh
        args:
          - -cx
          - |
            ls -lah ./the-output
            echo "Get ready to error!"
            cat ./the-output/file1 ./the-output/file2

Example Three - Input/Output Name Mapping

Sometimes the names of inputs and outputs don't match, or they do match and you don't want them overwriting each other, like in the previous example. That's when input_mapping and output_mapping become helpful. Both of these features map the inputs/outputs in the task's config to some artifact name in the job plan.

This pipeline has one job with four tasks.

The first task outputs a file with the date to the the-output directory. the-output is mapped to the new name demo-disk. The artifact demo-disk is now available in the rest of the job plan for future steps to take as inputs. The remaining steps do this in various ways.

The second task reads and prints the contents of the file under the new name demo-disk.

The third task reads and prints the contents of the file under another name, generic-input. The demo-disk artifact in the job plan is mapped to generic-input.

The fourth task tries to use the artifact named the-output as its input. This task fails to even start because there was no artifact with the name the-output available in the job plan; it was remapped to demo-disk.

Here's a visualization of the job.

Here's the pipeline YAML for you to run on your local Concourse.

---
jobs:
- name: a-job
  plan:
  - task: create-one-output
    # The task config has the artifact `the-output`
    # output_mapping will rename `the-output` to `demo-disk`
    # in the rest of the job's plan
    output_mapping:
      the-output: demo-disk
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: {repository: busybox}
      outputs:
        - name: the-output
      run:
        path: /bin/sh
        args:
          - -cx
          - |
            ls -lah
            date > ./the-output/file
  # this task expects the artifact `demo-disk` so no mapping is needed
  - task: read-ouput-from-previous-step
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: {repository: busybox}
      inputs:
        - name: demo-disk
      run:
        path: /bin/sh
        args:
          - -cx
          - |
            ls -lah
            cat ./demo-disk/file
  - task: rename-and-read-output
    # This task expects the artifact `generic-input`.
    # input_mapping will map the tasks `generic-input` to
    # the job plans `demo-disk` artifact
    input_mapping:
      generic-input: demo-disk
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: {repository: busybox}
      inputs:
        - name: generic-input
      run:
        path: /bin/sh
        args:
          - -cx
          - |
            ls -lah
            cat ./generic-input/file
  - task: try-and-read-the-output
    input_mapping:
      generic-input: demo-disk
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: {repository: busybox}
      # `the-output` is not available in the job plan
      # so this task will error while initializing
      # since there's no artiact named `the-output` in
      # the job's plan
      inputs:
        - name: the-output
      run:
        path: /bin/sh
        args:
          - -cx
          - |
            ls -lah
            cat ./generic-input/file

Example Four - Can you add files to an existing output artifact?

This pipeline will also have two jobs in order to illustrate this point. What happens if we add a file to an output? If you think back to example two you may already know the answer.

The first task will create the-output with file1. The second task will add file2 to the the-output. The last task will read the contents of file1 and file2.

As long as you re-declare the input as an output in the second task you can modify any of your outputs.

This means you can pass something between a bunch of tasks and have each task add or modify something in the artifact.

---
jobs:
- name: add-file-to-output
  plan:
  - task: create-one-output
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: {repository: busybox}
      outputs:
        - name: the-output
      run:
        path: /bin/sh
        args:
          - -cx
          - |
            ls -lah
            date > ./the-output/file1
  - task: add-file-to-previous-output
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: {repository: busybox}
      # this task lists the same artifact as
      # its input and output
      inputs:
        - name: the-output
      outputs:
        - name: the-output
      run:
        path: /bin/sh
        args:
          - -cx
          - |
            ls -lah
            date > ./the-output/file2
  - task: read-ouput-from-previous-step
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: {repository: busybox}
      inputs:
        - name: the-output
      run:
        path: /bin/sh
        args:
          - -cx
          - |
            ls -lah ./the-output
            cat ./the-output/file1 ./the-output/file2

Here's a visualization of the job.

Example Five - Multiple Outputs

What happens if you have a task that has multiple outputs and a second task that only lists one of the outputs? Does the second task get the extra outputs from the first task?

The answer is no. A task will only get the artifacts that match the name of the inputs listed in the task's config.

---
jobs:
- name: multiple-outputs
  plan:
  - task: create-three-outputs
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: {repository: busybox}
      outputs:
        - name: the-output-1
        - name: the-output-2
        - name: the-output-3
      run:
        path: /bin/sh
        args:
          - -cx
          - |
            ls -lah
            date > ./the-output-1/file
            date > ./the-output-2/file
            date > ./the-output-3/file
  - task: take-one-output
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: {repository: busybox}
      # only one of the three outputs are
      # listed as inputs
      inputs:
        - name: the-output-1
      run:
        path: /bin/sh
        args:
          - -cx
          - |
            ls -lah ./
            cat ./the-output-1/file
  - task: take-two-outputs
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: {repository: busybox}
      # this task pulls in the other
      # two outputs, just for fun!
      inputs:
        - name: the-output-2
        - name: the-output-3
      run:
        path: /bin/sh
        args:
          - -cx
          - |
            ls -lah ./
            cat ./the-output-2/file
            cat ./the-output-3/file

Here's a visualization of the job.

Example Six - Get Steps

The majority of Concourse pipelines have at least one resource, which means they have at least one get step. Using a get step in a job makes an artifact with the name of the get step available for later steps in the job plan to consume as inputs.

---
resources:
- name: concourse-examples
  type: git
  source: {uri: "https://github.com/concourse/examples"}

jobs:
- name: get-step
  plan:
  # there will be an artifact named
  # "concourse-examples" available in the job plan
  - get: concourse-examples
  - task: take-one-output
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: {repository: busybox}
      inputs:
        - name: concourse-examples
      run:
        path: /bin/sh
        args:
          - -cx
          - |
            ls -lah ./
            cat ./concourse-examples/README.md

Here's a visualization of the job.

I hope you found these example helpful with figuring out how inputs and outputs work within a single Concourse job. Leave a comment if you still have any questions.

The Learning Test

Taylor Silva — Sun, 03 Mar 2019 20:48:36 +0000

I'm reading TDD By Example by Kent Beck right now and one of the Red Bar Patterns really resonated with me. It's called the Learning Test.

Here's the scenario. You have a new library or framework that you're implementing. Instead of just diving into TDD'ing the implementation, you can write some learning tests. These are tests where you validate how you expect the 3rd party tool to work for you. I see this as testing the contract between your application and the 3rd party tool.

Here's an example that I did using github.com/aclindsa/ofxgo.

Now this library does a lot and I really only care about one small part. This library can interact with OFX servers but my current reason for using it is to parse OFX files that have already been downloaded to the local file system.

package main

import (
    . "github.com/onsi/gomega"
    "testing"
)

func TestReadLocalOFX(t *testing.T) {
    g := NewGomegaWithT(t)  
}

So after looking through some of the functions it looks like I want to use this ParseResponse() function.

I'll want to assert something after parsing to verify that it worked. Looking at the example in the README it looks like I can parse how many banks are contained in the OFX file.

func TestReadLocalOFX(t *testing.T) {
    g := NewGomegaWithT(t)

    g.Expect(len(banks)).To(Equal(1))
}

Now I need to learn how to get that banks variable.

From ofxgo I don't need to initialize anything, I can just directly call ParseResponse() and pass in an open file. That seems like a good first step; parse the OFX file!

import (
    "github.com/aclindsa/ofxgo"
    . "github.com/onsi/gomega"
    "testing"
)

func TestReadLocalOFX(t *testing.T) {
    g := NewGomegaWithT(t)

    resp, err := ofxgo.ParseResponse(ofxFile)

    g.Expect(len(banks)).To(Equal(1))
}

I'm going to need an OFX file for testing as well. So I head over to my banks site and download a OFX/QFX file and put it in the same directory as my test. I can now open the file from my test:

func TestReadLocalOFX(t *testing.T) {
    g := NewGomegaWithT(t)

    ofxFile, err := os.Open("ofxFile.qfx")
    g.Expect(err).ShouldNot(HaveOccurred())

    resp, err := ofxgo.ParseResponse(ofxFile)
    g.Expect(err).ShouldNot(HaveOccurred())

    g.Expect(len(banks)).To(Equal(1))
}

Now I can get the banks variable from the *Reponse from ParseResponse().

func TestReadLocalOFX(t *testing.T) {
    g := NewGomegaWithT(t)

    ofxFile, err := os.Open("ofxFile.qfx")
    g.Expect(err).ShouldNot(HaveOccurred())

    resp, err := ofxgo.ParseResponse(ofxFile)
    g.Expect(err).ShouldNot(HaveOccurred())

    banks := resp.Bank
    g.Expect(len(banks)).To(Equal(1))
}

And now if I run my test it should work!

=== RUN   TestReadLocalOFX
--- FAIL: TestReadLocalOFX (0.00s)
....    
        Expected error:
            <*errors.errorString | 0xc000091050>: {
                s: "OFX SECURITY header not NONE",
            }
            OFX SECURITY header not NONE
        not to have occurred
FAIL
FAIL    ofxreader   0.018s

It failed! Already this test is teaching me something about this library. It's expecting this OFX SECURITY header to be set to NONE. Looking at my OFX file I see this as the header:

OFXHEADER:100
DATA:OFXSGML
VERSION:102
SECURITY:TYPE1
ENCODING:USASCII
CHARSET:1252
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:NONE

It says security is TYPE1. I decide to search the repo for this error and find this: https://github.com/aclindsa/ofxgo/blob/3e8a9c5a5382dccc0fd89b4d869e2202a48e8a66/response.go#L90-L92

This library only works if the security header is set to NONE. I'll want to add this learning to my test so I don't forget.

import (
    "github.com/aclindsa/ofxgo"
    . "github.com/onsi/ginkgo"
    . "github.com/onsi/gomega"
    "io/ioutil"
    "strings"
    "testing"
)

func TestReadLocalOFX(t *testing.T) {
    g := NewGomegaWithT(t)

    By("setting OFX security header to NONE")
    file, err := ioutil.ReadFile("ofxFile.qfx")
    g.Expect(err).ShouldNot(HaveOccurred())
    newOFXFile := strings.Replace(string(file), "SECURITY:TYPE1", "SECURITY:NONE", 1)

    ofxFile := strings.NewReader(newOFXFile)

    resp, err := ofxgo.ParseResponse(ofxFile)
    g.Expect(err).ShouldNot(HaveOccurred())

    banks := resp.Bank
    g.Expect(len(banks)).To(Equal(1))
}

And now if I run the test:

=== RUN   TestReadLocalOFX
STEP: setting OFX security header to NONE
--- PASS: TestReadLocalOFX (0.01s)
PASS
ok      ofxreader   0.022s

It passes! That's one learning test for this library.

I'm hoping to bring learning tests to my team at work because I think it'll help spread knowledge among all team members and become useful reference points.

Do you use learning tests at work? Have they been helpful to you and your team?

Small Lesson About Pointers & Slices in Go

Taylor Silva — Sat, 02 Feb 2019 02:05:45 +0000

Today I learned a little bit more about how pointers work in Go.

We had this object that we want other functions to directly modify. So we pass a pointer to this object to various functions and then in our tests we check to see that the original object has been modified correctly. We also have these helper functions that are supposed to return a pointer to an object buried inside of the struct.

Here's an example of our object:

type obj struct {
    num   int
    str   string
    slice []int
}

And then we had a helper function that is supposed to return a reference to an element inside of the slice.

Working with our example object, I had a helper function that looked like this:

func FindNumber(object *obj, findInt int) *int {
    for _, singleInt := range object.slice {
        if singleInt == findInt {
            return &singleInt
        }
    }
    return nil
}

It's supposed to go into a given object and return a pointer to the matching element inside the obj.slice parameter. If we put this together right now we get this program:

package main

import "fmt"

type obj struct {
    num   int
    str   string
    slice []int
}

func main() {
    myObject := obj{num: 1, str: "hello", slice: []int{1, 2, 3}}

    pointerToNumber := FindNumber(&myObject, 2)

    *pointerToNumber = 5 // change the value from 2 to 5
    fmt.Printf("%p - %v - from FindObject()\n", pointerToNumber, *pointerToNumber)
    fmt.Printf("%p - %v - Pointer address I actually wanted\n", &myObject.slice[1], myObject.slice[1])
}

func FindNumber(object *obj, findInt int) *int {
    for _, singleInt := range object.slice {
        if singleInt == findInt {
            return &singleInt
        }
    }
    return nil
}

And when we run it we get this output:

0xc00001a0b8 - 5 - from FindObject()
0xc0000182e8 - 2 - Pointer address I actually wanted

The value didn't change to 5 in my original object and we can see the pointer address doesn't match. Any changes I make to the value of this pointer won't be reflected in the myObject variable.

Let's think about why this is happening, going line by line.

First we make our object:

myObject := obj{num: 1, str: "hello", slice: []int{1, 2, 3}}

and we pass a pointer of our object to FindNumber(), telling it to return a pointer to the element within the slice that equals 2:

pointerToNumber := FindNumber(&myObject, 2)

Inside FindNumber() we we get the slice from the pointer and iterate over it:

for _, singleInt := range object.slice {
    ...
}

Let's stop and think about this for loop. We have a pointer to object and we're using that pointer to access the slice parameter. Let's keep in mind that a slice is just a pointer to an array and not a pointer to each element contained in the array.

So when range gets an element from whatever slice it's been passed, it passes the value of that element to our singleInt variable. So singleInt is not a pointer to the element, it's just a copy of the original element! If I want to return a pointer to the actual element in the slice I'll have to use the index returned from range and return the pointer from the original slice:

func FindNumber(object *obj, findInt int) *int {
    for index, singleInt := range object.slice {
        if singleInt == findInt {
            return &object.slice[index]
        }
    }
    return nil
}

Now my output is this:

0xc0000182e8 - 5 - from FindObject()
0xc0000182e8 - 5 - Pointer address I actually wanted

The addresses match and the value changed!

There's also another way to solve this problem that shows some of the dangers with pass-by-value in Go.

If we change FindNumber() to not accept a pointer:

func FindNumber(object obj, findInt int) *int {
    fmt.Printf("%p - address of object in FindNumber\n", &object)
    ...
}

And not pass in a pointer to our object:

fmt.Printf("%p - address of myObject\n", &myObject)
pointerToNumber := FindNumber(myObject, 2)

and run our code now:

0xc000076180 - address of myObject
0xc0000761b0 - address of object in FindNumber
0xc000090028 - 5 - from FindObject()
0xc000090028 - 5 - Pointer address I actually wanted

We see that FindNumber() got a copy of the original object but the pointer it returned still allowed us to change the value in the original object. So when we pass-by-value any struct, the slices inside the struct point to the same array. And this makes sense because a pointer is just a memory address and copying an address doesn't change the location of the address. If a pointer says go to 0xc000076180, when I copy that, my new pointer will also say go to 0xc000076180.

If you want to avoid accidentally modifying the original slice then you should create a copy() of the slice.

Hope this taught you a little more about slices and pointers in Go.

Creating Your Own BeforeEach() in Go

Taylor Silva — Mon, 07 Jan 2019 02:59:46 +0000

I'm developing my own budgeting app using Go and have been using it to learn new things as well. For testing, because I'm TDD'ing the entire project, I decided to just use the built-in Go testing framework. I use Ginkgo at work and enjoy it, but decided to see what vanilla testing in Go is like.

After writing a bunch of tests for a lot of my boilerplate functions, I quickly had a lot of duplicated set-up code that looked like this at the start of all my tests:

g := NewGomegaWithT(t)

db, err := createTestDb()
g.Expect(err).To(BeNil(), ErrorDidNotOccur)

api := NewApi(db, SelectedQueryLang)

I was having trouble figuring out how to pull this out into it's own function. Finally I saw a Stack Overflow answer that provided me with a solution (having trouble finding the link now!).

Here's how you create your own BeforeEach() function when using Go's testing framework.

First, create a struct that will hold the objects your test needs to access. Mine looks like this:

type TestingBasics struct {
    GoMega *GomegaWithT
    Db     *sql.DB
    Api    *api
}

Then create your BeforeEach() function that will create this struct for your tests.

func BeforeEach(t *testing.T, SelectedQueryLang *QueryLang) (tb TestingBasics) {
    tb.GoMega = NewGomegaWithT(t)

    var err error
    tb.Db, err = createTestDb()
    tb.GoMega.Expect(err).To(BeNil(), ErrorDidNotOccur)

    tb.Api = NewApi(tb.Db, SelectedQueryLang)

    return tb
}

Now all you need to do is call BeforeEach() at the beginning of your tests. I was able to reduce 4 lines of code down to 1 across all of my tests. Now the start of each of my tests look like this:

t.Run("should add one transaction", func(t *testing.T) {
    tb := BeforeEach(t, SelectedQueryLang)
...

This can definitely be cleaned up even further, to the point where I wouldn't even need to call BeforeEach() at the start of each test. At that point though, I might as well have gone with Ginkgo. For me and this project, I am fine with this set up.