DEV Community: Christian Danielsen

Slice and dice your commits with these git flags

Christian Danielsen — Mon, 13 Jan 2020 18:16:09 +0000

Ah git. The canonical example of "easy to learn, acquire a PhD to master". Perhaps someday I'll learn about the subtle mysteries of git bisect or git rev-parse (wut?) but for now, here are some practical (and simple!) commands you can use every day:

git add --pick

You always only work on one thing at a time in between commits, right? Uh, yeah, me too! 😁 For those rare times you don't, you can use this command to only add selected parts of a changed file. git adorably calls these selections "hunks" 💪

Let's see what this would look like! Here's a repo with just one file that I've made changes to, with the changes displayed via git diff:

$ git diff
diff --git a/sweet-sweet-file.js b/sweet-sweet-file.js
index 85db7f4..e3752aa 100644
--- a/sweet-sweet-file.js
+++ b/sweet-sweet-file.js
@@ -1,3 +1,7 @@
// sweet-sweet-file.js

+export const relevantFunction = () => 'I am so relevant to this commit';
+
 export const initialFunction = () => 'Ahoy!';
+
+export const notReleventFunction = () => 'I am so related to something else';

Here the + signs indicate lines in the file that are additions that haven't been added to the staging area yet (- signs would indicate removed lines, but we don't have any).

I'm ready to commit relevantFunction, but got distracted and also fleshed out notRelevantFunction, so I don't want to just git add this file in its entirety. Enter git add --pick (or git add -p for short), which presents me with a command line interface for picking which parts of the file I want to add to the staging area:

$ git add -p
diff --git a/sweet-sweet-file.js b/sweet-sweet-file.js
index 85db7f4..e3752aa 100644
--- a/sweet-sweet-file.js
+++ b/sweet-sweet-file.js
@@ -1,3 +1,7 @@
 // sweet-sweet-file.js

+export const relevantFunction = () => 'I am so relevant to this commit';
+
 export const initialFunction = () => 'Ahoy!';
+
+export const notReleventFunction = () => 'I am so related to something else';
Stage this hunk [y,n,q,a,d,s,e,?]?

Ooo, cryptic one letter options! You can ~~git~~ get more detail by entering ?

Stage this hunk [y,n,q,a,d,s,e,?]? ?
y - stage this hunk
n - do not stage this hunk
q - quit; do not stage this hunk or any of the remaining ones
a - stage this hunk and all later hunks in the file
d - do not stage this hunk or any of the later hunks in the file
s - split the current hunk into smaller hunks
e - manually edit the current hunk
? - print help
@@ -1,3 +1,7 @@
 // sweet-sweet-file.js

+export const relevantFunction = () => 'I am so relevant to this commit';
+
 export const initialFunction = () => 'Ahoy!';
+
+export const notReleventFunction = () => 'I am so related to something else';
Stage this hunk [y,n,q,a,d,s,e,?]?

Now we're talking! Git will work its way through the file and identify "hunks" of changes, asking you what you want to do with each one. y will add the hunk, n will skip it, and q will get you out of here.

There are some additional interesting options, particularly s, which allows you to get really surgical and split up hunks into smaller pieces! This is exactly what we want here, as we only want relevantFunction. Let's do it:

Stage this hunk [y,n,q,a,d,s,e,?]? s
Split into 2 hunks.
@@ -1,3 +1,5 @@
 // sweet-sweet-file.js

+export const relevantFunction = () => 'I am so relevant to this commit';
+
 export const initialFunction = () => 'Ahoy!';
Stage this hunk [y,n,q,a,d,j,J,g,/,e,?]?

Hooray! Now we can hit y to just add this part of the file that we want, and then n to discard the second hunk with notRelevantFunction in it, returning us to the command line as we've reached the end of the file:

Stage this hunk [y,n,q,a,d,j,J,g,/,e,?]? y
@@ -3 +5,3 @@
 export const initialFunction = () => 'Ahoy!';
+
+export const notReleventFunction = () => 'I am so related to something else';
Stage this hunk [y,n,q,a,d,K,g,/,e,?]? n

$

Let's see where we stand with git status:

$ git status
On branch master
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

        modified:   sweet-sweet-file.js

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   sweet-sweet-file.js

The first time I used this command, it blew my mind a bit: sweet-sweet-file.js is both staged... and not-staged! 🤯 Once you get past the idea of committing files and instead committing changes (which git is perfectly able and willing to do) this feels totally normal (and awesome!).

git diff --staged

Now that we've partially committed a file, how can we confirm we did it right? Let's try git diff:

$ git diff
diff --git a/sweet-sweet-file.js b/sweet-sweet-file.js
index c48b8c3..e3752aa 100644
--- a/sweet-sweet-file.js
+++ b/sweet-sweet-file.js
@@ -3,3 +3,5 @@
 export const relevantFunction = () => 'I am so relevant to this commit';

 export const initialFunction = () => 'Ahoy!';
+
+export const notReleventFunction = () => 'I am so related to something else';

Probably what we should expect: git is telling us that notRelevantFunction is an un-added addition to sweet-sweet-file.js. With no options, git diff will just look at what's different between the file's existing state and what's staged and/or committed. If you just want to see the difference between what's staged and what's committed, you can add the --staged flag (--cached also works!):

$ git diff --staged
diff --git a/sweet-sweet-file.js b/sweet-sweet-file.js
index 85db7f4..c48b8c3 100644
--- a/sweet-sweet-file.js
+++ b/sweet-sweet-file.js
@@ -1,3 +1,5 @@
 // sweet-sweet-file.js

+export const relevantFunction = () => 'I am so relevant to this commit';
+
 export const initialFunction = () => 'Ahoy!';

There we go -- confirmation that committing what's in the staging area will just commit relevantFunction 🎉

git checkout --pick

Sometimes instead of affirmatively adding hunks, it's easier to just subtract a few from an otherwise solid set of changes to a file.

You can use git checkout <file> to reset a file to it's last committed state, but if you only want to remove a few things, --pick works for git checkout too! Think of it as basically the opposite of git add --pick. Here we've made another set of changes:

$ git diff
diff --git a/sweet-sweet-file.js b/sweet-sweet-file.js
index c48b8c3..cb9c9c7 100644
--- a/sweet-sweet-file.js
+++ b/sweet-sweet-file.js
@@ -1,5 +1,9 @@
 // sweet-sweet-file.js

+export const no = () => 'This is something experimental that should go away';
+
 export const relevantFunction = () => 'I am so relevant to this commit';

 export const initialFunction = () => 'Ahoy!';
+
+export const anEvenBetterFunction = () => 'This function is so good';

Let's rid ourselves of the no function, so we can just have the changes we want and commit the entire file:

$ git checkout --pick
diff --git a/sweet-sweet-file.js b/sweet-sweet-file.js
index c48b8c3..cb9c9c7 100644
--- a/sweet-sweet-file.js
+++ b/sweet-sweet-file.js
@@ -1,5 +1,9 @@
 // sweet-sweet-file.js

+export const no = () => 'This is something experimental that should go away';
+
 export const relevantFunction = () => 'I am so relevant to this commit';

 export const initialFunction = () => 'Ahoy!';
+
+export const anEvenBetterFunction = () => 'This function is so good';
Discard this hunk from worktree [y,n,q,a,d,s,e,?]?

Notice the new command line prompt asking if you want to Discard this hunk from the "worktree" (your unstaged/uncommitted changes). Like before, we need to split the hunk in two:

Discard this hunk from worktree [y,n,q,a,d,s,e,?]? s
Split into 2 hunks.
@@ -1,5 +1,7 @@
 // sweet-sweet-file.js

+export const no = () => 'This is something experimental that should go away';
+
 export const relevantFunction = () => 'I am so relevant to this commit';

 export const initialFunction = () => 'Ahoy!';
Discard this hunk from worktree [y,n,q,a,d,j,J,g,/,e,?]?

Then rid ourselves of the hunk we want to discard by entering y:

Discard this hunk from worktree [y,n,q,a,d,j,J,g,/,e,?]? y
@@ -3,3 +5,5 @@
 export const relevantFunction = () => 'I am so relevant to this commit';

 export const initialFunction = () => 'Ahoy!';
+
+export const anEvenBetterFunction = () => 'This function is so good';
Discard this hunk from worktree [y,n,q,a,d,K,g,/,e,?]?

And keeping the one we want (i.e. don't discard) with n, returning us to the command line. Now we can use plain old git diff to see that only the changes we want are ready to be git add'ed:

λ git diff
diff --git a/sweet-sweet-file.js b/sweet-sweet-file.js
index c48b8c3..cb8545c 100644
--- a/sweet-sweet-file.js
+++ b/sweet-sweet-file.js
@@ -3,3 +3,5 @@
 export const relevantFunction = () => 'I am so relevant to this commit';

 export const initialFunction = () => 'Ahoy!';
+
+export const anEvenBetterFunction = () => 'This function is so good';

Conclusion

Adding some options to git add, git checkout and git diff allows you to commit more surgically within files, and make sure you did it right! Are there other options for fine grain commit polishing? Let me know in the comments!

Testing streams: a primer

Christian Danielsen — Wed, 13 Nov 2019 05:44:16 +0000

In the last post, I showed how nicely streams and promises play well together. We designed this simple function to demonstrate:

const streamToFile = (inputStream, filePath) => {
  return new Promise((resolve, reject) => {
    const fileWriteStream = fs.createWriteStream(filePath)
    inputStream
      .pipe(fileWriteStream)
      .on('finish', resolve)
      .on('error', reject)
    })
}

What would a good set of tests for this function look like and how would we write them? My general mental model when I write unit tests for a function is to think about the question "What is this function's job?" In this case, I'd say this function has two high level responsibilities/behaviors to verify:

Signal if the incoming data is streamed correctly
Error gracefully if not

I try to answer this question without any technical jargon as a reminder that, as much as possible, tests should avoid testing implementation details and focus instead on desired outcomes. Unfortunately, this is a case where the function's main duty is a side effect (writing to the file system), so we will probably have to bend or break this rule.

This function actually presents a couple of testing pitfalls that we will want to carefully navigate:

This function writes data to disk, which we don't want to do in a unit test
This function has async behavior via listening for events, which we will want to simulate and verify

I've been using jest to unit test lately. Here's where I might start with this test:

describe('streamToFile', () => {
  it('rejects with an error if a stream error occurs', async () => {
    await expect(streamToFile()).rejects.toEqual('ahoy!')
  })
})

If all tests follow the classic stages of Arrange, Act, Assert, I like to start at the end with the Assert stage, to remind myself what I'm working toward, and to let the test runner give me a road map for what I need to fix.

I also always put in a purposefully dumb assertion (ahoy!) so that I know when I've got my test right because it's failing for the right reason (because false positives are... the worst). We'll update that at the very end to make it pass.

This test will currently fail because we're not passing the correct arguments to streamToFile. Let's start fixing!:

const { PassThrough } = require('stream')

describe('streamToFile', () => {
  it('rejects/errors if a stream error occurs', async () => {
    // Arrange
    const mockReadable = new PassThrough()
    const mockFilePath = '/oh/what/a/file.txt'

    // Act & Assert
    await expect(streamToFile(mockReadable, mockFilePath))
      .rejects.toEqual('ahoy!')
  })
})

Now we've satisfied streamToFile's signature with two things:

mockReadable is a PassThrough steam. As the docs note, this is a very simple implementation of a Transform stream that simply takes the input and passes it right through. It implements the Readable and Writeable APIs, making it very handy for testing.
mockFilePath is... a fake file path. When mocking, I think it's important to stick as close to what the actual code wants as possible, so your tests also serve as documentation. In this case, we're going to use jest's auto-mocking feature to hijack the fs.createWriteStream method so we don't actually touch the file system. Since this doppelgänger won't actually be doing anything with the input, we could technically (obnoxiously?) pass that hijacked function whatever we want, but this way someone can look at the test and start to understand the ins and outs of the function under test.

Since we don't actually want to write any data to our fake file path, let's do the hijacking:

const fs = require('fs')
const { PassThrough } = require('stream')

jest.mock('fs')

describe('streamToFile', () => {
  it('rejects/errors if a stream error occurs', async () => {
    // Arrange
    const mockReadable = new PassThrough()
    const mockWriteable = new PassThrough()
    const mockFilePath = '/oh/what/a/file.txt'
    fs.createWriteStream.mockReturnValueOnce(mockWriteable)

    // Act & Assert
    await expect(streamToFile(mockReadable, mockFilePath))
      .rejects.toEqual('ahoy!')
  })
})

Now we've done three things:

Use jest's automocking to replace all methods on the fs object with jest functions that do nothing and return undefined by default
Overridden that default behavior in this test by using the mockReturnValueOnce method
Returning what our code would expect from fs.createWriteStream : something that implements the Writeable interface (another PassThrough stream!).

Now our code will no longer attempt to touch the file system, and we control/have references to the streams. Noice.

Our test will now fail with a timeout because the error event is never called (we haven't emitted one).

And this is where things get a little weird. Usually you just have one "Act" statement in a unit test - invoking the thing you're testing - but in this case we actually need two.

Invoking streamToFile hooks the streams up using .pipe but then we also need to emit an event to run something through those beautiful tubes (an error in this case).

We're currently using the await/expect combined syntax to Act and Assert in the same line. Usually this is fine/terse/convenient, but in this case, not so much, because we want to do something after the promise has been created, but before it has settled. Let's separate those out:

describe('streamToFile', () => {
  it('rejects/errors if a stream error occurs', async () => {
    // Arrange
    const mockReadable = new PassThrough()
    const mockWriteable = new PassThrough()
    const mockFilePath = '/oh/what/a/file.txt'
    fs.createWriteStream.mockReturnValueOnce(mockWriteable)

    // Act
    const actualPromise = streamToFile(mockReadable, mockFilePath)

    // Assert
    await expect(actualPromise).rejects.toEqual('ahoy!')
  })
})

And now we can augment our "Act" section by emitting an error event.

describe('streamToFile', () => {
  it('rejects/errors if a stream error occurs', async () => {
    // Arrange
    const mockReadable = new PassThrough()
    const mockWriteable = new PassThrough()
    const mockFilePath = '/oh/what/a/file.txt'
    const mockError = new Error('You crossed the streams!')
    fs.createWriteStream.mockReturnValueOnce(mockWriteable)

    // Act
    const actualPromise = streamToFile(mockReadable, mockFilePath)
    mockReadable.emit('error', mockError)

    // Assert
    await expect(actualPromise).rejects.toEqual('ahoy!')
  })
})

This should work consistently (and you do want deterministic, flake free tests, don't you?), but there's a sneaky race condition afoot. If we think through what's happening in this test you might reasonably conclude the order of operations would be something like this:

Invoke streamToFile with mock inputs/dependencies and create a reference to the promise it generates
Emit an error event from one of those mock streams that simulates a file write error
Assert that the promise rejects with that error

That will likely be what happens, but it isn't deterministic. It's possible that the event could be emitted before the error event handler is registered, leading to a failed test with Jest complaining about an unhandled error event.

This very likely wouldn't be an issue in the actual application, as emitting the events would likely occur well after the streams had been connected. As such, it's better to have our test model that likely flow (and eliminate the race condition), by using good old setTimeout.

describe('streamToFile', () => {
  it('rejects/errors if a stream error occurs', async () => {
    // Arrange
    const mockReadable = new PassThrough()
    const mockWriteable = new PassThrough()
    const mockFilePath = '/oh/what/a/file.txt'
    const mockError = new Error('You crossed the streams!')
    fs.createWriteStream.mockReturnValueOnce(mockWriteable)

    // Act
    const actualPromise = streamToFile(mockReadable, mockFilePath)
    setTimeout(() => {
      mockReadable.emit('error', mockError)
    }, 100)

    // Assert
    await expect(actualPromise).rejects.toEqual('ahoy!')
  })
})

100 milliseconds is a trivial delay in human time, but it's a small eternity for the event loop. This should pretty much guarantee that our event is emitted only after we've hooked up a listener for it.

If we run this test we can see that it's failing for the right reason: remember that we gave it a dumb assertion on purpose at the very beginning. Now we can switch it out for the actual error and we have a test that proves streamToFile captures errors correctly.

describe('streamToFile', () => {
  it('rejects/errors if a stream error occurs', async () => {
    // Arrange
    const mockReadable = new PassThrough()
    const mockWriteable = new PassThrough()
    const mockFilePath = '/oh/what/a/file.txt'
    const mockError = new Error('You crossed the streams!')
    fs.createWriteStream.mockReturnValueOnce(mockWriteable)

    // Act
    const actualPromise = streamToFile(mockReadable, mockFilePath)
    setTimeout(() => {
      mockReadable.emit('error', mockError)
    }, 100)

    // Assert
    await expect(actualPromise).rejects.toEqual(mockError)
  })
})

Additional Tests

One of the many underrated benefits of writing tests is that you often end up discovering code paths (or catching bugs) you hadn't thought of while working through an implementation.

In this case, the test above proves that our implementation will handle an error from the writeable stream, but what about if the readable stream emits an error. Will it propagate through to our error handler? Let's find out:

describe('streamToFile', () => {
  it('rejects/errors if a READ stream error occurs', async () => {
    // Arrange
    const mockReadable = new PassThrough()
    const mockWriteable = new PassThrough()
    const mockFilePath = '/oh/what/a/file.txt'
    const mockError = new Error('You crossed the streams!')
    fs.createWriteStream.mockReturnValueOnce(mockWriteable)

    // Act
    const actualPromise = streamToFile(mockReadable, mockFilePath)
    setTimeout(() => {
      mockReadable.emit('error', mockError)
    }, 100)

    // Assert
    await expect(actualPromise).rejects.toEqual(mockError)
  })

  it('rejects/errors if a WRITE stream error occurs', async () => {
    // Arrange
    const mockReadable = new PassThrough()
    const mockWriteable = new PassThrough()
    const mockFilePath = '/oh/what/a/file.txt'
    const mockError = new Error('You crossed the streams!')
    fs.createWriteStream.mockReturnValueOnce(mockWriteable)

    // Act
    const actualPromise = streamToFile(mockReadable, mockFilePath)
    setTimeout(() => {
      mockWriteable.emit('error', mockError)
    }, 100)

    // Assert
    await expect(actualPromise).rejects.toEqual(mockError)
  })
})

Now we have two tests: quite similar, but the first emits an error from the readable stream, while the second emits an error from the writeable one. The writeable one passes... but the readable one does not!

This is because each stream has its own error event, and .pipe says nothing about sending errors through to the next stream. If we want to catch (and then reject) from them both, we need to update our implementation and register an error event on the readable stream too.

const streamToFile = (inputStream, filePath) => {
  return new Promise((resolve, reject) => {
    const fileWriteStream = fs.createWriteStream(filePath)
    inputStream
      .on('error', reject)
      .pipe(fileWriteStream)
      .on('finish', resolve)
      .on('error', reject)
  })
}

describe('streamToFile', () => {
  it('rejects/errors if a READ stream error occurs', async () => {
    // Arrange
    const mockReadable = new PassThrough()
    const mockWriteable = new PassThrough()
    const mockFilePath = '/oh/what/a/file.txt'
    const mockError = new Error('You crossed the streams!')
    fs.createWriteStream.mockReturnValueOnce(mockWriteable)

    // Act
    const actualPromise = streamToFile(mockReadable, mockFilePath)

    setTimeout(() => {
      mockReadable.emit('error', mockError)
    }, 100)

    // Assert
    await expect(actualPromise).rejects.toEqual(mockError)
  })
})

Now we're gracefully handling errors on the write AND read stream. Thanks, tests!

And now that we've tested the 'sad' code paths, we can finally test the happy path - a successful run of streamToFile:

describe('streamToFile', () => {
  it('resolves if the data writes successfully', async () => {
    // Arrange
    const mockReadable = new PassThrough()
    const mockWriteable = new PassThrough()
    const mockFilePath = '/oh/what/a/file.txt'
    const mockError = new Error('You crossed the streams!')
    fs.createWriteStream.mockReturnValueOnce(mockWriteable)

    // Act
    const actualPromise = streamToFile(mockReadable, mockFilePath)

    setTimeout(() => {
      mockReadable.emit('data', 'beep!')
      mockReadable.emit('data', 'boop!')
      mockReadable.emit('end')
    }, 100)

    // Assert
    await expect(actualPromise).resolves.toEqual(undefined)
  })
})

Note that we're not resolving the promise with a value, forcing us to assert - ick - on the default value of undefined. But perhaps this function would evolve to resolve with some useful value for further downstream work.

Conclusion

In this post, we covered some techniques for testing promised-wrapped streams. Thoughts? Critiques? Let me know in the comments!

In the next post, I'll discuss a better way to handle the error propagation gotcha we ran into, with Stream.pipeline.

Wrap your streams with promises for fun and profit

Christian Danielsen — Thu, 07 Nov 2019 08:02:18 +0000

I recently worked on a service that involved consuming large amounts of data, processing it in various ways, and then sending it somewhere else. This seemed like the perfect opportunity to dig into one of node's core concepts: streams.

Streams are everywhere in node, and once you get the hang of using them, they are delightful. There are many solid articles that cover the basics, but I wanted to touch on something that came up while combining streams with another core JS concept: promises.

Let's say we wanted to create a function that would stream a large data source to a file:

const streamToFile = (inputStream, filePath) => {
  const fileWriteStream = fs.createWriteStream(filePath)
  inputStream
    .pipe(fileWriteStream)
}

Assuming the arguments were valid, this would get the job done, but this function would start the streaming process and then return immediately with undefined. Not very helpful, as we probably would want to know when the stream had finished (or if something went wrong), so we could react accordingly.

Luckily, streams are all event emitters, so we can register listener callbacks on various events that the stream emits. Stream methods like .pipe and .on generally return the stream itself, so we can succinctly chain these registrations on the back of the .pipe call:

const reactToStreamEvents = error => {
  if (error) {
    console.log('Uh oh!')
  } else {
    console.log('All done!')
  }
}

const streamToFile = (inputStream, filePath, callback) => {
  const fileWriteStream = fs.createWriteStream(filePath)
  inputStream
    .pipe(fileWriteStream)
    .on('finish', callback)
    .on('error', callback)
}

// Some later usage...

streamToFile(
  someYugeReadableStream,
  '/path/to/bigFile.txt',
  reactToStreamEvents
)

This works, but promises provide a nicer way to accomplish the same thing, by wrapping the stream code:

const streamToFile = (inputStream, filePath) => {
  return new Promise((resolve, reject) => {
    const fileWriteStream = fs.createWriteStream(filePath)
    inputStream
      .pipe(fileWriteStream)
      .on('finish', resolve)
      .on('error', reject)
  })
}

Now streamToFile immediately returns a promise object that we can pass around, await in some other async function, or chain .then and .catch handlers to. It abstracts away the core of what we were doing with the reactToStreamEvents callback: signaling for success and failure, while offloading the responsibility of reacting to whatever code might be consuming the promise, giving us more flexibility. Neat.

While I think promises are pretty straightforward once you use them for a while, the async/await sugar syntax makes them really simple to reason about. If I can use either (which is almost always) I generally use async/await. In this case though, we cannot use async/await inside streamToFile, because the event emitter interface expects you to pass a function (a callback!) as the second argument to the .on(event, callback) method signature.

The un-sugary (but still quite sexy, imho) Promise executor function gives us just the callbacks we need: resolve and reject. As such, plain jane promises and streams play quite nicely together.

Conclusion

In this post, we covered how promises can aid your use of node's powerful concept of streams. In the next post, I'll cover the peculiarities of testing the streamToFile function.