DEV Community: Karey Higuera

Finding the coordinates of an element in a 2D array

Karey Higuera — Wed, 25 May 2022 12:59:42 +0000

While working on a coding challenge that @cassidoo sent out in her latest newsletter, I needed to find the coordinates of a particular number in a two-dimensional array of numbers. Here's a quick method for doing so!

Finding the coordinates using loops

Let's assume that we have a 3x3 array of numbers, and we want to find the location of the number '5'. I've arbitrarily assigned the axes x and y here.

const grid = [
---> y
|  [8,6,7],
|  [5,3,0],
v  [9,1,2]
x
]

While it may be a simple approach, looping through our array is a straightforward way to find our target element. Check out the function below:

const find2dCoordinates = <T>(
  grid: T[][],
  target: T
): {x: number, y: number} => {
  let coords = {x: -1, y: -1}
  grid.every((row, i) => row.every((item, j) => {
    if (item === target) {
      coords = {x: i, y: j}
      return false
    }
    return true
  }))
  return coords
}

We first define a set of coordinates: let coords = {x: -1, y: -1}; since we know we'll never have negative coordinates with an array, we can know that we ran into an error if the returned object has negative values.

Next, we're going to loop through each item in each row. Since we want the indices as well as the items themselves, we use the .every() function instead of for...of loops. We then check if each item is equal to our target. If you had an array of objects instead of numbers, you'd want to use some other equality check here.

Now, when we find a match, we store the coordinates in our coords variable. But we have a slight problem! Since we're not using for...of loops, we can't use a break to jump out of our loops early. 😕 Have no fear, because we planned for this 😎 By using .every() instead of .forEach(), we have an escape hatch.

The .every() method accepts a function as an argument and will run that function on each item in its array. This function should return true or false. If the every() function receives true back for every value in the array, it returns true. However if it finds an item that fails, it returns false.

In our case, we return true ourselves for each item that doesn't match our target so that every() will keep looking. Once we find our target, we return false, which causes every() to jump out early, and which bubbles up to the other .every() function as well. Success!

What about a flattened array?

Sometimes, we may have a two-dimensional array that has been flattened into one dimension. If you've ever worked with PNG images before, this is how their pixel information is stored. So our grid from earlier would be [8,6,7,5,3,0,9,1,2]. How can we find our coordinates now?

First, let's find the index of the number we're looking for. Then, if we know the original length of each row, we can calculate the coordinates it would have had!

export const find2dCoordinatesFlattened = <T>(
  grid: T[],
  target: T,
  rowLength: number
): {x: number, y: number} => {
  const position = grid.findIndex(item => item === target)
  return {
    x: Math.floor(position / rowLength),
    y: position % rowLength
  }
}

We first use the .findIndex() function to find the index of our target item in the one dimensional array. Then, we use two tricks to find the x and y coordinates. By dividing the index we found by the original row length and rounding down, we can find which row the item belongs on. Then, if we use the modulo operator on the index with the row length, we can see which column it belongs to.

Final thoughts

This was a fun little project, and I recommend you take a shot at the puzzle from the newsletter I mentioned in the intro!

Also, you might have asked yourself if flattening the array ourselves and using the second method might be faster than the first method of loops. Check out these benchmarks results comparing both methods.

First, I ran it on a very small 2x3 array.

Pretty comparable scores! But then I ran it against a 50x10 array.

Yikes 😬 Flattening it ourselves falls way behind. This makes sense, though, as we have to manipulate the entire array first to flatten it, while the loops can simply dive right in and jump out early when they find the result.

Cracking a Captcha with Tesseract.js

Karey Higuera — Thu, 13 May 2021 22:02:39 +0000

I wanted to automate some tasks on a site that I frequently use, but every now and then the site presents a captcha. I figured that with a little cleanup of the captcha images, I'd be able to use OCR (Optical Character Recognition) to read the captchas and provide the correct answer.

The Problem with Captchas and OCR

The captcha images on the site consist of a string of five characters with various colored lines drawn through. For example, look at these three captchas.

The letters aren't distorted and they're on a nice white background, so we shouldn't have any problem dropping them straight into an OCR program, right? Well, not quite. The problem is that these little lines are very effective at thwarting OCR engines because the majority of those programs rely on edge detection to identify each letter. Tesseract.js is one of the most popular OCR libraries for JavaScript. Let's visit their site and try our captcha on their demo.

Instead of reading FCWVw, Tesseract produced ECYA w-; this is definitely not what we're looking for. We're going to have to clean up this image so that it doesn't confuse Tesseract.

Cleaning up the Captchas

We need to get rid of these lines if we want Tesseract.js to stand a fair chance at reading our image. Thankfully, there's an interesting attribute of these captchas that will help us: all of the characters are a solid color, and each line is a different color. These colors change every time a new captcha is produced.

So, if we break our png down pixel by pixel and count how many of each color pixel appears, we can find:

Which color appears most often (the white background)
Which color appears second most often (our characters)
The rest of the colors (the lines)

Identifying the characters

Let's pull in a png library that will help us analyze the image and then break down our color count.



const fs = require('fs')
const PNG = require('pngjs').PNG

// open image
const captcha = fs.readFileSync(path)
const pngImage = PNG.sync.read(captcha)
let {data, height, width} = pngImage

// create a dictionary to keep track of our pixel counts
let colorOccurrences = {}

for (let y = 0; y < height; y++) {  // rows
  for (let x = 0; x < width; x++) { // columns
    /**
     * Each pixel is a set of 4 values:
     * Red, Green, Blue, Alpha (transparency)
     */
    let index = (width * y + x) * 4;

    // create a string of the R-G-B color values
    let color = `${data[index]}-${data[index+1]}-${data[index+2]}`
    // we can ignore white since it will always be the background
    if(color !== "255-255-255"){
      // increase the count by 1 (or set it to 1 if the color wasn't there yet)
      colorOccurrences[color] = (colorOccurrences[color] || 0) + 1
    }
  }
}

We first open our image as a file and pass it in to the PNG library, which gives us a pngImage object. This object holds all the information about our image, but we're only concerned with 3 parts: the height, the width, and the data.

The data is a 1-dimensional array of all the color information that makes up our image. Each pixel is represented by 4 values from 0-255: Red, Green, Blue, and Alpha (the transparency). So for each round of the loop, we need to multiply our index by 4 to jump to the next pixel.

For example, let's look at a 2x2 (4 pixel) image.

The data array for this image would be:



/* R     G     B     A */
[
  252,  215,  64,   255, // pixel 1 (0,0)
  64,   252,  89,   255, // pixel 2 (0,1)
  0,    180,  239,  255, // pixel 3 (1,0)
  126,  64,   252,  255  // pixel 4 (1,1)
]

Now that we have our data, we can loop through each pixel and grab the RGB values (we don't need the alpha value). We'll store the RGB value as a string like R-G-B to use it as a key in our colorOccurrences object, then keep a count of how may pixels of each color occur. We'll just ignore the white pixels since it's the background color.

Finally, we can find our color that appeared most frequently. This will correspond to the color of our characters.



// grab all of the colors in the pattern [R-G-B, # of occurrences]
let colors = Object.entries(colorOccurrences)
// find the color that occurred most
let highestColor = colors.reduce((highColor, currentColor) => {
  if(highColor[1] > currentColor[1]) {
    return highColor
  } else {
    return currentColor
  }
})
// grab just the R-G-B as an array, we don't need the number of occurrences
let highestColorRGB = highestColor[0].split('-')

Now that we know what color our characters are, we can preserve those while removing the unnecessary noise from the picture.

Removing the lines

Since we know which colors we want to keep (the white background and whatever color our characters are), we can loop back through our image and repaint any pixel we see that doesn't match.



for (let y = 0; y < height; y++) {      // rows
  for (let x = 0; x < width; x++) {   // columns
    let index = (width * y + x) * 4;

    // grab the RGB values of the current pixel
    let RGB = [data[index], data[index+1], data[index+2]]

    // ignore white pixels so we don't alter the background
    if (RGB[0] === 255 && RGB[1] === 255 && RGB[2] === 255) continue

  /**
   * We need to be a little forgiving when checking the colors.
   * Sometimes individual pixels are only 1-3 points of R, G, or B away,
   * especially on the edges of the characters.
   */
  // find how far each pixel color channel is from the color of the characters
    let [red, green, blue] = [
      Math.abs(highestColorRGB[0] - RGB[0]),
      Math.abs(highestColorRGB[1] - RGB[1]),
      Math.abs(highestColorRGB[2] - RGB[2])
    ]

    // if any color channel is more than 3 points away
    if (red > 3 || green > 3 || blue > 3){
      // paint the pixel...
    }
  }
}

We set up a loop again, row by row and column by column, and grab the RGB values of the current pixel. We'll ignore any white pixel because we want to leave the background alone. Then, we check to see if the current pixel's color matches the color of the characters.

We have to leave a little leeway for each color check; sometimes there's a discrepancy of 1-3 points on each color channel between adjacent pixels, especially around the edges of the characters. So the main color might be 10-30-59, but one of the pixels on the character might be 11-29-57. We'll let these close pixels slide by unscathed. If we see that any color channel is more than 3 points off, we'll paint the pixel to get rid of the line. But what color do we paint the pixel?

The first option that comes to mind is to just erase the lines by painting each pixel white like the background.



// if any color channel is more than 3 points away
if (red > 3 || green > 3 || blue > 3){
  data[index] = 255;
  data[index + 1] = 255;
  data[index + 2] = 255;
}

This will produce the following image.

Not too bad! Let's see what happens when we pass this to Tesseract.

Uh oh. 😕 This time, Tesseract read VAN FCW\Vw. While the captcha looks a lot cleaner to us, we've actually created a lot of new edges which confuses Tesseract. Let's take a different approach.

Instead of painting the pixels white and leaving holes in our characters, we can try to fill in the gaps instead. The simplest thing we can do is just paint our current pixel the same color as the one above it.



// if any color channel is more than 3 points away
if (red > 3 || green > 3 || blue > 3){
  // Grab the pixel that is one row up (y-1)
  let aboveIndex = (width*(y-1) + x) * 4

  // Paint our pixel to match the pixel above
  data[index] = data[aboveIndex];
  data[index + 1] = data[aboveIndex + 1];
  data[index + 2] = data[aboveIndex + 2];
}

With this new technique, we get:

We still get some odd drips and artifacts, but this looks a lot better! Let's see what Tesseract thinks about our new version.

A perfect success! By eliminating extra edges and solidifying our characters, we've cleaned up the captcha enough to give Tesseract a fighting chance.

Future Improvements

I had a little over 800 of the captchas saved, so I decided to run both the white-line method and the full-character method against all of them to see the difference in their effectiveness.

The full-character method had a 69% success rate, compared to a 26% success rate for the white-line method. Based on the data I gathered from these tests, it's clear that Tesseract.js makes some common misreads:

8 as g
9 as o
Q as O
J as ]
J as 1
q as g

As a future improvement, we could provide our own training data to Tesseract.js based on the specific font used in these captchas! That would alleviate the confusion that occurs between these similar characters.

Conclusion

By cleaning up the lines crossing through our captcha, we were able to use the Tesseract.js OCR library to decode the captchas and allow for a bit more automation. And who doesn't love some more automation in their life?

Here's the full code:



const fs = require('fs')
const PNG = require('pngjs').PNG

// open image
const captcha = fs.readFileSync(path)
const pngImage = PNG.sync.read(captcha)
let {data, height, width} = pngImage

// create a dictionary to keep track of our pixel counts
let colorOccurrences = {}

for (let y = 0; y < height; y++) {  // rows
  for (let x = 0; x < width; x++) { // columns
    /**
     * Each pixel is a set of 4 values:
     * Red, Green, Blue, Alpha (transparency)
     */
    let index = (width * y + x) * 4;

    // create a string of the R-G-B color values
    let color = `${data[index]}-${data[index+1]}-${data[index+2]}`
    // we can ignore white since it will always be the background
    if(color !== "255-255-255"){
      // increase the count by 1 (or set it to 1 if the color wasn't there yet)
      colorOccurrences[color] = (colorOccurrences[color] || 0) + 1
    }
  }
}

// grab all of the colors in the pattern [R-G-B, # of occurrences]
let colors = Object.entries(colorOccurrences)
// find the color that occurred most
let highestColor = colors.reduce((highColor, currentColor) => {
  if(highColor[1] > currentColor[1]) {
    return highColor
  } else {
    return currentColor
  }
})
// grab just the R-G-B as an array, we don't need the number of occurrences
let highestColorRGB = highestColor[0].split('-')

for (let y = 0; y < height; y++) {      // rows
  for (let x = 0; x < width; x++) {   // columns
    let index = (width * y + x) * 4;

    // grab the RGB values of the current pixel
    let RGB = [data[index], data[index+1], data[index+2]]

    // ignore white pixels so we don't alter the background
    if (RGB[0] === 255 && RGB[1] === 255 && RGB[2] === 255) continue

  /**
   * We need to be a little forgiving when checking the colors.
   * Sometimes individual pixels are only 1-3 points of R, G, or B away,
   * especially on the edges of the characters.
   */
  // find how far each pixel color channel is from the color of the characters
    let [red, green, blue] = [
      Math.abs(highestColorRGB[0] - RGB[0]),
      Math.abs(highestColorRGB[1] - RGB[1]),
      Math.abs(highestColorRGB[2] - RGB[2])
    ]

    // if any color channel is more than 3 points away
    if (red > 3 || green > 3 || blue > 3){
      // if any color channel is more than 3 points away
      if (red > 3 || green > 3 || blue > 3){
        // Grab the pixel that is one row up (y-1)
        let aboveIndex = (width*(y-1) + x) * 4

        // Paint our pixel to match the pixel above
        data[index] = data[aboveIndex];
        data[index + 1] = data[aboveIndex + 1];
        data[index + 2] = data[aboveIndex + 2];
      }
    }
  }
}

// save new image
const imageBuffer = PNG.sync.write(pngImage)
fs.writeFileSync(`${path.replace('.png', '')}-clean.png`, imageBuffer)

Creating a CircleCI Orb

Karey Higuera — Mon, 05 Oct 2020 13:47:46 +0000

A guide for creating a CircleCI orb with the newly released CircleCI Orb Development Kit. I'll share the basics of what I learned while creating my Rome orb for Hacktoberfest in a much simpler orb.

Introduction to Orbs

CircleCI is a continuous integration tool that allows you to run packages of code called orbs against your code every time a commit is made to your repository, or every time a pull request is created or merged. This allows you to perform automated testing or other tasks to help maintain the quality of your projects.

CircleCI has just released a new tool, the Orb Development Kit, to simplify and streamline the process of creating new orbs. To demonstrate how to use the kit, we'll be creating a simple orb for cowsay, a CLI tool that has a cow say a message in your console.


❯ cowsay CircleCI is great!
 ____________________
< CircleCI is great! >
 --------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Though this particular cowsay orb is silly and a bit simple, creating a new orb that positively impacts developers and helps solve a challenge is a great way that you can contribute to Hacktoberfest! See the CircleCI Hacktoberfest page for more details. Use this article as a guide and go out there and create something truly awesome.

Setup for Making an Orb

In order to use the new development kit, we need to set up the CircleCI CLI. This command line tool provides a lot of handy tools that we can use for interacting with CircleCI and our pipelines.

In order to be able to create an orb, you must first enable it in your Organization Settings. Navigate to the CircleCI Dashboard and on the menu on the left hand side of the screen, click Organization Settings.

Click on the Security tab and choose yes under Allow Uncertified Orbs. This will give you permission to publish orbs.

We will also need to create a GitHub repository for our new orb. You can head to the new repository page by clicking this fancy link: github.new. Go ahead and leave the repository empty (don't add a README, license, or .gitignore file). Once you create it, take note of the git URL as the CLI will ask for it soon!

Scaffolding Out the Orb

With the new orb development kit, much of the initial setup is taken care of for us as we only need to answer a few questions. Make sure that you are on a recent version of the CLI (at least v0.1.10211) to be able to use this new command. Run circleci version to see your current version, and circleci update to pull down a newer version if necessary.

To get started with the toolkit, run the command circleci orb init <path to new orb>. The path can be a new folder, and the tool will create it. For ours, let's create an orb named cowsay-orb, so our command will be circleci orb init cowsay-orb.

The toolkit will walk us through the steps to initialize and build out the orb. Follow the instructions as presented in the CLI; you will usually except the default options. See here a screenshot of my log:

Note that once you provide the GitHub link to the tool, it will create an initial commit for you locally and display the message An initial commit has been created - please run 'git push origin master' to publish your first commit!. Open a new terminal (don't cancel the toolkit process that's still running!), navigate to your new orb directory, and push the commit to GitHub.

cd cowsay-orb
git push origin master

Once you push the commit, return the toolkit and hit enter on Done. The toolkit will wrap up it's work and give you a URL to view your new orb's build pipeline! You can view your pipeline at https://circleci.com/gh/<namespace>/<orb-name>.

The pipeline is set up to require your manual action to actually publish versions of your orb. Head to the pipeline for your orb, and you'll see a purple message that says On Hold.

Click on the hold-for-dev-publish step that is next to the purple pause icon and click Approve to give the pipeline permission to publish the first test copy of your orb.

After a short time, the pipeline will finish running two jobs and your first development copy of the orb is now published!

Now, on to our actual development.

Orb Layout

Open up your new orb directory in your code editor and ensure that you are not on the main branch. The toolkit should place you on the branch alpha automatically. You will always want to do your development on a separate branch like alpha, then merge that branch into your main branch later.

Let's take a look at the folder structure that was created for us.

.
├── .circleci
│   ├── config.yml
│   └── README.md
├── .git
│   └── ...
├── src
│   ├── commands
│   │   ├── greet.yml
│   │   └── README.md
│   ├── examples
│   │   ├── example.yml
│   │   └── README.md
│   ├── executors
│   │   ├── default.yml
│   │   └── README.md
│   ├── jobs
│   │   ├── hello.yml
│   │   └── README.md
│   ├── scripts
│   │   ├── greet.sh
│   │   └── README.md
│   ├── tests
│   │   ├── greet.bats
│   │   └── README.md
│   ├── @orb.yml
│   └── README.md
├── .gitignore
├── CHANGELOG.md
├── LICENSE
└── README.md

The .circleci directory contains administrative resources for the orb itself. We'll look at it in a bit more detail later on.

The .git directory provides some templates for GitHub issues and pull requests which you can customize if you like, but we won't discuss it in this article.

The src directory contains all of the different subdirectories that we'll be working in for this project. An orb can be made up of commands, examples, jobs, scripts, executors, and tests.

Commands are small, reusable pieces of functionality.
Examples are sample scripts that you provide to the user showing how to use your orb.
Jobs are longer processes made up of steps that can combine multiple commands to complete a task.
Scripts are pieces of code in other languages that can be used by your commands to accomplish more than what you could easily express in yaml, the language that the orbs are written in.
Executors are custom definitions of what type of "machine" or image you want your orb to be run on.
Tests and test resources can be placed in the test folder and executed by your integration tests.

For this simple orb that we are creating, we will not be defining jobs, scripts, executors, or separate tests, so those folders can safely be deleted. If you want to configure any of those yourself, look at the README provided in each folder, the examples provided, or the pages I have linked in the list above.

Open up the @orb.yml file located in the src folder. This is the main starting point for your orb. Whenever you publish the orb, the toolkit automatically takes each of the .yml files in your project and packs them together into one file. In this main @orb.yml file, you can define details about your orb and import any other orbs you might need.

Let's look at our fields here:

version - the configuration file version that we are using. This will be 2.1, the latest.
description - should be a short and sweet snippet about what our orb will do.
home_url - a reference to the home page of the tool we're building an orb for.
display_url - a reference to our GitHub repository for this orb.
orbs - any other orbs that we'll pull in to our project.

Our @orb.yml will look like this:


version: 2.1

description: >
  A simple CircleCI Orb to bring some beef to your logs.

display:
  home_url: "https://en.wikipedia.org/wiki/Cowsay"
  source_url: "https://github.com/kbravh/cowsay-orb"

orbs:
  node: circleci/node@4.0.1

We're now ready to develop our orb functionality!

Create Our Orb's Command

Open up the commands directory. We can open the greet.yml to learn what a command should look like.

⚠️ Note: From here on out, we'll be writing YAML, which is indent dependent and is very picky (especially with the YAML linter that is built into the toolkit). If you run into errors, please check the following:

Ensure your indentations are correct.
There shouldn't be any whitespace at the end of lines. The Trailing Spaces VS Code extension is very helpful here.
There must be a blank line at the end of every YAML file.


description: >
  This command echos "Hello World" using file inclusion.
# What will this command do?
# Descriptions should be short, simple, and clear.
parameters:
  to:
    type: string
    default: "World"
    description: "Hello to whom?"
steps:
  - run:
      environment:
        PARAM_TO: <<parameters.to>>
      name: Hello Greeting
      command: <<include(scripts/greet.sh)>>

The description should succinctly describe what this command will do.
The parameters section defines arguments that we can pass to our orb to let the user customize what it will do.
The steps are the individual commands that will be run. Notice that any parameters that we define above can be referenced using double angle brackets << parameters.<name> >> and the parameter's name using dot notation.

Let's change the name of this file to speak.yml. The names of the files in the command folder will become the actual commands available for our orb. We can now build out our own functionality. We'll inform the user that this speak command will have a cow say a command in the logs.


description: >
  This command will have a cow say a message in your logs.

Next, let's provide a parameter for the message the cow will say. The name of the parameter is the first line written under parameters. We can define a type (string, boolean, enum) and also define a default value in case the user does not pass one. If defaults are not provided, the job will fail for any missing parameters.


parameters:
  message:
    type: string
    default: "Mooo!"
    description: "What will the cow say?"

Finally, let's the define the steps of this command itself. The run operator will let us run a command as if we were typing in the terminal. We can provide a name for the command, and then pass the command itself. We'll need to run the command npx cowsay to fetch the cowsay package, and then we pass in the message from our parameters. By passing the -q flag to npx, we can have it be quiet and not print any of its own output, like download times.


steps:
  - run:
      name: Let the cow speak
      command: npx -q cowsay << parameters.message >>

That's all there is to defining a small command! So our speak.yml file will come together to look like this:


description: >
  This command will have a cow say a message in your logs.
parameters:
  message:
    type: string
    default: "Mooo!"
    description: "What will the cow say?"
steps:
  - run:
      name: Let the cow speak
      command: npx -q cowsay << parameters.message >>

Define an Integration test

Now that we have our command written, let's create an integration test. This will serve to make sure our orb command runs correctly, and we can also use this as the basis for our example.

Navigate to the .circleci folder and open config.yml. The toolkit uses this configuration to publish development and production versions of our orb. We can clean up this file a bit to accommodate our orb. Under the orbs heading, let's add the node orb.

orbs:
  node: circleci/node@4.0.1

As we haven't included any Bash scripts, we can remove the bats and shellcheck orbs. This means we should also head down to the workflows section and remove the shellcheck/check and the bats/run commands. Since we no longer have these jobs, we should also remove them from the requires list under the hold-for-dev-publish job.

Now, let's define our integration test. Under the jobs heading (which should be around line 22), remove everything beneath the integration-test-1 command. We'll add our own configuration here. We need to tell the job what environment to execute the job in, so set the executor to node/default, an executor defined in the node orb that comes with Node.js installed. We need Node.js to run the npx cowsay command. Then, define the steps we want the job to run. For us, this will just be our cowsay/speak command. Don't forget to pass in a message!


jobs:
  # Define one or more jobs which will utilize your orb's commands and parameters to validate your changes.
  integration-test-1:
    executor: node/default
    steps:
      - cowsay/speak:
          message: "Howdy!"

This integration test is very basic and just ensures that our command can be run without throwing any errors or crashing the pipeline. See the orb Testing Methodologies page for more information about testing.

⚠️ Note: Since the only published version of our orb that exists right now doesn't have our speak command that we created, this integration test will fail! To get around that for now, let's comment out the call to run integration-test-1 under the integration-test_deploy job, and comment out the two requires lines under orb-tools/dev-promote-prod-from-commit-subject. We can re-enable it once we successfully run the pipeline. Our code sections will now appear like this:

integration-test_deploy:
    when: << pipeline.parameters.run-integration-tests >>
    jobs:
      # Run any integration tests defined within the `jobs` key.
      # - integration-test-1

...

- orb-tools/dev-promote-prod-from-commit-subject:
      orb-name: kbravh/cowsay-orb
      context: orb-publishing
      add-pr-comment: false
      fail-if-semver-not-indicated: true
      publish-version-tag: false
      # requires:
      #   - integration-test-1
      filters:
        branches:
          only: master

Provide an Example

We can finally wrap up one of the last steps: writing an example for our users. This will resemble our integration test.

Open up the examples folder; you can delete the example.yml. Create a new file congratulate.yml, and let's start building out our example.

We can include a description just like in other files, so let's describe what will happen in this test.


description: >
  Congratulate the user on the pipeline passing.

Next comes the usage tag. This is where we'll actually build out everything that will appear as the example for the user. We'll include the same version number 2.1, and we'll list the orbs that we'll use. The development version of our orb is automatically tagged as dev:alpha by the toolkit, which is why we include it like that here. Once you have a published version of your orb, you can update this example with the actual version.


usage:
  version: 2.1
  orbs:
    cowsay: kbravh/cowsay@dev:alpha
    node: circleci/node@4.0.1

Finally, we'll build out the job we would need to run. We can call it congratulate. We define our executor just as we did in the integration test, and call our cowsay/speak command with a custom message.


  jobs:
    congratulate:
      executor: node/default
      steps:
        - cowsay/speak:
            message: "Your pipeline looks great!"

We're finally ready to try out our orb!

Testing and Publishing the Orb

Before we push our orb to the pipeline for testing, we can do a quick local validation using the circleci orb validate tool in the CLI. This command will verify that our orb is formatted correctly. However, since our orb is currently split into many different files, we'll need to pack it together before validating. Both of these steps can be done using these two commands in your terminal.


circleci orb pack src/ > orb.yml
circleci orb validate orb.yml

The pack command squashes our various YAML files down into one consolidated orb file. Normally this command would just print the result out to the terminal, so we instead direct it into a file orb.yml. Then, our validate command checks our formatting.

Having any issues with YAML formatting? Try dropping your YAML files into this YAML linter to catch syntax errors.

If you see the message Orb at 'orb.yml' is valid. then you are ready to publish!

Publishing a development version is rather straightforward: you just need to push your changes to GitHub on your alpha branch!

Once you push your changes, head over to the pipeline at https://circleci.com/gh/<namespace>/<orb-name>. You'll have to click the On Hold button again to pass the job through. Once the pipeline finishes, you'll have a new dev version with your commands!

You can now re-enable your integration tests that we disabled above and push the code to GitHub once more. This will now run your integration test and publish another dev version. You can open the pipeline details and see the result of your integration test once it passes.

Once the pipeline finishes, you are finally ready to publish a live, production version of your orb!

Publish a Production Orb

To start the process of upgrading your dev orb to a production version, you'll need to head over to GitHub and create a pull request from your alpha branch to your default branch. You have to include a special [semver:<increment>] tag in the title of your pull request. This is how the toolkit will know what version to assign to your orb when it is published. Check the following table to know what increment to include.

Increment	Description
major	Issue a 1.0.0 incremented release
minor	Issue a x.1.0 incremented release
patch	Issue a x.x.1 incremented release
skip	Do not issue a release

If we wanted our first version of our cowsay orb to be 0.0.1, we'd include [semver:patch] in the title. However, let's put [semver:major] so that our first version is 1.0.0.

After clicking Create pull request, you'll see a window below with a status. This window will show the status of the alpha pipeline, which should indicate that all the checks have passed. If everything is ready, you'll have the option to Merge pull request. It is recommended to click the small arrow next to the Merge button and choose to Squash and merge, as it will preserve your pull request title.

You'll be asked to confirm the merge. Make sure your semver tag is present still in the title! By merging in this pull request to your default branch, a final pipeline run will be started. Follow our same On Hold ritual from before and allow the pipeline to finish running.

Once the pipeline finishes, your orb should now be live! Visit it at https://circleci.com/developer/orbs/orb/<namespace>/<orb-name>. Note: If you try to search for your orb at the Orb Registry and it can't be found, try waiting a little while before searching again. It seems to take a bit for new orbs to be indexed by the search engine.

Congratulations on making an orb! Hopefully this guide has been helpful. Go out there and do something awesome!

Extra information

Want to make an orb private? Orbs are public by design, but you can unlist an orb so that it won't show up in the Registry. Learn how to Unlist an orb.

Avoid Losing Errors in a Bash Pipe

Karey Higuera — Sun, 04 Oct 2020 15:48:28 +0000

Have you ever wanted to string together some bash commands using pipes, but weren't sure how to not lose the error codes in the process?

tl;dr Add set -o pipefail to the beginning of your script!

For the CircleCI Rome Orb that I created for Hacktoberfest, I was seeing very odd output to the pipeline logs for the commands. This was due to the way that the Rome linter streams its output. Each message was appearing one letter per line instead of showing the full message.

In order to alleviate this, I wanted to pipe my linting command through to cat to be displayed all at once instead of being streamed in character by character.

npx rome check | cat

Are you curious as to why we can't use echo here? The echo command only takes input from the arguments passed to it directly, not from stdin. If we used echo here, the console would only output a blank line!

However, just doing this piping would cause a problem in our script execution; if linting failed, the error code would just disappear! The problem stems from the fact that Bash does not always act like higher level programming languages. If a command fails, Bash will continue execution like normal. For a linting command, we want to get back our exit code so we know if linting failed without losing it in the pipe.

Luckily, there's a handy option for this.

set -o pipefail

By running set -o pipefail in our script before our pipe command, bash will finish executing the entire pipe and throw the error code of the right-most error that occurs, if any. So if npx rome check fails, that error code will still be thrown even though cat succeeds.

If you want to get some more in depth information and see some other good options to set for your bash scripts, check out vaneyckt's great article.

Hero photo by Valentin Petkov https://unsplash.com/@thefreak1337

Versions and Aliases in AWS Lambda

Karey Higuera — Tue, 30 Jun 2020 01:53:40 +0000

Have you ever wanted to make a change to a Lambda function that's running in your live environment, but you've been worried that you'd make a mistake and have some down time? Or worse, have you made the change and brought down your site? 😬 AWS Lambda provides a way to create versions of a function in order to test functionality in lower environments before moving it into higher environments, all without the risk of affecting your deployed code.

Creating a version

First, create a new function or choose a function that you've already created. At the top of the page under the Actions dropdown, choose to publish a new version of your function.

You can add a description here to your version. This is essentially saving a snapshot of your function as it is now (like a git commit).

You'll notice that the window says it is creating a new version from $LATEST. This is the tag that AWS gives to your current working state. Any changes you make to your function are saved in the $LATEST version even if you don't publish an explicit version yourself.

Once we publish our new version, we'll be taken to this new copy of our lambda function. We can tell what version we're on by the number appended to the function name. We can return to our $LATEST function version by using the Version dropdown or the breadcrumb trail at the top left.

We can't make changes to or publish a new version of this snapshot, so let's return to our $LATEST version and make a change to our code! We'll publish it as version 2.

There we go! We now have multiple versions of our functions that we can use for testing in different environments. These different versions can be referenced by using the ARN or function name with the version number appended like so: AliasAndVersioning:2.

But what if we publish a third version and want a CloudWatch rule or an S3 trigger to point at our new version? With version numbers, we'd have to go and update the resource information with the new number each time. That's where aliases come in to save the day.

The Power of Aliases

Aliases are essentially nicknames or tags that we can point at certain versions of our function. When we create a new version of our function, we can simply move our alias to point there instead. Let's see it in action.

Creating a Production alias

Under the Actions dropdown, choose to create an alias. Here we can set the name, a description, and which version we'll point the alias at. Let's set PROD (Production) at version 1 that we created.

The weighted alias section at the bottom allows you to divert a percentage of your traffic to different versions of your function, like for A/B testing.

When we save this alias, we'll see a screen very similar to what we saw when we created a new version, but the alias name is now appended to the function name instead of the version number!

Creating a Development alias

Let's now create a DEV (development) alias. Instead of pointing this at a version number, we can point it directly at $LATEST to always reflect our latest changes.

Now we have our aliases for our different development environments! So when I reference this function while setting up any other resources, I can reference the alias of the function like AliasAndVersioning:DEV.

Update alias reference

Let's say that we've finished testing version 2 of our function in the development environment and we're ready to use it in Production. If we have our references set up to point to the alias, it's as easy as just adjusting the alias definition.

Let's choose our PROD alias from the Alias dropdown so that we can edit its settings. In the Alias configuration box at the bottom, click the Edit button.

From the version dropdown on the page, we can now choose version 2 and save our changes.

Production will now be using our newer version! 🎉

Final Thoughts

Congratulations! You'll now be a pro working with versions and aliases in AWS Lambda. Your work mates will be in awe of your awesome devops skills, and your clients will be well pleased knowing their sites will be more resilient and much less likely to have down time.

Automatically Adjust AWS Cron Events for Daylight Savings Time

Karey Higuera — Thu, 11 Jun 2020 16:14:09 +0000

In order to lower costs for unused resources in AWS (which add up way faster than you think), we spin-down servers in our non-production environments every night during non-working hours and bring them back up in the morning. This entire process is handled by a single lambda function that is kicked off by some cron triggers in CloudWatch Event rules.

We have the environments set to sleep every weekday between 7pm and 7am, so they'll be back up in time for the first developers who get in at 7:30am. They also sleep all weekend since we don't do any development on Saturday or Sunday. Things were working flawlessly! That is, until Daylight Savings Time hit.

The Problem

For those who may not be familiar with Daylight Savings Time (DST), it's the practice of setting the clocks forward one hour in spring and back one hour in fall to better take advantage of the daylight hours. Unfortunately, since the cron triggers in AWS are set up on UTC time, the DST time change caused our spin-up function to run an hour late.

/* Original CRON triggers when not in DST */
0 13 ? * MON-FRI * // spin-up at 13:00 UTC (7am CST)
0 1 ? * TUE-SAT * // spin-down at 1:00 UTC (7pm CST)

Curious about how cron events work? Cron expressions in AWS are made up of 6 slots. From left to right:

Minutes (0-59)

Hours (0-23)

Day of the Month (0-31)

Month (1-12 or JAN-DEC)

Day of the Week (1-7 or SUN-SAT)

Year (1970-2199)

An asterisk can stand in as a wildcard in any slot.

I adjusted our cron events to account for the time change as follows:

/* New CRON triggers during DST */
0 12 ? * MON-FRI * // spin-up at 12:00 GMT (7am CST)
0 0 ? * TUE-SAT * // spin-down at 0:00 GMT (7pm CST)

This would work, but it was a short term fix. I wanted a longer term solution so we wouldn't run into this problem again in November when the time changed back.

The Plan

It occurred to me that I could have the very same spin-up/spin-down function adjust its own cron triggers based on whether or not we were in Daylight Savings Time. To do this, I would just need to add a little bit of time logic.

First, I needed to get the current time local time and determine whether or not it was Daylight Savings Time. Thankfully, Python has some very useful date functionality that would make this easy to do.

from datetime import datetime
import pytz

utc_time = datetime.utcnow().replace(tzinfo=pytz.utc)
cst_timezone = pytz.timezone('US/Central')
cst_time = utc_time.astimezone(cst_timezone)

We import the datetime library, the main Python library for working with any date or time objects, and pytz, a library for accurately handling timezones. We create utc_time and set it equal to the current time in UTC with datetime.utcnow(). Note that we have to manually replace the timezone information on the native datetime object with the utc timezone from pytz.

Next, we create a central time timezone object cst_timezone and pull that information from the pytz library. Finally, we convert our UTC time to CST time using the function astimezone(), which takes in a timezone object as a parameter.

So now that we have our current time in the CST timezone, we can determine if it's Daylight Savings Time.

# This will show the offset in seconds for DST ⏰
# If it's zero, we're not in DST
is_dst = cst_time.tzinfo._dst.seconds != 0

If we dig down into our current time object cst_time, we can get more information about the timezone. By accessing our timezone information (tzinfo), then our DST information (_dst), we can find how many seconds of an offset is currently on our time. If we are in DST, the offset will not be 0 (it will be 3600 seconds, really, since time gets shifted an hour).

The Triggers

Now that we know if it's Daylight Savings Time or not, we can adjust our cron triggers. First, let's define what our expressions will be based on if we're in DST. During DST, we'll want the earlier triggers.

if is_dst:
  start_schedule = 'cron(0 12 ? * * *)'
  stop_schedule = 'cron(0 0 ? * * *)'
else:
  start_schedule = 'cron(0 13 ? * * *)'
  stop_schedule = 'cron(0 1 ? * * *)'
# Curious why these cron expressions don't have the weekdays anymore? Read on!

Next, we need to actually update the rules in CloudWatch.

  import boto3
  client = boto3.client('events')

  START_RULE = "start_rule"
  STOP_RULE = "stop_rule"

  start_settings = client.list_targets_by_rule(Rule=START_RULE)
  stop_settings = client.list_targets_by_rule(Rule=STOP_RULE)

We create our boto3 client to be able to access our AWS resources. Assuming our rules are named "start_rule" and "stop_rule", we can then fetch information about these rules from CloudWatch. Here, we are using the function list_targets_by_rule() to get back the targets these rules have (that would be this function).

# Remove targets from current rules
client.remove_targets(
  Rule=START_RULE,
  Ids=[(start_settings['Targets'][0]['Id'])]
)
client.remove_targets(
  Rule=STOP_RULE,
  Ids=[(stop_settings['Targets'][0]['Id'])]
)
# Delete rules
client.delete_rule(Name=START_RULE)
client.delete_rule(Name=STOP_RULE)

Technically here we aren't updating the cron triggers, but rather replacing them. In order to delete a cron event trigger in AWS, you must first remove its targets by passing in an array of IDs. We'll give it the information we got from our previous call. We only have one target, so we pull the first (and only) item from our settings that we queried like start_settings['Targets'][0]['Id'] . Once we delete our targets, we can delete our rules.

# Add new rules
client.put_rule(
  Name=START_RULE,
  ScheduleExpression=start_schedule,
  State='ENABLED',
  Description="Automatic trigger for the Spinup-Spindown function."
)
client.put_rule(
  Name=STOP_RULE,
  ScheduleExpression=stop_schedule,
  State='ENABLED',
  Description="Automatic trigger for the Spinup-Spindown function"
)

Now, we can add our new cron rules in. We'll set the same name as before and set our new cron expressions that we defined as start_schedule and stop_schedule. We enable the rule, and add our description.

# Add targets
client.put_targets(
  Rule=START_RULE,
  Targets=[
    {
      'Id': start_settings['Targets'][0]['Id'],
      'Arn':start_settings['Targets'][0]['Arn'],
      'Input': start_settings['Targets'][0]['Input']
    }
  ]
)
client.put_targets(
  Rule=STOP_RULE,
  Targets=[
    {
      'Id': stop_settings['Targets'][0]['Id'],
      'Arn':stop_settings['Targets'][0]['Arn'],
      'Input': stop_settings['Targets'][0]['Input']
    }
  ]
)

Finally, we add our targets back in. We can do this by pulling all the information out of our settings objects. Our updated cron rules are in place!

Final Adjustments

So now our function can update its own cron triggers, awesome! But wait; what happens on the days that the time change actually happens? Won't our function run an hour late or an hour early on that one particular day? It will indeed, but we come out lucky here.

If you remember from the cron expressions we had at the beginning, we only ran this function Monday through Friday because we don't want our environments up on the weekend. In the United States, DST begins the 2nd Sunday in March and ends on the 1st Sunday in November. So our 1 hour shift would happen on a weekend, and it would never affect us!

Like I said though, our orignal cron expressions don't run on weekends, so we'll need to change our cron expressions slightly so they will. If you notice from the code snippet earlier, we already did this:

if is_dst:
  start_schedule = 'cron(0 12 ? * * *)'
  stop_schedule = 'cron(0 0 ? * * *)'
else:
  start_schedule = 'cron(0 13 ? * * *)'
  stop_schedule = 'cron(0 1 ? * * *)'

These cron expressions will now run every day of the week, instead of just Monday through Friday.

Now that our function runs every day, we'll need to stop it from doing the spin-up and spin-down on weekends with a bit more time logic.

weekday = cst_time.weekday()
#in the python calendar, 5 and 6 correspond to Saturday and Sunday
if weekday == 5 or weekday == 6:
  return {
      'statusCode': 200,
      'message': 'Today is a not a weekday, so the spin-up/spin-down process will not proceed.'
  }
else:
  begin_startstop_process()

We take our cst_time object and pull the day of the week out using weekday(). We then check whether it's a Saturday or a Sunday, and if so, we return a statusCode and message and stop execution. Otherwise we can get things going!

Final Thoughts

So there we go, we can now automatically adjust our cron triggers in AWS so that Daylight Savings Time won't affect us again! However, there are definitely some future improvements we could make. For example, you might ask the following questions:

Why do we adjust the rules every single day even when the rules will only change twice a year?

What if we have a function that needs to run on weekends that we don't want to be late twice a year?

This could be resolved pretty simply by extracting all of this functionality into a separate function. Using cron expressions, we could also set this function to only run on Sundays in March and November when the time changes would happen! Let's wrap this all up:

/* CRON triggers for DST function */
0 12 8-14 MAR SUN * // 13:00 UTC (6am CST) for the second Sunday in March
0 0 1-7 NOV SUN * // 1:00 UTC (6pm CST) for the first Sunday in November

Thanks for sticking around and reading. Hopefully this will help you out with your work in AWS!

Want to dig a bit deeper and learn a bit more? Here are some useful links:
Daylight Savings Time in the US | Wikipedia
Schedule Expressions for Rules | Amazon Web Services
Quick and Simple Cron Expression Editor | crontab guru

Error Handling in API Gateway with AWS Lambda and Node.js

Karey Higuera — Sat, 15 Feb 2020 16:40:55 +0000

For a recent project, I set up a lambda function that would hydrate a template using data posted to it through API Gateway. The template name would be specified in the request, which means I needed to send back some errors if anything went wrong, such as the template not being found.

These errors are handled differently in async and non-async functions. We will go over those in detail below, then look at how to catch those errors in the Gateway and return the correct API response.

Setup

We can prepare two test events in the Lambda console to trigger errors and successful runs. Create a new lambda function using Node.js 12. You can leave the default execution role. Once the function is created, you will see options at the top right of the page. Choose to configure test events.

Create two test events using the code snippets below.

Success

{
  "trigger": true
}

Failure

{
  "trigger": false
}

We can now manually trigger successes and failures very easily using our test events. Let's set up our functions.

Async Lambda Function Errors

Async functions allow for errors and results to be returned using throw and return, respectively. The following sample function demonstrates this functionality.

exports.handler = async event => {
  let result = await new Promise((resolve, reject) => {
    if (event.trigger) {
      resolve(`Success!`)
    } else {
      throw `Failure!`
      // reject(`Failure!`) This will also bubble up as an error!
    }
  })
  return result
}

Alternatively, you can return a Promise from an async function, and the lambda function will automatically handle resolve and reject for responses and errors.

exports.handler = async event => {
  return new Promise((resolve, reject) => {
    if (event.trigger) {
      resolve(`Success!`)
    } else {
      reject(`Failure!`)
    }
  })
}

Non-Async Lambda Function Errors

Handling responses and errors in non-async functions comes down to context and callback, two parameters passed in to the handler functions. The original way to return a response or error was using context.succeed() and context.fail(). This, although still functional, was superseded by callback() once AWS added support for Node 4.3.2. These are implemented as follows:

Context Method

exports.handler = (event, context) => {
  if (event.trigger) {
    /* outdated but included for compatibility */
    context.succeed(`Success!`)
  } else {
    /* outdated but included for compatibility */
    context.fail(`Failure!`)
  }
}

Callback Method

exports.handler = (event, context, callback) => {
  if (event.trigger) {
    /* the first parameter is for errors, so we pass null for a success */
    callback(null, `Success!`)
  } else {
    callback(`Failure!`)
  }
}

API Gateway Error Mapping

Now that we have our errors propagating correctly from our Lambda functions, we can catch those errors and map them to HTTP responses. First, create a new REST API. We can leave default settings and add a name and description. Next, we'll add a POST method to it by choosing Add Method from the Actions menu, then choosing POST.

Once we click the check mark, we'll see a setup screen. We can choose any of our Lambda functions for the integration (leave Lambda proxy integration unchecked!).

Once the method is created, we'll see a screen that will show us the flow of our method.

First, we'll choose the bottom-left box that says Method Response. Here we can add a response (an HTTP response code) that our method will be able to send back. Add 400 as a response and click the check box to save changes.

Click Method Execution at the top of the screen to return to the previous overview.

We'll now be able to add the actual mapping of the error we throw in the lambda function to the 400 HTTP code we added! Click on the bottom-right box, the Integration Response. When we add an integration response here, we'll be presented with three boxes.

The first box will be a regular expression to match the error code we throw back. Let's set our regular expression to check for our error "Failure!". We'll need to add a wildcard in front as the error will be inside of a JSON object, and will thus have other characters in front of it. .*Failure!.*

API Gateway won't run these checks against successful responses! If your lambda doesn't throw an error, it will pass through as a 200 response. You can gain a bit more control over this with Lambda Proxy Integration.

We can now choose the 400 error code from the dropdown and save our changes.

Test It Out

That's it for the setup! We can now test out our error mapping. On the overview screen, we see the leftmost box marked Test.

On the test page, we are given a box to provide a test body. We can use the same bodies that we used for the lambda function test cases. Add the failure request body and click Test. Your result will pop up on the right side of the screen.

Congrats! You've successfully set up error mapping in API Gateway.

Extra: Custom Error Objects

What if we want to send back something more than just a string as an error? We might want to send back more information such as a stack trace, a list of multiple errors, or a JSON object. Luckily we can do this without too much more effort!

First, let's edit our Lambda function to send a JSON object instead of just our Failure! string by itself (make sure to still return this string somewhere in your response as that's how we're catching the error with our regular expression).

exports.handler = (event, context, callback) => {
  if (event.trigger) {
    /* the first parameter is for errors, so we pass null for a success */
    callback(null, `Success!`)
  } else {
    // callback(`Failure!`)

    // We have to stringify our object or we'll just get back "[object Object]"
    callback(JSON.stringify({
      error: `Failure!`,
      detail: `The request has failed due to the trigger passed in`
    }))
  }
}

If we test this, we see our error coming through! However, it would nice if we could send this back in JSON format instead of leaving it as a string. This is very easy to do with a mapping template.

Let's head back to the integration response page where we set up our regular expression. If we open up our failure mapping, we can see that there is a mapping templates section at the bottom of the page. We'll choose to add a template; type in the default MIME type already shown (application/json) and click the checkbox.

Next, we'll add the actual mapping template. There's a ton that you can do with these mapping templates (see the link at the end of the article), but we'll keep it extremely simple in this article. Add the following template:

$input.path('$.errorMessage')

This tells API Gateway to look at the errorMessage key on the incoming input (where our stringified JSON object is) and pull that object out. Save your changes, and you can test again.

Congratulations, you've got it! You can now pass your custom Lambda errors through API Gateway to the client!

Want to dig a bit deeper and learn a bit more? Here are some useful links:

Handle Lambda Errors in API Gateway | Amazon Web Services

AWS Lambda Function Handler in Node.js| Amazon Web Services

Create Models and Mapping Templates for Request and Response Mappings | Amazon Web Services

Node.js 4.3.2 Runtime Now Available on Lambda | Amazon Web Services