DEV Community: SerDigital64

Terraform Scripting Concepts: Part 2 of 2

SerDigital64 — Fri, 31 Dec 2021 01:25:02 +0000

Overview

This is the second part of the tutorial. If not done already, please read the first part: Terraform Scripting Concepts: Part 1 of 2

Using functions

Another particular characteristic of the Terraform Language is that it doesn't support user-defined functions. Instead, it provides groups of built-in functions that can be used when assigning value to variables and attributes:

Numeric
String
Collection
Encoding
Filesystem
Date and Time
Hash and Crypto
Networking
Type conversion

Use the following template for calling functions: FUNCTION_NAME(PARAMETER1, PARAMETER2, PARAMETERN)

For example:

locals {
  # Find the maximum value from a list of numbers
  numeric_max = max(2, 3, 10, 1, 20)
  # Find the minimum value from a list of numbers
  numeric_min = min(2, 3, 10, 1, 20)
  # Conver string to lowercase
  string_lower = lower("ServerX.DomainY")
  # Extrat dir from path
  filesystem_dir = dirname("/full/path/to/file")
  # Extrat file name from path
  filesystem_file = basename("/full/path/to/file")
  # Record timestamp
  date_time_timestamp = timestamp()
}

Managing infrastructure

Up until now, the script takes no action. In order for Terraform to actually manage the target infrastructure and make changes to reach the required end-state, the script needs providers.

Terraform defines the following block types for working with the target infrastructure:

terraform: used to set workspace-wide run-time options.
required_providers: used to declare what providers will be used
provider: used to define provider-specific options.
resource: used to create a logical representation of the managed resource. Each provider will further specialize the resource with attributes and subtypes.
data: optional block that is used to query infrastructure resources managed by the provider. The query result is then made available for other blocks to use as variables.

Declare Providers

Providers are declared by adding a required_providers nested block to the terraform block. Each provider will be assigned to an attribute of type map:

Map name: used to reference the provider from other blocks.
Attribute version: used to define version requirements.
Attribute source: defines from where the provider will be downloaded.

Additional provider-specific attributes can be defined by using the provider block. The block will use the same label as the map name in required_providers.

The following example declares two providers: aws and azure:

terraform {
  required_providers {
    aws = {
      version = ">= 3.0.0"
      source = "hashicorp/aws"
    }
    azurerm = {
      version = ">= 2.0.0"
      source = "hashicorp/azurerm"
    }
  }
}
provider "aws" {
  region = "us-east-1"
}
provider "azurerm" {
  features {}
}

Define Managed Resources

Once the provider has been declared, it will make available additional resource and data types. For example:

Provider: AWS
- Resource Type: aws_instance
- Data Type: aws_ami
Provider: Azure
- Resource Type: azurerm_linux_virtual_machine
- Data Type: azurerm_image

Use the following template to define blocks:

data "RESOURCE_TYPE" "RESOURCE_LABEL" {
  ATTRIBUTE1 = VALUE
  ATTRIBUTEN = VALUE
}

resource "RESOURCE_TYPE" "RESOURCE_LABEL" {
  ATTRIBUTE1 = VALUE
  ATTRIBUTEN = VALUE
}

The data and resource blocks can be used together to improve automation. In the following example, the data block is used to query AWS for a particular AMI and then the resource block will use the resulting AMI ID to create the VM instance:

data "aws_ami" "coreos" {
  most_recent = true    # Return the latest one if more than one result is available

  filter {              # Look for fedora coreos v35
    name   = "name"
    values = ["fedora-coreos-35**"]
  }

  filter {              # Look for HVM virtualization
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["125523088429"]     # CentOS / Fedora account that owns AMIs
}

resource "aws_instance" "aws_test_instance" {
  ami = data.aws_ami.coreos.id    # Use the AMI that was found in the data query
  instance_type = "t2.micro"      # Select the instance size
}

Next Steps

Explore Terraform resources:

Source code: GitHUB Repository
Tool documentation: Manual

Learn advanced features:

Understand provider-specific implementation details:

Providers

Copyright information

This article is licensed under a Creative Commons Attribution 4.0 International License. For copyright information on the product or products mentioned inhere refer to their respective owner.

Disclaimer

Opinions presented in this article are personal and belong solely to me, and do not represent people or organizations associated with me in a professional or personal way. All the information on this site is provided "as is" with no guarantee of completeness, accuracy or the results obtained from the use of this information.

Terraform Scripting Concepts: Part 1 of 2

SerDigital64 — Fri, 31 Dec 2021 01:23:54 +0000

Overview

Terraform is an automation tool focused on infrastructure provisioning. The product is developed by HashiCorp, and the command line version is provided as an Open Source Project available on GitHub.

Terraform has the following core components:

Terraform CLI: stand-alone app that provides the run-time environment for Terraform Scripts and maintains build-state information of Managed Resources.
Terraform Script: text file with Terraform Language code used to describe the required end-state of the Managed Resources.
Managed Resource: logical representation of a real infrastructure resource in the target infrastructure.
Terraform Provider: stand-alone apps (plugins) that are used by the Terraform CLI to interact with Managed Resources.
Terraform Workspace: working directory where Terraform Scripts are located and where the Terraform CLI stores data files (state, temporary, plugins, etc.)

Using Terraform

CLI Installation

The recommended method for production environments is to add HashiCorp's package repository to the OS package manager and install it using native tools.

For more relaxed environments the binary package provides a simpler alternative that does not require root privilege:

# Create the installation repository
INSTALL_PATH="${HOME}/terraform-cli"; mkdir "${INSTALL_PATH}"; cd "${INSTALL_PATH}"
# Download the binary package
curl -LO https://releases.hashicorp.com/terraform/1.1.2/terraform_1.1.2_linux_amd64.zip
# Uncompress the binary package
ZIP_PACKAGE='terraform_1.1.2_linux_amd64.zip'; unzip "${ZIP_PACKAGE}" && rm -f "${ZIP_PACKAGE}"

Running scripts

The execution process of Terraform Scripts is divided into stages:

1) init: prepares the Terraform Workspace, sets run-time parameters, downloads, and installs providers.
2) plan: creates/updates state information of managed resources and executes a dry-run to create the change plan. For already existing resources, if the provider is unable to modify attributes on the fly it will propose a destructive change (remove and recreate the resource with the new end-state).
3) apply: executes the change plan, and updates resource state information. The provider will modify the target infrastructure by creating, updating and removing resources until the end-state is reached.

To execute the script the Terraform CLI must be already inside the workspace directory:

cd WORKSPACE_PATH
# Initialize the workspace
terraform init
# Prepare and review the action plan
terraform plan
# Execute the action plan
terraform apply

Notice that the user running the process doesn't need root privilege, just write permissions to the workspace path.

Script structure

The first thing to understand about Terraform Scripts is that the Terraform Language is not for general-purpose development. Based on HCL (Hashicorp Configuration Language), it's specifically designed to describe infrastructure resources.

Terraform Scripts are developed by defining blocks: a complex data structure that describes a configuration object through its attributes.

Use the following template for declaring blocks:

BLOCK_TYPE "RESOURCE_TYPE" "RESOURCE_LABEL1" {

  ATTRIBUTE1 = VALUE
  ATTRIBUTE2 = VALUE
  ATTRIBUTEN = VALUE

  # Nested block
  BLOCK_TYPE "RESOURCE_TYPE" "RESOURCE_LABEL2" {
    ATTRIBUTE = VALUE
  }

}

BLOCK_TYPE: defines what type of configuration object is going to be described. Notice that Blocks can have any number of attributes and other blocks (nested).
RESOURCE_TYPE: optional field used to select object subtypes. Not all objects will require one.
RESOURCE_LABEL: optional field to assign a label (ID) to the object.
ATTRIBUTE: name of the object's attribute.
VALUE: value that will be assigned to the object's attribute.

The following list shows common built-in block types:

variable
output
local
terraform
provider
resource
data

Additional types and subtypes are implemented through Terraform Providers. For example:

Scripts can have comments anywhere:

//: single-line comment. Anything at the right is ignored.
#: single-line comment. Anything at the right is ignored.
/* */: multi-line comment. Anything between the delimiters is ignored.

As a good practice, after the script is created/updated run the built-in linter and code-formatter:

cd WORKSPACE_PATH
terraform fmt
terraform validate

Using Variables

Declaration

As everything in Teraform, variables are also a block type. Use the following template to declare variables:

locals {
  VARIABLE_NAME = VALUE
}

Data Types

Both variables and attributes have types:

string: a sequence of Unicode characters.
number: integer or float numbers.
bool: boolean values.
list: indexed sequence of values. The index is numerical and starts at cero. VALUEs can be of any type.
map: group of key + value pairs. VALUEs can be of any type.
null: empty value.

Types are inferred from the value (i.e. no explicit data type declaration). For example:

locals {
  test_string = "abc123"
  test_number_integer = 15
  test_number_float = 15.1
  test_bool = true
  test_list = [
    "value 1",
    100
  ]
  test_map = {
    key1 = "value 1",
    key2 = 100
  }
}

Parameters

Parameters are variables that are used to provide run-time data to Terraform Scripts. They are declared using the variable block type:

variable "PARAMETER_NAME" {
  # Optional parameter attributes
  ATTRIBUTE = VALUE
}

They are different from regular variables as they allow attributes to further describe the parameter:

default: value that is assigned if none is provided.
description: text message that is used to describe the parameter.
type: expected variable type. This is not used to declare the type, but rather to constrain the provided value.
validation: validation block used to perform additional checks.
sensitive: flag to restrict script output. Used to mask sensitive values such as passwords.
nullable: flag to indicate if the value can be null.

For example:

variable "aws_region" {
  type = string
  default = "us-west-1a"
  description = "target AWS region"
  nullable = false
  sensitive = false
}

variable "xaas_test_api_secret" {
  type = string
  description = "API secret for the Test API"
  nullable = false
  sensitive = true
}

Output

In addition to the local and variable block types Terraform provides the output type to define variables that will be exported and made available to external apps and scripts. The following attributes are available:

value: the value assigned to the variable.
description: describe what the value represents.
sensitive: flag to restrict script output. Used to mask sensitive values such as passwords.

For example:

output "sysadmin_key" {
  value = "secret-api-key"
  description = "API secret for the new service"
  sensitive = true
}

Retreive variable's value

In general, any block attribute can be referenced from within other blocks by using the following structure: BLOCK_TYPE.ATTRIBUTE or BLOCK_TYPE.BLOCK_LABEL.ATTRIBUTE

In addition to the generic format there are special cases to consider:

Locals: use local instead of locals: local.ATTRIBUTE
Parameters: use the short form var instead of the full type name variable: var.ATTRIBUTE
Outputs: requires the module name where the output is declared: module.MODULE NAME.OUTPUT_NAME

Variables can also be used in String Templates (simple programming code that can be embedded inside string declarations): "${BLOCK_TYPE.ATTRIBUTE}"

The following example shows the three variable types seen so far:

locals {
  api_secret_seed = "abc123"
}

variable "new_service_name" {
  type = string
  default = "new_service_name"
  description = "What name will the new service have"
  nullable = false
  sensitive = false
}

output "service_secret" {
  value = "${local.api_secret_seed}${var.new_service_name}"
  description = "API secret for the new service"
  sensitive = true
}

Next Steps

Continue reading the second part of the tutorial: Terraform Scripting Concepts: Part 2 of 2

Copyright information

This article is licensed under a Creative Commons Attribution 4.0 International License. For copyright information on the product or products mentioned inhere refer to their respective owner.

Disclaimer

Bash Scripting Concepts: Part 2 of 2

SerDigital64 — Wed, 22 Dec 2021 03:07:56 +0000

Overview

This is the second part of the tutorial. If not done already, please read the first part: Bash Scripting Concepts: Part 1 of 2

Working with loops

Bash provides three ways of creating loops:

for: loop for a predefined number of times
while: loop while the exit condition is true. The condition is evaluated before executing tasks.
until: loop until the exit condition is true. The condition is evaluated after executing tasks.

In addition to the loop condition, Bash provides two statements that can be used to control the loop execution flow:

break: forces the loop to stop.
continue: forces the loop to skip remaining tasks and start the next iteration.

Command: `while`

In the following example, the loop iterates 5 times. The loop condition is evaluated after all commands in the code block are executed:

#!/bin/bash

declare -i count=1
declare -i max=5

while ((count <= max)); do
  printf 'counter: %s\n' "${count}"
  count=$((count + 1))
done

Command: `until`

Using the same structure as in the while example, notice that now the loop iterates 4 times only. This is because the loop condition is evaluated before executing the code block:

#!/bin/bash

declare -i count=1
declare -i max=5

until ((count == max)); do
  printf 'counter: %s\n' "${count}"
  count=$((count + 1))
done

Command: `for`

In the case of the for loop the iteration is predefined. Instead of having a loop condition, the loop variable count will be assigned each value in the list:

#!/bin/bash

declare -i count

for count in 1 2 3 4 5 ; do
  printf 'counter: %s\n' "${count}"
done

Working with conditionals

Bash provides the following options for implementing conditional execution:

||: (logical OR) evaluates the execution of two commands and sets the exit status to zero if any associated exit status is zero.
&&: (logical AND) evaluates the execution of two commands and sets the exit status to zero if all associated exit statuses are zero.
\!: (logical NOT) evaluates the execution of a command and sets the exit status to zero if the associated exit status is not zero.
(( )): performs logical evaluation on the integer expression and sets the exit status to zero if true
[[ ]]: evaluates the literal expression and sets the exit status to zero if true.
if: executes a command and if the exit status is zero then performs additional actions.
case: compares the provided value against a list of patterns and executes the commands upon match.

As mentioned before, Bash interprets the exit status of commands as:

0: true
>0: false

In the following examples true and false are external commands that emulates true and false values (exist status 0 and 1 respectively)

Logical OR: `||`

#!/bin/bash

true || false
printf 'evaluation result of (true || false): %s\n' $?
true || true
printf 'evaluation result of (true || true): %s\n' $?
false || false
printf 'evaluation result of (false || false): %s\n' $?
false || true
printf 'evaluation result of (false || true): %s\n' $?

Logical AND: `&&`

#!/bin/bash

true && false
printf 'evaluation result of (true && false): %s\n' $?
true && true
printf 'evaluation result of (true && true): %s\n' $?
false && false
printf 'evaluation result of (false && false): %s\n' $?
false && true
printf 'evaluation result of (false && true): %s\n' $?

Logical NOT: `!`

#!/bin/bash

! true
printf 'evaluation result of (! true): %s\n' $?
! false
printf 'evaluation result of (! false): %s\n' $?

Arithmetic Expression Evaluation: `(( ))`

The (( )) form accepts several logical operators. Some of them are:

==: equal
!=: not equal
>: greater than
<: less than
>=: greater than or equal
<=: less than or equal

#!/bin/bash

declare -i test=$RANDOM
(( ${test} > 5000 ))
printf 'evaluation result of (( %s > 5000 )): %s\n' ${test} $?

Expression Evaluation: `[[ ]]`

The [[ ]] form accepts several logical operators and tests. Some of them are:

==: equal
!=: not equal
-z: string is empty
-n: string is not empty
-f: path is a file

For == and != the special character * can be used as a wildcard to match zero or more characters to the right.

#!/bin/bash

declare test="$RANDOM"
[[ "${test}" == 1* ]]
printf 'evaluation result of [[ "%s" == 1* ]]: %s\n' "${test}" $?

Command: `if`

In the following example, arithmetic evaluation is used. Notice that quotes are not required within (( ))

#!/bin/bash

declare -i test=${RANDOM}
if (( ${test} >= 10000 )); then
  printf 'test value (%s) is equal or greater than 10000\n' ${test}
elif (( ${test} > 5000 || ${test} < 10000 )); then
  printf 'test value (%s) is between 5001 and 9999\n' ${test}
else
  printf 'test value (%s) is less than than 5001\n' ${test}
fi

Command: `case`

#!/bin/bash

declare -i test=${RANDOM}
case ${test} in
  1*|2*) printf 'Random number (%s) starts with 1 or 2\n' ${test};;
  3*) printf 'Random number (%s) starts with 3\n' ${test};;
  *) printf 'Random number (%s) does not start with 1,2 or 3\n' ${test};;
esac

Redirecting data flows

Bash provides two alternatives for establishing data flows:

Redirection
- Set read source for STDIN
- Set write destination for STDOUT
- Set write destination for STDERR
Pipelines: integrate two commands by plugging the STDOUT from the first one to the STDIN of the second one

Redirection

In the following example, two functions will communicate with each other using a common file:

#!/bin/bash

readonly DATA_BRIDGE="$(mktemp)"

function produce_data() {
  printf 'produce_data(): write data to the temporary file (%s) by redirectin the STDOUT of the printf command\n' "${DATA_BRIDGE}"
  printf '[sample data]\n' >"${DATA_BRIDGE}"
}

function ingest_data() {
  printf 'ingest_data(): read data from the temporary file (%s) by redirecting the STDIN of the cat command: ' "${DATA_BRIDGE}"
  cat < "${DATA_BRIDGE}"
}

produce_data
ingest_data

rm -f "${DATA_BRIDGE}"

Pipelines

This example shows an alternative way of integrating both functions using pipelines:

#!/bin/bash

function produce_data() {
  printf '[sample data]\n'
}

function ingest_data() {
  cat
}

printf 'Integrate functions produce_data() and ingest_data() by piping their STDIN and STDOUT: '
produce_data | ingest_data

Next Steps

Discover advanced features by exploring the Bash Reference Manual:

jobs
signals
traps
parallelism
error handling
configuration settings

Organize your code by choosing a coding style. For example: Google Shell Style Guide

Enhance script's quality by incorporating linter and testing tools:

Copyright information

This article is licensed under a Creative Commons Attribution 4.0 International License. For copyright information on the product or products mentioned inhere refer to their respective owner.

Disclaimer

Bash Scripting Concepts: Part 1 of 2

SerDigital64 — Wed, 22 Dec 2021 03:06:52 +0000

Overview

Bash is a tool that provides a text-based user interface for managing operating system resources. It's also the run-time environment for a simple interpreted programming language that can be used to create scripts for automating tasks.

Bash is the evolution of the popular Born Shell (sh), a de facto standard among Unix-like operating systems. It provides all the features of its predecessor plus a new set of built-ins and configuration settings.

Using Bash

There are three common use cases for Bash:

User shell: configured at the operating system level, provides the shell for user login, either local or remote.
Script run-time: provides the execution run-time for Bash scripts
Temporal shell: provides an interactive shell session on top of the user shell. The new session is a sub-process of the actual user or a different user when using privilege delegation tools such as SuDO.

This tutorial will focus on the scripts run-time use case.

Script structure

Bash scripts are text files describing a sequence of commands:

Commands can be either built-in or external Linux apps.
Commands are separated from each other by the new-line character (standard line separator for Linux/Unix systems).
Long lines can be split using the scape character \
Commands can be grouped on the same line using the semicolon separator ;
Commands and arguments are separated using one or more spaces.
Although not required, it's recommended that the first line of the script contains: #!/bin/bash
Script files must have execution and read permissions if used directly, or just read-only permissions if called using Bash.

For example:

#!/bin/bash

echo 'echo and printf are built-in commands'

printf 'pwd is an external Linux command that shows the current directory: '
pwd

printf 'multiple '; printf 'commands '; printf 'on the same line\n'

printf '%s %s %s\n' \
  'single command' \
  'using multiple' \
  'lines'

Using Variables

Declaration

Variables in Bash are created using a declaration command and the equal = symbol to associate the variable name to its value. Notice that there should be no spaces before and after the assignment symbol:

declare variable_name='variable value'

Bash provides the following built-in commands for declaring variables:

Command	Usage	Scope
`export`	Declare global variables	Script-wide and sub-processes
`readonly`	Declare readonly variables (constants)	Script-wide
`declare`	Declare script level variables	Script-wide
`local`	Declare function level variables	Function-wide

Consider the following best practices:

Declare variables at the beginning of the code block.
Declare and initialize all the variables that the script will use.
Readonly and exported variables should be named using all uppercase.
Use single quotes for static content.

Retreive variable's value

To use the value of a variable, prepend the dollar symbol $ to the name of the variable surrounded by brackets {}.

As a best-practice, surround variable expansion with double quotes:

echo "${test_variable}"

Bash also provides processing functions that can be used for altering the value before the expansion.

For example:

#!/bin/bash

declare var1='word1'
declare var2='WORD2'
declare var3='AbCdE/12345'

printf 'replace value with its length: "${#var1}" = "%s"\n' "${#var1}"
printf 'right strip from pattern: "${var3##*/}" = "%s"\n' "${var3##*/}"
printf 'left strip from pattern: "${var3%%/*}" = "%s"\n' "${var3%%/*}"
printf 'convert to uppercase: "${var1^^}" = "%s"\n' "${var1^^}"
printf 'convert to lowercase: "${var2,,}" = "%s"\n' "${var2,,}"

Data Types

Bash supports four data types:

Type	Declaration command
string	declare
integer	declare -i
array	declare -a
associative array	declare -A

Even though there is no explicit boolean data type, Bash interprets the exit status of commands as:

exit status == 0: true
exit status > 0: false

Parameters

Parameters are special variables that are automatically created when a function or script is called with arguments.

The variable name is created by using a single number to represent the position of the word in the argument list:

bash say_hello.bash 'to' 'my' 'little' 'friend'

#!/bin/bash

declare msg1="$1" # Variable $1: assigned to the first variable
declare msg2="$2" # Variable $2: assigned to the second variable
declare msg3="$3" # Variable $3: assigned to the third variable
declare msg4="$4" # Variable $4: assigned to the forth variable

printf 'say hello %s %s %s %s\n' "${msg1}" "${msg2}" "${msg3}" "${msg4}"

Shell Expansion

In addition to variables Bash provides additional features for generating dynamic values:

$( ): Command Expansion: Execute the command in a sub-process and expand its STDOUT.
$(( )): Arithmetic Expansion: Evaluate the arithmetic expression and use the resulting value for the expansion.
$(< ): File Expansion: Read the content of a file and use it for the expansion.

For example:

#!/bin/bash

printf 'Command Expansion: "$(echo hello-world)" = "%s"\n\n' "$(echo 'hello-world')"
printf 'Arithmetic Expansion:  "$(( 2 + 2 ))" = %s\n\n' "$(( 2 + 2 ))"
printf 'File Expansion: "$(</etc/os-release)"\n %s\n' "$(</etc/os-release)"

Working with processes

Before going into the details, let's review the following key concepts about Unix processes:

A process is a running app that is executed from an already running process (parent). For scripts, the parent process is the one running the Bash run-time.
The Operating System assigns unique integer identifiers to each process (PID: Process ID).
Processes have 3 data paths:
- STDIN: standard input: process can read data from this path.
- STDOUT: standard output: process can write data to this path.
- STDERR:: standard error: process can write error diagnostic data to this path.
Exit status: numeric value that represents the final execution status of the process. In general:
- 0: successful execution
- >0: failed execution. The app can assign different numbers to further describe the error.

Bash provides the following features for interacting with processes:

Variables: show process information
- $?: exit status of the last executed command
- $BASHPID: PID of the current Bash process
Redirection: redirect the data flow from the STDIN, STDOUT, and STDERR
Pipelines: integrate two processes by creating a pipe between the STDOUT from one to the STDIN of the other

Additional features are available but no covered in the current tutorial (jobs, signals, traps, parallelism, etc.)

Implementing Functions

To declare a function in Bash use the following structure:

function <FUNCTION_NAME>() {
  <COMMANDS>
  return <EXIT_STATUS>
}

Functions in Bash behave in a similar way to scripts and commands:

Can be called directly from the script or command line
Use positional parameters that are automatically assigned to $N variables
Have exit status
Can send data to STDOUT and STDERR
Can receive data from STDIN
Can be used in complex command sequences (&&, ||, |)

Let's create a function with all the concepts seen so far:

#!/bin/bash

readonly TEST_READONLY='content of this variable is constant and can not be modified afterwords'
export TEST_EXPORT='content of this variable is visible everywhere'
declare test_declare='this variable is declared at the script level'
declare -i function_result=0

function test_function() {

  local test_parameter="$1"
  local test_parameter_default="${2:-default-value}"

  local test_local='this variable is available only inside this function'
  local test_static=''  # initialize a static variable
  local test_dynamic='' # initialize a dynamic variable

  test_static='this variable has static content'
  test_dynamic="$(echo 'this dynamic variable is assigned at execution time')"

  printf 'Show variable content from inside a function:\n\n'
  printf '  first parameter: [%s]\n' "${test_parameter}"
  printf '  second parameter, default value: [%s]\n' "${test_parameter_default}"
  printf '  local static variable: [%s]\n' "${test_static}"
  printf '  local dynamic variable: [%s]\n' "${test_dynamic}"
  printf '  exported variable: [%s]\n' "${TEST_EXPORT}"
  printf '  readonly variable: [%s]\n' "${TEST_READONLY}"
  printf '  declared variable: [%s]\n' "${test_declare}"

  return 5

}

test_function 'this value is assigned to the first parameter'
function_result=$?

printf '\nShow the same variables but outside the function:\n\n'
  printf '  local variable: [%s]\n' "${test_local}"
  printf '  local static variable: [%s]\n' "${test_static}"
  printf '  local dynamic variable: [%s]\n' "${test_dynamic}"
  printf '  exported variable: [%s]\n' "${TEST_EXPORT}"
  printf '  readonly variable: [%s]\n' "${TEST_READONLY}"
  printf '  declared variable: [%s]\n' "${test_declare}"

printf '\nShow the exit status (return value) of the function: [%s]\n' "${function_result}"
printf '\nShow current PID of the Bash run-time: [%s]\n' "${BASHPID}"

Next Steps

Continue reading the second part of the tutorial: Bash Scripting Concepts: Part 2 of 2

Copyright information

This article is licensed under a Creative Commons Attribution 4.0 International License. For copyright information on the product or products mentioned inhere refer to their respective owner.

Disclaimer

Implementing infrastructure-as-code with Ansible and GIT

SerDigital64 — Fri, 01 Oct 2021 22:17:04 +0000

What is "infrastructure-as-code"?

To keep the concept simple, think of your infrastructure as a picture (end-state) and the main characteristics (configuration) that you would use to describe it.

Using this information you should be able to reproduce the picture any time you need, and the result should always be the same as the original.

There are several tools and methods to implement this approach but the most important thing to consider is that you also need to adapt your management procedures to switch from the traditional imperative model to the infrastructure-as-code declarative model.

Challenges

To successfully implement infrastructure-as-code special attention must be paid to the selection of supporting tools and the definition of the conceptual model that will represent the managed environment:

Automation tool: takes control of the infrastructure configuration and performs the necessary actions to reach the desired end-state.
Code Repository and Versioning: stores the infrastructure-model and automation scripts to manage the infrastructure and tracks changes.
Infrastructure-model: conceptual data model that describes the desired end-state of the infrastructure.

Implementing infrastructure-as-code

Let's take the following example scenario to demonstrate the implementation procedure:

Environment: Home Office
Ansible Control Node: small PC or VM installed with Centos 8.4
Ansible Managed Nodes: notebooks with Ubuntu and workstations with Centos and Fedora

Before starting the implementation procedure, make sure the following requirements are meet:

Control Node:
- OpenSSH client
- Sudo
- Python3
- GIT
- Regular user with SUDO configured for password less root privilege
Managed Nodes:
- Fresh OS install (standard setup)
- OpenSSH server
- Sudo
- Python3
- Regular user with SUDO configured for password less root privilege

Define the Infrastructure Model

The infrastructure-model will have the following data structures:

Site: Represents a group of Nodes that are managed by the same Control Node.
Node: Compute node that is capable of hosting software components and that is fully managed by the Control Node.
Component: Individual software product that is installed in a Node.
Service: Group of Components configured in one or more Nodes to serve a particular function.

The data structures will be implemented using Ansible inventories, host_vars, and group_vars:

Inventory: the hosts.ini file will be used to declare all the Nodes for the target Site. Nodes will be grouped based on the Service they provide or use.
GroupVars: individual YAML files for Components and Services will be created for each Node group declared in the Inventory.
HostVars: individual YAML files will be used for cases where the Node requires further customization.

Create the Code Repository

Create a dedicated Linux account. This is to isolate content from regular users and facilitate activity auditing. Modify the shell variable PROJECT_OWNER to change the default name.

PROJECT_OWNER='sitectl'
sudo useradd -m "$PROJECT_OWNER"
sudo su - "$PROJECT_OWNER"

Create the project directory structure that will contain the infrastructure-model and automation scripts. Refer to the Ansible Best Practices document to further learn about the directory structure.
Modify the shell variable PROJECT_PATH to change the default project location.

export PROJECT_OWNER='sitectl'
export PROJECT_PATH="/home/${PROJECT_OWNER}/manager"
mkdir "$PROJECT_PATH"
cd "$PROJECT_PATH"
mkdir \
  'collections' \
  'files' \
  'inventories' \
  'inventories/group_vars' \
  'inventories/host_vars' \
  'playbooks' \
  'vars' \
  'etc' \
  'filter_plugins' \
  'library' \
  'module_utils' \
  'roles' \
  'var' \
  'templates'

Create a simple shell script to set environment variables for Ansible. Refer to the Ansible Configuration documentation to understand what are the ANSIBLE_* shell variables doing.

cat > 'load_environment.sh' <<-EEOF
#!/bin/bash

declare -x PROJECT_OWNER='$PROJECT_OWNER'
declare -x PROJECT_PATH='$PROJECT_PATH'
declare -x PROJECT_END_STATE='${PROJECT_PATH}/inventories'

declare -x ANSIBLE_INVENTORY="\${PROJECT_END_STATE}/hosts.ini"
declare -x ANSIBLE_PRIVATE_KEY_FILE="/home/\${PROJECT_OWNER}/.ssh/id_rsa"
declare -x ANSIBLE_COLLECTIONS_PATHS="\${PROJECT_PATH}/collections"
declare -x ANSIBLE_ROLES_PATH="\${PROJECT_PATH}/roles"
declare -x ANSIBLE_GALAXY_CACHE_DIR="\${PROJECT_PATH}/var"
declare -x ANSIBLE_LOG_PATH="\${PROJECT_PATH}/var/ansible.log"
declare -x ANSIBLE_PYTHON_INTERPRETER='/usr/bin/python3.9'
declare -x ANSIBLE_PLAYBOOK_DIR="\${PROJECT_PATH}/playbooks"

PATH='/home/${PROJECT_OWNER}/.local/bin:/usr/bin:/usr/sbin'

EEOF

Create the Code Repository using GIT. Configure the .gitignore file to avoid tracking changes in the ansible-galaxy install location target (collections/) and in the repository for temporary and volatile files (/var)

source load_environment.sh
cat > '.gitignore' <<-EEOF
collections/
var/

EEOF
git config user.email "${PROJECT_OWNER}@localhost.localdomain"
git config user.name "$PROJECT_OWNER"
git init
git add .
git commit -m "Initial commit"

Configure Ansible

Install the latest Ansible engine to the control node user. This method keeps the environment isolated, minimizes os-distribution dependencies, and facilitates module upgrades.

"$ANSIBLE_PYTHON_INTERPRETER" -m pip install ansible

Prepare remote access to Ansible Managed Nodes using OpenSSH keys. Modify the shell variable PROJECT_MANAGED_NODES to represent your environment and set the PROJECT_MANAGED_USER variable to the remote user name with password-less root privilege.

PROJECT_MANAGED_NODES='host1 host2 host3 host4'
PROJECT_MANAGED_USER='sysadmin'
ssh-keygen -t rsa -f "$ANSIBLE_PRIVATE_KEY_FILE" -N ""
for x in $PROJECT_MANAGED_NODES; do
  ssh-copy-id -i "$ANSIBLE_PRIVATE_KEY_FILE" ${PROJECT_MANAGED_USER}@${x}
done

Create the initial Ansible inventory registering the following Node groups:

[control_node]: defines the Ansible Node.
[managed_nodes]: defines Ansible Managed nodes for the target site.
[office_nodes]: defines Nodes that will consume the office-service. The service provides users with common productivity applications. For this example, we'll use the image editor GIMP.

cat > "$ANSIBLE_INVENTORY" <<-EEOF
[control_node]
localhost

[managed_nodes]
$(for x in $PROJECT_MANAGED_NODES; do echo "$x"; done)

[office_nodes]
$(for x in $PROJECT_MANAGED_NODES; do echo "$x"; done)

EEOF

Create end-state configuration repositories. This is where the infrastructure-model will be implemented.

group_vars/: one directory per host group
group_vars/host_group_x/: one or more YAML files representing the components that will be available for all hosts in the group
host_vars/: one directory per host
host_vars/hostx/: one or more YAML files representing the components that will be available for the host

for x in control_node managed_nodes office_nodes; do
  mkdir "${PROJECT_END_STATE}/group_vars/${x}"
done
for x in $PROJECT_MANAGED_NODES; do
  mkdir "${PROJECT_END_STATE}/host_vars/${x}"
done

Define component end-states

Now that the data repository for the infrastructure-model is created it can be populated with end-state targets. Notice that you can also add behaviour definitions to keep variable data separated from the code.

Define how is the Ansible engine going to connect to the managed nodes:

cat > "$PROJECT_END_STATE/group_vars/managed_nodes/ansible.yml" <<-EEOF
---
ansible_user: "$PROJECT_MANAGED_USER"
ansible_become_method: "sudo"
...

EEOF

Define the attributes for the Linux Users component. This definition will be applied to all hosts in the group managed_nodes:

cat > "$PROJECT_END_STATE/group_vars/managed_nodes/users.yml" <<-EEOF
---
managed_nodes_users:
  - name: "user1"
    description: "Regular User 1"
    uid: "10100"
  - name: "user2"
    description: "Regular User 2"
    uid: "10101"
...

EEOF

Define the attributes for the Linux Package component. This definition will be applied to all hosts in the group office_nodes:

cat > "$PROJECT_END_STATE/group_vars/office_nodes/packages.yml" <<-EEOF
---
office_nodes_packages:
  flatpak:
    - "flatpak"
  gimp:
    - "org.gimp.GIMP"
...

EEOF

Bring the site to the target end-state

At this point end-state, and behaviour definitions are set. Now it's time to write the code that will apply it to the target hosts.

Create the Ansible Playbook that will configure the managed_nodes host group:

cat > "$ANSIBLE_PLAYBOOK_DIR/managed_nodes.yml" <<-EEOF
---
- name: "Manage Ansible Nodes"
  hosts: "managed_nodes"
  gather_facts: false

  tasks:
    - name: "Create Regular User Accounts"
      become: true
      ansible.builtin.user:
        create_home: true
        state: "present"
        name: "{{ item['name'] }}"
        comment: "{{ item['description'] | default( omit ) }}"
        uid: "{{ item['uid'] | default( omit ) }}"
      loop: "{{ managed_nodes_users }}"
...

EEOF

Create the Ansible Playbook that will configure the office_nodes host group:

cat > "$ANSIBLE_PLAYBOOK_DIR/office_nodes.yml" <<-EEOF
---
- name: "Manage Office Nodes"
  hosts: "office_nodes"
  gather_facts: false

  pre_tasks:

    - name: "Install FlatPak tools"
      become: true
      ansible.builtin.package:
        name: "{{ office_nodes_packages['flatpak'] }}"
        state: "present"

    - name: "Prepare FlatPak repository"
      become: true
      ansible.builtin.command:
        argv:
          - "/usr/bin/flatpak"
          - "--system"
          - "remote-add"
          - "flatpak"
          - "https://flathub.org/repo/flathub.flatpakrepo"
      register: result
      changed_when:
        - result['rc'] == 0

  tasks:
    - name: "Install GIMP from FlatHub"
      become: true
      community.general.flatpak:
        name: "{{ office_nodes_packages['gimp'] }}"
        state: "present"
...

EEOF

Execute the playbooks to apply the end-state:

ansible-playbook playbooks/managed_nodes.yml
ansible-playbook playbooks/office_nodes.yml

Save the changes to the repository:

git add inventories
git add playbooks
git commit -m "add office_node and managed_node plays"

Next steps

Now the base structure is up and running more content can be added, either from Ansible Galaxy or developed in-house.

In addition to the automation engine and code repository you should evaluate incorporating:

Code linters: Ansible Lint and YAMLlint can help to keep code consistent and standardized.
Testing: Ansible Molecule can be used to build and run test environments for testing in-house roles
Provisioning: Terraform can be used to automate the creation of standardized VMs

Explore the A:Platform64 project that facilitates the implementation of infrastructure-as-code by automating most of the tasks described in this tutorial.

Copyright information

This article is licensed under a Creative Commons Attribution 4.0 International License. For copyright information on the product or products mentioned inhere refer to their respective owner.

Disclaimer

Ansible Concepts

SerDigital64 — Sat, 17 Jul 2021 00:00:00 +0000

Ansible Overview

Ansible is an automation engine that executes tasks described in YAML + Jinja2 to bring the target system to a known state.

At its core, Ansible is a python module:

(ansible) [sysadmin@cnt84 ~]$ pip list | egrep -i -- 'ansible|jinja|yaml|markup'
ansible      4.3.0
ansible-core 2.11.3
Jinja2       3.0.1
MarkupSafe   2.0.1
PyYAML       5.4.1

The engine along with other helpers are available as stand-alone programs that can be run directly from the command line:

ansible : used to run individual tasks.
ansible-playbook : used to execute a batch of tasks in multiple targets.
ansible-config : used to define engine configuration settings.
ansible-doc : used to show documentation of several ansible components.
ansible-galaxy : used to interact with the ansible packaging system.

Ansible Tasks

The minimum execution unit in Ansible is a task. Each task will focus on managing a particular component of the target host.

The main difference between an Ansible Task and traditional shell script commands is that the task defines the what (target state) and not the how (action).

For example, to set file permissions using a Bash script you need to define how it will be done:

#!/bin/bash

echo "Example: Set file permissions"
/usr/bin/chmod '0644' '/tmp/test.file'

To achieve the same result using Ansible Tasks you define what will be the end-state of the component:

--------
- name: "Example: Set file permissions"
  ansible.builtin.file:
    mode: "0644"
    path: "/tmp/test.file"

Here the component is the Linux file /tmp/test.file managed by the ansible.builtin.file Ansible Module and the end-state is defined by the mode attribute.

See below common shell script actions and their equivalent Ansible Tasks

Bash	Ansible
Action: Copy FileA to DestinationB	Target State: FileA must be present in DestinationB
Action: Set FileA Owner to UserX	Target State: UserX must own FileA
Action: Start the ServiceX	Target State: ServiceX must be in started state
Action: Install PackageX	Target State: PackageX must be installed

Ansible Modules

Ansible Modules are used to represent components in Ansible Tasks and are intended to hide the complexity of how the component is manipulated to achieve the desired end-state.

A set of modules are already included in Ansible for managing common components. For example:

Component	Module Name	End-State
Linux File	copy	Source File is present in the Target Path
Linux File	file	File attributes are set (ownership, permissions, etc)
Linux File	lineinfile	The text line is included (or not) in the target file
Linux OS Package	package	The OS package is present (or not) in the target host
Linux OS Service	service	The OS service is started (or not) in the target host
OS User	user	OS User is created (or not) in the target host
OpenSSH	authorized_keys	OpenSSH key is present (or not) in the authorized_keys file

Additional modules developed by product owners or the OSS community are available at the Ansible Galaxy site and can be installed using the ansible-galaxy command.

For example, to add the openssh_keypair Ansible Module included in the community provided community.crypto.openssh_keypair Ansible Collection:

(ansible) [sysadmin@cnt84 ~]$ ansible-galaxy collection install community.crypto
Starting galaxy collection install process
Process install dependency map
Starting collection install process
Downloading https://galaxy.ansible.com/download/community-crypto-1.7.1.tar.gz to /home/sysadmin/.ansible/tmp/ansible-local-321219he000qkm/tmpd12y1_sb/community-crypto-1.7.1-xrp950jz
Installing 'community.crypto:1.7.1' to '/home/sysadmin/.ansible/collections/ansible_collections/community/crypto'
community.crypto:1.7.1 was installed successfully

Ansible Playbooks

Similar to what scripts are for traditional shells like bash, Ansible Playbooks are YAML files used to create automation jobs.

The minimum Ansible Playbook contains one or more Ansible Tasks and the explicit declaration of the target host where the end-state will be set.

The following example playbook sets test-server21 as the target host where the end-state for the package component will be set. In this case, the desired end-state is to have the package lsof installed.

- hosts: test-server21
  tasks:
    - name: "Deploy LSOF tool"
      ansible.builtin.package:
        name:
          - "lsof"
      state: "present"

Ansible Playbooks can contain additional features to facilitate the creation of complex automation jobs:

Job management: defines how tasks are going to be executed based on the number of target hosts (sequentially, in parallel, etc.)
Error handling: provides features for recovering from failed tasks.
Event Handlers: defines special tasks that are only executed when certain conditions are met. For example, the application X configuration reload handler is executed when the configuration update task sets a new parameter value
External Variable Definition: allows the inclusion of YAML files that contains variables definitions only.
Roles: allows the inclusion of roles. Roles are similar to Ansible Modules but implemented using Ansible Tasks.

To run Ansible Playbooks use the command ansible-playbook:

(ansible) [sysadmin@cnt84 ~]$ ansible-playbook hello_world.yml

PLAY [localhost] *****************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************
ok: [localhost]

TASK [Hello World] ***************************************************************************************************
ok: [localhost] => {
    "msg": "Hello World"
}

PLAY RECAP ***********************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

YAML and Jinja

Ansible doesn't have a language on its own for defining tasks.
Instead, it uses the data definition language YAML for declaring desired end-state and the template engine Jinja2 for embedding simple programming functions into YAML files.

For example, let's define the file_path variable in a stand-alone YAML file:

--------
file_path: "/tmp/test.file"

Now we can use the file_path variable to set the path attribute.
To use the variable a Jinja2 template must be added to the YAML declaration using the expression container "{{ }}":

--------
- name: "Example: Include variable from file"
  include_vars:
    file: "example_var_definition.yml"
- name: "Example: Set file permissions"
  ansible.builtin.file:
    mode: "0644"
    path: "{{ file_path }}"

Ansible Infrastructure

Ansible defines two types of infrastructure components:

Control Node: central compute node where the automation engine is installed and from where Ansible Tasks are executed.
Managed Node: target node that Ansible Tasks will manage to reach the defined end-state. Nodes can be regular compute nodes (RedHat Linux Enterprise, Ubuntu, etc.) or non-compute nodes such as storage devices, network devices, appliances, etc.

Communication between Control Nodes and Managed Nodes is implemented using Connection Plugins. The default plugin for Linux nodes is ssh.

Use the command ansible-doc -t connection -l to show available plugins:

(ansible) [sysadmin@cnt84 ~]$ ansible-doc -t connection -l | egrep '^ssh'
ssh                            connect via ssh client binary

For tasks that requires privileged access to manage the component Ansible provides Become Plugins. The default plugin for Linux based nodes is sudo.

Use the command ansible-doc -t become -l to show available plugins:

(ansible) [sysadmin@cnt84 ~]$ ansible-doc -t become -l | egrep '^sudo|^su'
su                           Substitute User
sudo                         Substitute User DO

Content Organization

Ansible doesn't enforce a strict directory structure for content organization. Instead, it provides configuration parameters that can be used to define locations based on the resource type.

The following is a basic directory structure that can be used for simple to medium size deployments:

Path	Content	Ansible Parameter
etc/	Ansible configuration files	ANSIBLE_CONFIG
files/	Site wide data files
inventories/	Ansible Playbooks inventory files, host_vars and group_vars	ANSIBLE_INVENTORY
collections/	Collections installed from Ansible-Galaxy	ANSIBLE_COLLECTIONS_PATHS
roles/	Ansible Roles	ANSIBLE_ROLES_PATH
playbooks/	Ansible Playbooks	ANSIBLE_PLAYBOOK_DIR
logs/	Execution logs	ANSIBLE_LOG_PATH

Once resources are organized you should consider managing the content using a version control system like GIT. This is key to implement the infrastructure as code strategy.

References & Resources

Ansible Community Documentation: https://docs.ansible.com/ansible_community.html
Ansible Galaxy: https://galaxy.ansible.com
Jinja2: https://jinja.palletsprojects.com/en/3.0.x
YAML: https://yaml.org/spec
GIT: https://git-scm.com
Infrastructure-as-code Tutorial: https://serdigital64.github.io
A:Platform64 Automation Platform: https://serdigital64.github.io

Copyright information

This article is licensed under a Creative Commons Attribution 4.0 International License. For copyright information on the product or products mentioned inhere refer to their respective owner.

DEV Community: SerDigital64

Terraform Scripting Concepts: Part 2 of 2

Overview

Using functions

Managing infrastructure

Declare Providers

Define Managed Resources

Next Steps

Copyright information

Disclaimer

Terraform Scripting Concepts: Part 1 of 2

Overview

Using Terraform

CLI Installation

Running scripts

Script structure

Using Variables

Declaration

Data Types

Parameters

Output

Retreive variable's value

Next Steps

Copyright information

Disclaimer

Bash Scripting Concepts: Part 2 of 2

Overview

Working with loops

Command: while

Command: until

Command: for

Working with conditionals

Logical OR: ||

Logical AND: &&

Logical NOT: !

Arithmetic Expression Evaluation: (( ))

Expression Evaluation: [[ ]]

Command: if

Command: case

Redirecting data flows

Redirection

Pipelines

Next Steps

Copyright information

Disclaimer

Bash Scripting Concepts: Part 1 of 2

Overview

Using Bash

Script structure

Using Variables

Declaration

Retreive variable's value

Data Types

Parameters

Shell Expansion

Working with processes

Implementing Functions

Next Steps

Copyright information

Disclaimer

Implementing infrastructure-as-code with Ansible and GIT

What is "infrastructure-as-code"?

Challenges

Implementing infrastructure-as-code

Define the Infrastructure Model

Create the Code Repository

Configure Ansible

Define component end-states

Bring the site to the target end-state

Next steps

Copyright information

Disclaimer

Ansible Concepts

Ansible Overview

Ansible Tasks

Ansible Modules

Ansible Playbooks

YAML and Jinja

Ansible Infrastructure

Content Organization

Command: `while`

Command: `until`

Command: `for`

Logical OR: `||`

Logical AND: `&&`

Logical NOT: `!`

Arithmetic Expression Evaluation: `(( ))`

Expression Evaluation: `[[ ]]`

Command: `if`

Command: `case`