DEV Community: Montana Mendy

Getting started with an open source NSA tool to construct distributed graphs

Montana Mendy — Sat, 26 Feb 2022 23:33:16 +0000

Datawave is a Java-based ingest and query framework that leverages Apache Accumulo to provide fast, secure access to your data. Datawave supports a wide variety of use cases, including but not limited to:

Data fusion across structured and unstructured datasets
Construction and analysis of distributed graphs
Multi-tenant data architectures, with tenants having distinct security requirements and data access patterns
Fine-grained control over data access, integrated easily with existing user-authorization services and PKI

Below is the basic structure of Datawave:

Here's a graphic I made that get's a little more in-depth:

Now that you know how the architecture flows a bit better let's start with installing Datawave, then we'll get into Edges.

Getting started with NSA's Datawave

Before we start, NB: You should have an understanding of using simple Bash scripting, Linux commands like grep awk and using piping.

What you'll need

Linux, Bash, and an Internet connection to wget tarballs (tar.gz)that you should be able to ssh to localhost without a passphrase.

Note that the quickstart Hadoop install will set up passphrase-less ssh for you automatically, unless it detects that you already have a private/public key pair generated

Familiarize yourself with swap and/or swapping if you haven't already. You'll need this. https://wiki.gentoo.org/wiki/Swap

Installing Datawave through the CLI in 5 commands

echo "source DW_SOURCE/contrib/datawave-quickstart/bin/env.sh" >> ~/.bashrc
source ~/.bashrc                                                              
allInstall                                                                   
datawaveWebStart && datawaveWebTest

So we're adding sources to our .bashrc this can be true too if you're using zsh.

The four commands above will complete the entire quickstart installation. However, it’s a good idea to at least skim over the sections below to get an idea of how the setup works and how to customize it for your own preferences.

To keep things simple, DataWave, Hadoop, Accumulo, ZooKeeper, and Wildfly will be installed under your DW_SOURCE/contrib/datawave-quickstart directory, and all will be owned by / executed as the current user, hence why a bash script in the background was being ran.

Overriding your default binaries

On some occasions you may need to override the default binaries (not in all machines, or setups). Let's open up Vim, and do this in case you do need to end up overriding your binaries. To override the quickstarts default version of a particular binary, simply override the desired DW_*_DIST_URI value as shown below. URIs may be local or remote. Local file URI values must be prefixed with file://, so let's start:

vi ~/.bashrc

export DW_HADOOP_DIST_URI=file:///my/local/binaries/hadoop-x.y.z.tar.gz
     export DW_ACCUMULO_DIST_URI=http://some.apache.mirror/accumulo/1.x/accumulo-1.x-bin.tar.gz
     export DW_ZOOKEEPER_DIST_URI=http://some.apache.mirror/zookeeper/x.y/zookeeper-x.y.z.tar.gz
     export DW_WILDFLY_DIST_URI=file:///my/local/binaries/wildfly-10.x.tar.gz
     export DW_JAVA_DIST_URI=file:///my/local/binaries/jdk-8-update-x.tar.gz
     export DW_MAVEN_DIST_URI=file:///my/local/binaries/apache-maven-x.y.z.tar.gz

We just grabbed Apache Hadoop, Accumulo, Zookeeper, Wildfly, and Maven. Now if this seems like a lot, we can always bootstrap your environment with a bash script, this doesn't give you as much flexibility (unless you want to add it), but it is quicker, add your own shebang line at the top. Make sure you make this bash script executable, when done copy/pasting, run chmod u+x bootstrap_datawave.sh:

DW_DATAWAVE_SERVICE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

DW_DATAWAVE_SOURCE_DIR="$( cd "${DW_DATAWAVE_SERVICE_DIR}/../../../../.." && pwd )"

DW_DATAWAVE_ACCUMULO_AUTHS="${DW_DATAWAVE_ACCUMULO_AUTHS:-PUBLIC,PRIVATE,FOO,BAR,DEF,A,B,C,D,E,F,G,H,I,DW_USER,DW_SERV,DW_ADMIN,JBOSS_ADMIN}"

# Import DataWave Web test user configuration

source "${DW_DATAWAVE_SERVICE_DIR}/bootstrap-user.sh"

# Selected Maven profile for the DataWave build

DW_DATAWAVE_BUILD_PROFILE=${DW_DATAWAVE_BUILD_PROFILE:-dev}

# Maven command

DW_DATAWAVE_BUILD_COMMAND="${DW_DATAWAVE_BUILD_COMMAND:-mvn -P${DW_DATAWAVE_BUILD_PROFILE} -Ddeploy -Dtar -Ddist -Dservices -DskipTests clean install --builder smart -T1.0C}"

# Home of any temp data and *.properties file overrides for this instance of DataWave

DW_DATAWAVE_DATA_DIR="${DW_CLOUD_DATA}/datawave"

# Temp dir for persisting our dynamically-generated ${DW_DATAWAVE_BUILD_PROFILE}.properties file

DW_DATAWAVE_BUILD_PROPERTIES_DIR="${DW_DATAWAVE_DATA_DIR}/build-properties"

DW_DATAWAVE_BUILD_STATUS_LOG="${DW_DATAWAVE_BUILD_PROPERTIES_DIR}/build-progress.tmp"

DW_DATAWAVE_INGEST_TARBALL="*/datawave-${DW_DATAWAVE_BUILD_PROFILE}-*-dist.tar.gz"

DW_DATAWAVE_WEB_TARBALL="*/datawave-ws-deploy-application-*-${DW_DATAWAVE_BUILD_PROFILE}.tar.gz"

DW_DATAWAVE_KEYSTORE="${DW_DATAWAVE_KEYSTORE:-${DW_DATAWAVE_SOURCE_DIR}/web-services/deploy/application/src/main/wildfly/overlay/standalone/configuration/certificates/testServer.p12}"

DW_DATAWAVE_KEYSTORE_PASSWORD=${DW_DATAWAVE_KEYSTORE_PASSWORD:-ChangeIt}

DW_DATAWAVE_KEYSTORE_TYPE="${DW_DATAWAVE_KEYSTORE_TYPE:-PKCS12}"

DW_DATAWAVE_TRUSTSTORE="${DW_DATAWAVE_TRUSTSTORE:-${DW_DATAWAVE_SOURCE_DIR}/web-services/deploy/application/src/main/wildfly/overlay/standalone/configuration/certificates/ca.jks}"

DW_DATAWAVE_TRUSTSTORE_PASSWORD=${DW_DATAWAVE_TRUSTSTORE_PASSWORD:-ChangeIt}

DW_DATAWAVE_TRUSTSTORE_TYPE="${DW_DATAWAVE_TRUSTSTORE_TYPE:-JKS}"

# Accumulo shell script for initializing whatever we may need in Accumulo for DataWave

function createAccumuloShellInitScript() {
   # Allow user to inject their own script into the env...
   [ -n "${DW_ACCUMULO_SHELL_INIT_SCRIPT}" ] && return 0

   # Create script and add 'datawave' VFS context, if enabled...

   DW_ACCUMULO_SHELL_INIT_SCRIPT="
   createnamespace datawave
   createtable datawave.queryMetrics_m
   createtable datawave.queryMetrics_s
   setauths -s ${DW_DATAWAVE_ACCUMULO_AUTHS}"

   if [ "${DW_ACCUMULO_VFS_DATAWAVE_ENABLED}" != false ] ; then
      DW_ACCUMULO_SHELL_INIT_SCRIPT="${DW_ACCUMULO_SHELL_INIT_SCRIPT}
   config -s table.classpath.context=datawave"
   fi

   DW_ACCUMULO_SHELL_INIT_SCRIPT="${DW_ACCUMULO_SHELL_INIT_SCRIPT}
   quit
   "
}

function createBuildPropertiesDirectory() {
   if [ ! -d ${DW_DATAWAVE_BUILD_PROPERTIES_DIR} ] ; then
      if ! mkdir -p ${DW_DATAWAVE_BUILD_PROPERTIES_DIR} ; then
         error "Failed to create directory ${DW_DATAWAVE_BUILD_PROPERTIES_DIR}"
         return 1
      fi
   fi
   return 0
}

function setBuildPropertyOverrides() {

   # DataWave's build configs (*.properties) can be loaded from a variety of locations based on the 'read-properties'
   # Maven plugin configuration. Typically, the source-root/properties/*.properties files are loaded first to provide
   # default values, starting with 'default.properties', followed by '{selected-profile}.properties'. Finally,
   # ~/.m2/datawave/properties/{selected-profile}.properties is loaded, if it exists, allowing you to override
   # defaults as needed

   # With that in mind, the goal of this function is to generate a new '${DW_DATAWAVE_BUILD_PROFILE}.properties' file under
   # DW_DATAWAVE_BUILD_PROPERTIES_DIR and *symlinked* as ~/.m2/datawave/properties/${DW_DATAWAVE_BUILD_PROFILE}.properties,
   # to inject all the overrides that we need for successful deployment to source-root/contrib/datawave-quickstart/

   # If a file having the name '${DW_DATAWAVE_BUILD_PROFILE}.properties' already exists under ~/.m2/datawave/properties,
   # then it will be renamed automatically with a ".saved-by-quickstart-$(date)" suffix, and the symlink for the new
   # file will be created as required

   local BUILD_PROPERTIES_BASENAME=${DW_DATAWAVE_BUILD_PROFILE}.properties
   local BUILD_PROPERTIES_FILE=${DW_DATAWAVE_BUILD_PROPERTIES_DIR}/${BUILD_PROPERTIES_BASENAME}
   local BUILD_PROPERTIES_SYMLINK_DIR=${HOME}/.m2/datawave/properties
   local BUILD_PROPERTIES_SYMLINK=${BUILD_PROPERTIES_SYMLINK_DIR}/${BUILD_PROPERTIES_BASENAME}

   ! createBuildPropertiesDirectory && error "Failed to override properties!" && return 1

   # Create symlink directory if it doesn't exist
   [ ! -d ${BUILD_PROPERTIES_SYMLINK_DIR} ] \
       && ! mkdir -p ${BUILD_PROPERTIES_SYMLINK_DIR} \
       && error "Failed to create symlink directory ${BUILD_PROPERTIES_SYMLINK_DIR}" \
       && return 1

   # Copy existing source-root/properties/${DW_DATAWAVE_BUILD_PROFILE}.properties to our new $BUILD_PROPERTIES_FILE
   ! cp "${DW_DATAWAVE_SOURCE_DIR}/properties/${DW_DATAWAVE_BUILD_PROFILE}.properties" ${BUILD_PROPERTIES_FILE} \
       && error "Aborting property overrides! Failed to copy ${DW_DATAWAVE_BUILD_PROFILE}.properties" \
       && return 1

   # Apply overrides as needed by simply appending them to the end of the file...

   echo "#" >> ${BUILD_PROPERTIES_FILE}
   echo "######## Begin overrides for datawave-quickstart ########" >> ${BUILD_PROPERTIES_FILE}
   echo "#" >> ${BUILD_PROPERTIES_FILE}

   echo "WAREHOUSE_ACCUMULO_HOME=${ACCUMULO_HOME}" >> ${BUILD_PROPERTIES_FILE}
   echo "WAREHOUSE_INSTANCE_NAME=${DW_ACCUMULO_INSTANCE_NAME}" >> ${BUILD_PROPERTIES_FILE}
   echo "WAREHOUSE_JOBTRACKER_NODE=${DW_HADOOP_RESOURCE_MANAGER_ADDRESS}" >> ${BUILD_PROPERTIES_FILE}
   echo "INGEST_ACCUMULO_HOME=${ACCUMULO_HOME}" >> ${BUILD_PROPERTIES_FILE}
   echo "INGEST_INSTANCE_NAME=${DW_ACCUMULO_INSTANCE_NAME}" >> ${BUILD_PROPERTIES_FILE}
   echo "INGEST_JOBTRACKER_NODE=${DW_HADOOP_RESOURCE_MANAGER_ADDRESS}" >> ${BUILD_PROPERTIES_FILE}
   echo "BULK_INGEST_DATA_TYPES=${DW_DATAWAVE_INGEST_BULK_DATA_TYPES}" >> ${BUILD_PROPERTIES_FILE}
   echo "LIVE_INGEST_DATA_TYPES=${DW_DATAWAVE_INGEST_LIVE_DATA_TYPES}" >> ${BUILD_PROPERTIES_FILE}
   echo "PASSWORD=${DW_ACCUMULO_PASSWORD}" >> ${BUILD_PROPERTIES_FILE}
   echo "ZOOKEEPER_HOME=${ZOOKEEPER_HOME}" >> ${BUILD_PROPERTIES_FILE}
   echo "HADOOP_HOME=${HADOOP_HOME}" >> ${BUILD_PROPERTIES_FILE}
   echo "MAPRED_HOME=${HADOOP_HOME}" >> ${BUILD_PROPERTIES_FILE}
   echo "WAREHOUSE_HADOOP_CONF=${HADOOP_CONF_DIR}" >> ${BUILD_PROPERTIES_FILE}
   echo "INGEST_HADOOP_CONF=${HADOOP_CONF_DIR}" >> ${BUILD_PROPERTIES_FILE}
   echo "HDFS_BASE_DIR=${DW_DATAWAVE_INGEST_HDFS_BASEDIR}" >> ${BUILD_PROPERTIES_FILE}
   echo "MAPRED_INGEST_OPTS=${DW_DATAWAVE_MAPRED_INGEST_OPTS}" >> ${BUILD_PROPERTIES_FILE}
   echo "LOG_DIR=${DW_DATAWAVE_INGEST_LOG_DIR}" >> ${BUILD_PROPERTIES_FILE}
   echo "FLAG_DIR=${DW_DATAWAVE_INGEST_FLAGFILE_DIR}" >> ${BUILD_PROPERTIES_FILE}
   echo "FLAG_MAKER_CONFIG=${DW_DATAWAVE_INGEST_FLAGMAKER_CONFIGS}" >> ${BUILD_PROPERTIES_FILE}
   echo "BIN_DIR_FOR_FLAGS=${DW_DATAWAVE_INGEST_HOME}/bin" >> ${BUILD_PROPERTIES_FILE}
   echo "KEYSTORE=${DW_DATAWAVE_KEYSTORE}" >> ${BUILD_PROPERTIES_FILE}
   echo "KEYSTORE_TYPE=${DW_DATAWAVE_KEYSTORE_TYPE}" >> ${BUILD_PROPERTIES_FILE}
   echo "KEYSTORE_PASSWORD=${DW_DATAWAVE_KEYSTORE_PASSWORD}" >> ${BUILD_PROPERTIES_FILE}
   echo "TRUSTSTORE=${DW_DATAWAVE_TRUSTSTORE}" >> ${BUILD_PROPERTIES_FILE}
   echo "TRUSTSTORE_PASSWORD=${DW_DATAWAVE_TRUSTSTORE_PASSWORD}" >> ${BUILD_PROPERTIES_FILE}
   echo "TRUSTSTORE_TYPE=${DW_DATAWAVE_TRUSTSTORE_TYPE}" >> ${BUILD_PROPERTIES_FILE}
   echo "FLAG_METRICS_DIR=${DW_DATAWAVE_INGEST_FLAGMETRICS_DIR}" >> ${BUILD_PROPERTIES_FILE}
   echo "accumulo.instance.name=${DW_ACCUMULO_INSTANCE_NAME}" >> ${BUILD_PROPERTIES_FILE}
   echo "accumulo.user.password=${DW_ACCUMULO_PASSWORD}" >> ${BUILD_PROPERTIES_FILE}

   echo "cached.results.hdfs.uri=${DW_HADOOP_DFS_URI}" >> ${BUILD_PROPERTIES_FILE}
   echo "type.metadata.hdfs.uri=${DW_HADOOP_DFS_URI}" >> ${BUILD_PROPERTIES_FILE}
   echo "mapReduce.hdfs.uri=${DW_HADOOP_DFS_URI}" >> ${BUILD_PROPERTIES_FILE}
   echo "bulkResults.hdfs.uri=${DW_HADOOP_DFS_URI}" >> ${BUILD_PROPERTIES_FILE}
   echo "jboss.log.hdfs.uri=${DW_HADOOP_DFS_URI}" >> ${BUILD_PROPERTIES_FILE}

   echo "lock.file.dir=${DW_DATAWAVE_INGEST_LOCKFILE_DIR}" >> ${BUILD_PROPERTIES_FILE}
   echo "server.keystore.password=${DW_ACCUMULO_PASSWORD}" >> ${BUILD_PROPERTIES_FILE}
   echo "mysql.user.password=${DW_ACCUMULO_PASSWORD}" >> ${BUILD_PROPERTIES_FILE}
   echo "jboss.jmx.password=${DW_ACCUMULO_PASSWORD}" >> ${BUILD_PROPERTIES_FILE}
   echo "jboss.managed.executor.service.default.max.threads=${DW_WILDFLY_EE_DEFAULT_MAX_THREADS:-48}" >> ${BUILD_PROPERTIES_FILE}
   echo "hornetq.cluster.password=${DW_ACCUMULO_PASSWORD}" >> ${BUILD_PROPERTIES_FILE}
   echo "hornetq.system.password=${DW_ACCUMULO_PASSWORD}" >> ${BUILD_PROPERTIES_FILE}
   echo "mapReduce.job.tracker=${DW_HADOOP_RESOURCE_MANAGER_ADDRESS}" >> ${BUILD_PROPERTIES_FILE}
   echo "bulkResults.job.tracker=${DW_HADOOP_RESOURCE_MANAGER_ADDRESS}" >> ${BUILD_PROPERTIES_FILE}
   echo "EVENT_DISCARD_INTERVAL=0" >> ${BUILD_PROPERTIES_FILE}
   echo "ingest.data.types=${DW_DATAWAVE_INGEST_LIVE_DATA_TYPES},${DW_DATAWAVE_INGEST_BULK_DATA_TYPES}" >> ${BUILD_PROPERTIES_FILE}
   echo "JOB_CACHE_REPLICATION=1" >> ${BUILD_PROPERTIES_FILE}
   echo "EDGE_DEFINITION_FILE=${DW_DATAWAVE_INGEST_EDGE_DEFINITIONS}" >> ${BUILD_PROPERTIES_FILE}
   echo "DATAWAVE_INGEST_HOME=${DW_DATAWAVE_INGEST_HOME}" >> ${BUILD_PROPERTIES_FILE}
   echo "PASSWORD_INGEST_ENV=${DW_DATAWAVE_INGEST_PASSWD_FILE}" >> ${BUILD_PROPERTIES_FILE}
   echo "hdfs.site.config.urls=file://${HADOOP_CONF_DIR}/core-site.xml,file://${HADOOP_CONF_DIR}/hdfs-site.xml" >> ${BUILD_PROPERTIES_FILE}
   echo "table.shard.numShardsPerDay=${DW_DATAWAVE_INGEST_NUM_SHARDS}" >> ${BUILD_PROPERTIES_FILE}

   generateTestDatawaveUserServiceConfig

   # Apply DW_JAVA_HOME_OVERRIDE, if needed...
   # We can override the JAVA_HOME location for the DataWave deployment, if necessary. E.g., if we're deploying
   # to a Docker container or other, where our current JAVA_HOME isn't applicable

   if [ -n "${DW_JAVA_HOME_OVERRIDE}" ] ; then
      echo "JAVA_HOME=${DW_JAVA_HOME_OVERRIDE}" >> ${BUILD_PROPERTIES_FILE}
   else
      echo "JAVA_HOME=${JAVA_HOME}" >> ${BUILD_PROPERTIES_FILE}
   fi

   # Apply DW_ROOT_DIRECTORY_OVERRIDE, if needed...
   # We can override any instances of DW_DATAWAVE_SOURCE_DIR within the build config in order to relocate
   # the deployment, if necessary. E.g., used when building the datawave-quickstart Docker image to reorient
   # the deployment under /opt/datawave/ within the container

   if [ -n "${DW_ROOT_DIRECTORY_OVERRIDE}" ] ; then
      sed -i "s~${DW_DATAWAVE_SOURCE_DIR}~${DW_ROOT_DIRECTORY_OVERRIDE}~g" ${BUILD_PROPERTIES_FILE}
   fi

   # Create the symlink under ~/.m2/datawave/properties

   setBuildPropertiesSymlink || return 1
}

function setBuildPropertiesSymlink() {
   # Replace any existing ~/.m2/datawave/properties/${BUILD_PROPERTIES_BASENAME} file/symlink with
   # a symlink to our new ${BUILD_PROPERTIES_FILE}

   if [[ -f ${BUILD_PROPERTIES_SYMLINK} || -L ${BUILD_PROPERTIES_SYMLINK} ]] ; then
       if [ -L ${BUILD_PROPERTIES_SYMLINK} ] ; then
           info "Unlinking existing symbolic link: ${BUILD_PROPERTIES_SYMLINK}"
           if ! unlink "${BUILD_PROPERTIES_SYMLINK}" ; then
               warn "Failed to unlink $( readlink ${BUILD_PROPERTIES_SYMLINK} ) from ${BUILD_PROPERTIES_SYMLINK_DIR}"
           fi
       else
           local backupFile="${BUILD_PROPERTIES_SYMLINK}.saved-by-quickstart.$(date +%Y-%m-%d-%H%M%S)"
           info "Backing up your existing ~/.m2/**/${BUILD_PROPERTIES_BASENAME} file to ~/.m2/**/$( basename ${backupFile} )"
           if ! mv "${BUILD_PROPERTIES_SYMLINK}" "${backupFile}" ; then
               error "Failed to backup ${BUILD_PROPERTIES_SYMLINK}. Aborting properties file override. Please fix me!!"
               return 1
           fi
       fi
   fi

   if ln -s "${BUILD_PROPERTIES_FILE}" "${BUILD_PROPERTIES_SYMLINK}" ; then
       info "Override for ${BUILD_PROPERTIES_BASENAME} successful"
   else
       error "Override for ${BUILD_PROPERTIES_BASENAME} failed"
       return 1
   fi
}

function datawaveBuildSucceeded() {
   local success=$( tail -n 7 "$DW_DATAWAVE_BUILD_STATUS_LOG" | grep "BUILD SUCCESS" )
   if [ -z "${success}" ] ; then
       return 1
   fi
   return 0
}

function buildDataWave() {

   if ! mavenIsInstalled ; then
      ! mavenInstall && error "Maven install failed. Please correct" && return 1
   fi

   [[ "$1" == "--verbose" ]] && local verbose=true

   ! setBuildPropertyOverrides && error "Aborting DataWave build" && return 1

   [ -f "${DW_DATAWAVE_BUILD_STATUS_LOG}" ] && rm -f "$DW_DATAWAVE_BUILD_STATUS_LOG"

   info "DataWave build in progress: '${DW_DATAWAVE_BUILD_COMMAND}'"
   info "Build status log: $DW_DATAWAVE_BUILD_STATUS_LOG"
   if [ "${verbose}" == true ] ; then
       ( cd "${DW_DATAWAVE_SOURCE_DIR}" && eval "${DW_DATAWAVE_BUILD_COMMAND}" 2>&1 | tee ${DW_DATAWAVE_BUILD_STATUS_LOG} )
   else
       ( cd "${DW_DATAWAVE_SOURCE_DIR}" && eval "${DW_DATAWAVE_BUILD_COMMAND}" &> ${DW_DATAWAVE_BUILD_STATUS_LOG} )
   fi

   if ! datawaveBuildSucceeded ; then
       error "The build has FAILED! See $DW_DATAWAVE_BUILD_STATUS_LOG for details"
       return 1
   fi

   info "DataWave build successful"
   return 0
}

function getDataWaveTarball() {
   # Looks for a DataWave tarball matching the specified pattern and, if found, sets the global 'tarball'
   # variable to its basename for the caller as expected.

   # If no tarball is found matching the specified pattern, then the DataWave build is kicked off

   local tarballPattern="${1}"
   tarball=""

   # Check if the tarball already exists in the plugin directory.
   local tarballPath="$( find "${DW_DATAWAVE_SERVICE_DIR}" -path "${tarballPattern}" -type f )"
   if [ -f "${tarballPath}" ]; then
      tarball="$( basename "${tarballPath}" )"
      return 0;
   fi

   ! buildDataWave --verbose && error "Please correct this issue before continuing" && return 1

   # Build succeeded. Set global 'tarball' variable for the specified pattern and copy all tarballs into place

   tarballPath="$( find "${DW_DATAWAVE_SOURCE_DIR}" -path "${tarballPattern}" -type f | tail -1 )"
   [ -z "${tarballPath}" ] && error "Failed to find '${tarballPattern}' tar file after build" && return 1

   tarball="$( basename "${tarballPath}" )"

   # Current caller (ie, either bootstrap-web.sh or bootstrap-ingest.sh) only cares about current $tarball,
   # but go ahead and copy both tarballs into datawave service dir to satisfy next caller as well

   ! copyDataWaveTarball "${DW_DATAWAVE_INGEST_TARBALL}" && error "Failed to copy DataWave Ingest tarball" && return 1
   ! copyDataWaveTarball "${DW_DATAWAVE_WEB_TARBALL}" && error "Failed to copy DataWave Web tarball" && return 1

   return 0
}

function copyDataWaveTarball() {
   local pattern="${1}"
   local dwTarball="$( find "${DW_DATAWAVE_SOURCE_DIR}" -path "${pattern}" -type f | tail -1 )";
   if [ -n "${dwTarball}" ] ; then
       ! cp "${dwTarball}" "${DW_DATAWAVE_SERVICE_DIR}" && error "Failed to copy '${dwTarball}'" && return 1
   else
       error "No tar file found matching '${pattern}'"
       return 1
   fi
   return 0
}

# Bootstrap DW ingest and webservice components as needed

source "${DW_DATAWAVE_SERVICE_DIR}/bootstrap-ingest.sh"
source "${DW_DATAWAVE_SERVICE_DIR}/bootstrap-web.sh"

function datawaveIsRunning() {
    datawaveIngestIsRunning && return 0
    datawaveWebIsRunning && return 0
    return 1
}

function datawaveStart() {
    datawaveIngestStart
    datawaveWebStart
}

function datawaveStop() {
    datawaveIngestStop
    datawaveWebStop
}

function datawaveStatus() {
    datawaveIngestStatus
    datawaveWebStatus
}

function datawaveIsInstalled() {
    datawaveIngestIsInstalled && return 0
    datawaveWebIsInstalled && return 0
    return 1
}

function datawaveUninstall() {
   datawaveIngestUninstall
   datawaveWebUninstall

   [[ "${1}" == "${DW_UNINSTALL_RM_BINARIES_FLAG_LONG}" || "${1}" == "${DW_UNINSTALL_RM_BINARIES_FLAG_SHORT}" ]] && rm -f "${DW_DATAWAVE_SERVICE_DIR}"/*.tar.gz
}

function datawaveInstall() {
   datawaveIngestInstall
   datawaveWebInstall
}

function datawavePrintenv() {
   echo
   echo "DataWave Environment"
   echo
   ( set -o posix ; set ) | grep -E "DATAWAVE_|WILDFLY|JBOSS"
   echo
}

function datawavePidList() {
   datawaveIngestIsRunning
   datawaveWebIsRunning
   if [[ -n "${DW_DATAWAVE_WEB_PID_LIST}" || -n "${DW_DATAWAVE_INGEST_PID_LIST}" ]] ; then
      echo "${DW_DATAWAVE_WEB_PID_LIST} ${DW_DATAWAVE_INGEST_PID_LIST}"
   fi
}

function datawaveBuildDeploy() {
   datawaveIsRunning && info "Stopping all DataWave services" && datawaveStop
   datawaveIsInstalled && info "Uninstalling DataWave" && datawaveUninstall --remove-binaries

   resetQuickstartEnvironment
   export DW_REDEPLOY_IN_PROGRESS=true
   datawaveInstall
   export DW_REDEPLOY_IN_PROGRESS=false
}

function datawaveBuild() {
   info "Building DataWave"
   rm -f "${DW_DATAWAVE_SERVICE_DIR}"/datawave*.tar.gz
   resetQuickstartEnvironment
}

Let's try Datawave out finally

So let's find some Wikipedia data based on page title, with a --verbose flag to see the cURL command in action with Datawave:

datawaveQuery --query "PAGE_TITLE:AccessibleComputing OR PAGE_TITLE:Anarchism" --verbose

Next let's grab TV show data from API.TVMAZE.COM(graph edge queries):

datawaveQuery --logic EdgeQuery --syntax JEXL --query "SOURCE == 'kevin bacon' && TYPE == 'TV_COSTARS'" --pagesize 30

Then, let's do our graph edge queries:

datawaveQuery --logic EdgeQuery --syntax JEXL --query "SOURCE == 'william shatner' && TYPE == 'TV_CHARACTERS'"

Let's try doing one more EdgeQuery:

datawaveQuery --logic EdgeQuery --syntax JEXL --query "SOURCE == 'westworld' && TYPE == 'TV_SHOW_CAST'"  --pagesize 20

That was cool right? To formulate some of your own graph/edge queries run the following:

datawaveQuery --help

This will give you a broader scope on how Edges work.

Edges

One of the things I thought was, "EdgeQueryLogic in Datawave needs a better date range filter".

So to be clear, currently as it stands EdgeQueryLogic currently uses a column qualifier range filter to skip keys that are not within a specified date range. We need to be able to incorporate seeks such that the filter will skip over entries not within the date range. This is expected to be significantly faster when there are large gaps in sequence of edges for a source value that are not within the date range.

As you can see, this can come up short. An example subroutine that would help with Edge queries, would look like this:

private boolean seekToStartKey(Key topKey, String date) throws IOException
   boolean seeked = false;

  if (startDate != null && date.compareTo(startDate) < 0) {
    // Date is before start date.  Seek to same key with date set to start date
    Key newKey = EdgeKeyUtil.getSeekToFutureKey(topKey, startDate);
    PartialKey pk = EdgeKeyUtil.getSeekToFuturePartialKey();

    fastReseek(newKey, pk);
  } else if (endDate != null & date.compareTo(endDate) > 0) {
    PartialKey part = EdgeKeyUtil.getSeekToNextKey();
    fastReseek(topKey.followingKey(part), part);
    seeked = true;
  }

  return seeked;
}

This now is getting into the core Datawave architecture. I've now helped you install it, run some queries on Wikipedia, and use an API for TV shows to demonstrate Edge querying.

Datawave Architecture

As you can imagine with things like ZooKeeper, Hadoop of course you'll be using things like MapReduce and Pig, below is the more-or-less the architecture of Datawave:

With this architecture, you might be thinking -- "What about run away queries?" Good question.

Your query can get into a state where the QueryIterator will not terminate. This is an edge case where yielding occurs, and the FinalDocument is returned as a result. This will get us into an infinite loop where the upon rebuilding the iterator, it will yield again and again.

Two things are required:

The yield callback is checked in the FinalDocumentIterator and the yield is passed through appropriately.
The underlying iterator is no longer checked once the final document is returned.

This is just mainly another NB, this can happen, and if it does there's ways to do command jujitsu to get out of it.

Conclusion

We used EdgeQuerying got info from Wikipedia and gave it conditionals, and used some API's via REST to do some cool things, these are just some of the VERY few things Datawave can do, and that I've personally have done myself. I may do a part 2 series on this.

Montana Mendy

Paralleling Code Coverage using Coveralls and Travis

Montana Mendy — Wed, 23 Feb 2022 15:36:25 +0000

Hey developer friends,

A short one today -- but an important one.

Let's say you have multiple environments you want to have coverage, first let's enable Coveralls for your repo, so if you haven't done so yet -- head over to coveralls.io and Auth the Coveralls app to up your GitHub account, and then sync your repos with Coveralls.

You'll want to have a .travis.yml file that looks similar to mine:

sudo: false
language: node_js
node_js:
  - "node"
env:
  global:
    - COVERALLS_PARALLEL=true
jobs:
  include:
    - script: COVERALLS_FLAG_NAME=test-1 make test-coveralls-1
    - script: COVERALLS_FLAG_NAME=test-2 make test-coveralls-2
notifications:
  webhooks: https://coveralls.io/webhook

You'll see we are running multiple COVERALLS_FLAG_NAME, this is using the Coveralls webhook to parallelize your coverage. If at first you don't see Coveralls covering your code, you may want to add your env var from Coveralls to Travis CI and name it something like COVERALLS_TOKEN.

Here's my repo so you can follow step by step! https://github.com/Montana/coveralls-node-travis

Creating a base build of Snapcraft in Travis CI

Montana Mendy — Thu, 17 Feb 2022 23:04:16 +0000

Sometimes you want to see how things work under the hood, and this is why I put together quickly how I used Snapcraft with Travis CI to get a LXD up and running, and download VLC and then get info on that version of VLC all inside Travis CI.

First let's start with out snapchat.yaml. This is how mine looks:

name: montana
version: v0.58.10-1.dev
summary: Im just testing out snapd with Travis
description: |
 My name is Montana Mendy and I'm a Software Engineer at Travis CI, I'm trying new things with Travis CI and the Brave Browser.
grade: stable
confinement: strict

architectures:
  - build-on: amd64

parts:
  brave:
    plugin: dump
    source: https://github.com/brave/brave-browser/releases/download/v0.58.10/brave-browser-dev_0.58.10_amd64.deb
    source-type: deb
    # Correct path to icon.
    override-pull: |
      snapcraftctl pull
      rm -rf etc/cron.daily/ 
      rm -rf usr/bin/brave-browser-dev
      chmod 4555 opt/brave.com/brave-dev/brave-sandbox
      unlink opt/brave.com/brave-dev/brave-browser
      sed -i 's|Icon=brave-browser|Icon=/opt/brave.com/brave-dev/product_logo_128\.png|g' usr/share/applications/brave-browser-dev.desktop
    after:
      - desktop-gtk3
    stage-packages:
      - gir1.2-gnomekeyring-1.0
      - libasound2
      - libgconf-2-4
      - libgl1-mesa-glx
      - libglu1-mesa
      - libgnome-keyring0
      - libcap2
      - libgcrypt20
      - libnotify4
      - libnspr4
      - libnss3
      - libpulse0
      - libxtst6
      - libxss1

apps:
  brave:
    command: bin/desktop-launch $SNAP/opt/brave.com/brave-dev/brave-browser-dev
    desktop: usr/share/applications/brave-browser-dev.desktop
    # Correct the TMPDIR path for Chromium Framework/Electron to
    # ensure libappindicator has readable resources.
    environment:
      TMPDIR: $XDG_RUNTIME_DIR
    plugs:
      - alsa
      - avahi-observe
      - browser-sandbox
      - camera
      - cups-control
      - desktop
      - gsettings
      - home
      - mount-observe
      - network
      - opengl
      - password-manager-service
      - pulseaudio
      - remove-media
      - screen-inhibit-control
      - unity7
      - upower-observe
      - x11

plugs:
  browser-sandbox:
    interface: browser-support
    allow-sandbox: true
  gtk-3-themes:
    interface: content
    target: $SNAP/data-dir/themes
    default-provider: gtk-common-themes
  icon-themes:
    interface: content
    target: $SNAP/data-dir/icons
    default-provider: gtk-common-themes
  sound-themes:
    interface: content
    target: $SNAP/data-dir/sounds
    default-provider: gtk-common-themes

This is for the Brave Browser, but I've added things like pulseaudio to the Snap plugs. So let's push that into a directory called snap. Now let's head back to your root directory and make your .travis.yml, and this is what I coded out for my .travis.yml:

language: shell
dist: xenial
os: linux
group: edge 

env:
  global:
    - LC_ALL: C.UTF-8
    - LANG: C.UTF-8
    - SNAPCRAFT_ENABLE_SILENT_REPORT: y
    - SNAPCRAFT_BUILD_INFO: 1
    - SNAPCRAFT_BUILD_ENVIRONMENT: 'lxd'

addons:
  snaps:
    - name: snapcraft
      channel: stable
      confinement: classic
    - name: lxd
      channel: stable

script:
  - sudo usermod --append --groups lxd $USER
  - sudo /snap/bin/lxd.migrate -yes
  - sudo /snap/bin/lxd waitready
  - sudo /snap/bin/lxd init --auto
  - sudo apt install snapd
  - sudo snap install hello-world
  - sudo snap install nethack
  - snap version 
  - snap list
  - snap connections nethack
  - snap services lxd
  - sudo snap install --channel=edge vlc
  - which vlc 
  - snapcraft extensions
  - snap connections vlc
  - sudo systemctl enable --now snapd.socket
  - sudo journalctl -xeb | grep -i snap

So let's trigger a build by using powerline in Vim, and if your build is successful it should look something like this:

Somethings you'll want to keep in mind with Snapcraft Multipass has 2 CPU's assigned to it, this can be variable depending on your settings:

$ export SNAPCRAFT_BUILD_ENVIRONMENT_CPU=8 
$ export SNAPCRAFT_BUILD_ENVIRONMENT_MEMORY=16G

You can put those in your script: hook in your .travis.yml file.

Migrating between bases

A base snap is a special kind of snap that provides a run-time environment with a minimal set of libraries that are common to most Linux distributions. Think of it as a minimal Gentoo install. At the simplest level, in your snapcraft.yaml file you can do:

- base: core18
+ base: core20

As you can see, only one of the base keywords need to be updated.

Enforcing Snap Policies

One thing you may want to consider when making this, is enforcing some security policies, the way you do that is, yes you guessed it another .yaml file, so let's call this one snap.yaml and it would look a little like this:

name: montana
version: 1.0
apps:
  bar:
    command: mendy
  baz:
    command: dig
    daemon: simple
    plugs: [network]

If you run this with your Travis CI build and don't like what you see you can always set a snap disconnect conditional so you can disconnect. Don't forget you can add a Cron job directly from Snapcraft by adding this line in your .travis.yml:

sudo snap set system refresh.timer=fri5,23:00-01:00

That's now a cron that will refresh Snap at a time of my choosing.

Conclusion

In some sense setting up Snapcraft for Travis CI was really fun, and got to see hands on how LXD containers are built from the ground up, setting your own policies, enforcing them and even picking your own plugins so you have a great foundation to build on when you're ready.

Here's my repository: https://github.com/Montana/travis-snap-lxd

Start Maxwell with Namespaced Topic Kafka Producer (Look for Idle Listeners in Kafka)

Montana Mendy — Thu, 02 Dec 2021 17:01:46 +0000

After running through the prerequisites, you will have:

A AWS Aurora instance
A Maxwell image named osheroff/maxwell
AKafka service named kafka, listening on kafka:9092

Start Maxwell with Namespaced Topic Kafka Producer
This is a slight variation of the prerequisite, AWS Aurora to Maxwell Kafka Producer.

In the prerequisite we ran Maxwell with the default Kafka Producer configuration which will produce messages on the Maxwell topic.

You'll want to get the number of messages in a Maxwell topic, you can run: bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic mytopic --time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'

In this example we are overriding the MAXWELL_OPTIONS environment variable and specifying a dynamic topic name, so that Maxwell will route messages from each table to topics by the same name, namespaced by the database name.

docker run -it --rm \
    --env MYSQL_USERNAME=AURORA_USERNAME \
    --env MYSQL_PASSWORD=AURORA_PASSWORD \
    --env MYSQL_HOST=AURORA_HOST \
    --link kafka \
    --env KAFKA_HOST=kafka \
    --env KAFKA_PORT=9092 \
    --env MAXWELL_OPTIONS="--kafka_topic=maxwell_%{database}_%{table}
    --name maxwell \
    osheroff/maxwell

Here's a graphical way of looking at things when it comes to Consumers. Now let's get back to Maxwell.

This will be the output of Maxwell:

17:44:34,901 INFO  ProducerConfig - ProducerConfig values: 
    request.timeout.ms = 30000
    retry.backoff.ms = 100
    buffer.memory = 33554432
    ssl.truststore.password = null
    batch.size = 16384
    ssl.keymanager.algorithm = SunX509
    receive.buffer.bytes = 32768
    ssl.cipher.suites = null
    ssl.key.password = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    ssl.provider = null
    sasl.kerberos.service.name = null
    max.in.flight.requests.per.connection = 5
    sasl.kerberos.ticket.renew.window.factor = 0.8
    bootstrap.servers = [kafka:9092]
    client.id = 
    max.request.size = 1048576
    acks = 1
    linger.ms = 0
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
    metadata.fetch.timeout.ms = 60000
    ssl.endpoint.identification.algorithm = null
    ssl.keystore.location = null
    value.serializer = class org.apache.kafka.common.serialization.StringSerializer
    ssl.truststore.location = null
    ssl.keystore.password = null
    key.serializer = class org.apache.kafka.common.serialization.StringSerializer
    block.on.buffer.full = false
    metrics.sample.window.ms = 30000
    metadata.max.age.ms = 300000
    security.protocol = PLAINTEXT
    ssl.protocol = TLS
    sasl.kerberos.min.time.before.relogin = 60000
    timeout.ms = 30000
    connections.max.idle.ms = 540000
    ssl.trustmanager.algorithm = PKIX
    metric.reporters = []
    compression.type = none
    ssl.truststore.type = JKS
    max.block.ms = 60000
    retries = 0
    send.buffer.bytes = 131072
    partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
    reconnect.backoff.ms = 50
    metrics.num.samples = 2
    ssl.keystore.type = JKS

17:44:34,952 INFO  AppInfoParser - Kafka version : 0.9.0.1
17:44:34,952 INFO  AppInfoParser - Kafka commitId : 23c69d62a0cabf06
17:44:35,012 INFO  Maxwell - Maxwell v1.7.0 is booting (MaxwellKafkaProducer), starting at BinlogPosition[mysql-bin-changelog.000002:84337]
17:44:35,680 INFO  MysqlSavedSchema - Restoring schema id 1 (last modified at BinlogPosition[mysql-bin-changelog.000002:3521])
17:44:38,991 INFO  OpenReplicator - starting replication at mysql-bin-changelog.000002:84337

The process is now waiting for new data events and looking for idle Kafka listeners.

Start a consumer (in another terminal window). This command will start an unnamed instance of Spotify/Kafka linked to the Kafka service, start a consumer, display existing messages from the Maxwell topic, and wait for new messages until you quit (which destroys the container):

docker run -it --rm --link kafka spotify/kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic maxwell_{AURORA_DATABASE}_{AURORA_TABLE} --from-beginning

Connect to the AWS Aurora instance, insert some records, and update some records. Data events from Maxwell will be printed in the Consumer terminal window:

{"database":"AURORA_DATABASE","table":"AURORA_TABLE","type":"update","ts":1484606003,"xid":1655558,"commit":true,"data":{"id":4,"first_name":"Mendy","last_name":"Montana"},"old":{"first_name":"Montana"}}
{"database":"AURORA_DATABASE","table":"AURORA_TABLE","type":"update","ts":1484606435,"xid":1658343,"commit":true,"data":{"id":4,"first_name":"Montana","last_name":"Mendy"},"old":{"first_name":"Tim"}}
{"database":"AURORA_DATABASE","table":"AURORA_TABLE","type":"update","ts":1484606451,"xid":1658455,"commit":true,"data":{"id":4,"first_name":"Tim","last_name":"Mendy"},"old":{"first_name":"Montana"}}

Travis CI Pipelines: 2 Approaches to Source Control Branching

Montana Mendy — Wed, 17 Feb 2021 13:22:23 +0000

Feature branching is a game-changing aspect of modern software development. Being able to have a developer implement a new feature in a body of code in a safe, independent, isolated manner using Git branching is an overall positive approach to the way companies make software.

Over the years, two techniques of feature branching have emerged. One is what I call branching from repo. The other is branching from fork. Let’s explore each technique, as well as their benefits and tradeoffs.

Branching from repo

Branching from repo is a pattern in which feature branches are created off an existing branch in a business’s repository and then assigned to or assumed by a particular developer for implementation.

Figure 1: The branching from repo pattern is used to implement feature branches created from a particular branch in a business’s repository.

The branching from repo pattern is confined to creating feature branches that are within the boundaries of a particular business’s repository. Thus, only personnel or services that have the proper permissions for working that repository can create a particular feature branch.

The important thing to understand about branching from repo is that all the work that goes with creating the feature branch, developing the feature’s code, and then issuing the pull request for the completed code is done from within the boundary of the given repository. Those people on the “outside” have no ability to contribute changes.

Branching from fork

In the branching from fork pattern, a developer interested in adding to and improving upon code in a given repository forks the code into their own repository. Then, the developer creates branches against the forked code.

Figure 2: Developers use the branching from fork pattern to fork code from the business’s repository into the developer’s external repository to then create a feature branch.

Once work on a given branch is complete, the developer makes a pull request to merge the code back into the originating repository.

Figure 3: GitHub supports pull requests for merging code between repositories in separate accounts.

The important thing to understand about the branching from fork pattern is that it allows any developer to make improvements to code in an external repository in a controlled manner. All that’s required is view access to a repository of interest and then the cloning of the code.

Once the code is cloned, the developer is free to work on improvements according to their own software development process. The original repository is independent of the forked code; the only intersection between the two is when the pull request is made from the forked repository back to the originating repository.

Advantages and tradeoffs

Both patterns have advantages and trade-offs. Let’s examine them.

Branching from repo is well suited to private governance

Branching from repo is well suited for companies that want to keep their software development process private and subject only to their own development and project management practices.

Typically, a feature branch is created to accommodate a request that exists in a company’s backlog of work. In some cases, a feature branch can be created ad hoc, but usually, they are created by a governing authority within the company, such as a project manager or tech lead. Either the feature branch is created directly or as an ancillary action taken by automation within a project management platform such as Jira. As such, branching from repo allows project managers and tech leads to exert a lot of control over who can work on code. Also, the governing authority can determine the status of the feature branch by viewing the code as well as the branch’s commit history.

The trade-off is that branching from repo requires that those working on the code have a known relationship to the repository from within the source control management service hosting the code. For example, if the code is hosted on GitHub, a developer must have permission to make commits to the feature branch. Typically, this means inviting the developer to be a direct contributor to the repository or a member of an organization associated with the project. While this degree of formality provides control, it also excludes the general public from working on the code in an ad hoc manner, so there is less opportunity to benefit from the informal interest of others. Fortunately, branching from fork addresses this shortcoming.

Branching from fork is well suited to open-source development

Branching from fork is well suited for volunteer-based open-source development. Branching from repo is pretty much a “We’re assigning you some code to work” approach. On the other hand, branching from fork says, “Here’s some code. Feel free to work on it, but you need to figure out what you want to work on to make it better.”

In order for a developer to improve code using branching from fork, they need to identify possible improvements. This can take many forms. For instance, a developer can find problems to solve by going over the list of open issues in the source control management service that’s hosting the code. Once an issue is identified, the developer can address it according to their own development style and timeframe.

In addition to addressing known issues, a developer can implement, in an unsolicited manner, a feature they would like to see in the code. There’s no need to ask for permission. All that needs to be done is to implement the feature and then submit a pull request back to the original project. At that point, it’s up to the project’s maintainers to use the improvement. If the unsolicited feature is not accepted, it still exists in the fork and is available to interested parties there.

One drawback of branching by fork is that making contributions can require a good deal of work. Regardless of whether the developer is working on a known issue, implementing an existing feature request, or offering an unsolicited improvement, they will still need to learn the code. That learning curve can be significant, particularly if the developer has limited access to others who are willing to help. Sometimes the help required to fill in the gaps might be available, but many times it’s not. If the documentation is poor or the code is cryptic, figuring out how to make a meaningful improvement can take a lot of time. As those who have worked with volunteers can attest, the best volunteer experience is the one that’s the easiest. The more difficult the undertaking, the greater the risk of losing the volunteer’s sense of investment.

Conversely, a developer working on a feature request according to the branching from repo pattern will usually be a member of the business or organization that owns the repository. Thus, they will usually enjoy the support of other members who have experience working with the code. There’s both formal and tribal knowledge that’s available, so things are easier.

Another drawback of branching from fork is that since anybody can make a contribution, anybody will. This means that the code submitted via a pull request will vary in quality: Some of it will be quite good, but some of it will be quite bad. This creates more work for those who actually have to review pull requests in order to make sure that a submission meets quality standards. While a certain degree of quality control can be accomplished using automation, there are still aspects of the pull request that will require human inspection, particularly if the code is complex.

In short, branching from fork allows more people to work on code in their own manner, but making meaningful contributions is harder for all parties involved.

Putting it all together

The beauty of being able to branch source code is that it allows developers to work on code in a safe, controllable manner. Modern software development depends on branching.

The beauty of forking is that it provides a way for independent parties to work on an entire repository hosted in a source control management platform, such as GitHub or BitBucket, in an independent manner without affecting the original repository. Combining branching and forking creates a new dimension in software development.

Whether you use the branching from repo technique or the branching from fork technique, the important thing is that both methods provide developers a significant degree of independence while ensuring that the code that eventually makes its way into production will be the highest quality possible.

Configuring Travis CI to Run a Deno Project

Montana Mendy — Wed, 17 Feb 2021 12:52:42 +0000

Nothing lasts forever in the world of ephemeral computing. It’s the nature of the beast. Today, more companies are maximizing their IT budgets by practicing the principles of infrastructure as code (IaC). They’re creating and destroying virtual assets on demand in order to meet the needs of the moment.

Getting started

I am a big fan of Deno. I am also a big fan of Travis CI. What’s not to like? Deno is a powerful new programming framework that picks up where Node.js left off. Travis CI is a CI/CD platform that integrates easily with the projects stored in my GitHub repo. They’re both very cool.

But there’s a problem: Travis CI does not support Deno out of the box. Let's see what we can do.

Deno is still in its infancy, so it makes sense that it is not quite yet on the Travis CI radar. Still, no biggie. Travis CI is flexible enough to allow me to build support for Deno right into the CI/CD workflow that kicks off when my Deno project runs on Travis CI. In fact, getting Deno to run on Travis CI was pretty easy, and the results are very useful, so allow me to share.

First I’m going to cover the overall process for deploying to Travis CI and running a workflow defined in a .travis.yml file. Then I’ll show you the custom .travis.yml file I wrote that has Travis CI install Deno and run the unit tests for my Deno project.

Travis CI 101

Travis CI runs a CI/CD process right out of the box as part of its signup process. The first time you come to travic-ci.org, you’ll be asked to log in to Travis CI using your GitHub or BitBucket credentials. (The Bitbucket signup is in beta.)

The signup process asks you to declare repositories on the source code management (SCM) service, into which a webhook will be installed. Once a webhook is installed on the given repository, Travis CI will receive notifications about events that happen in that repository. For example, when a developer commits code into a repository that has a webhook installed, Travis CI will be notified about the commit.

Travis CI will clone the contents of that repo into a virtual machine instance running in the Travis CI domain. Once the runner VM has the contents of the cloned repository, it will look for a special file named travis.yml that contains instructions for provisioning the VM and about tasks to perform once the runner VM is provisioned. Figure 1 below describes the process.

Figure 1: The Travis CI Workflow

As you can see in callout 1 above, the first step in the workflow process is that a developer commits updated code to the repository of interest in the SCM service running in the cloud. A webhook in the repository “sees” the commit and emits an event message that is captured by Travis CI, as shown at callout 2. Travis CI clones the repository’s code into a VM in its domain and then looks for the .travis.yml file in the cloned files. The .travis.yml file is identified, and the configuration and instructions in .travis.yml are executed, as shown at callout 3.

The process is straightforward, and it’s quite powerful in that we can construct the .travis.yml file to define custom configuration and execute tasks that are special, too. In fact, we’re going to use travis.yml to install the Deno language on the runner VM and then execute the unit tests that are part of my Deno project. This is very cool because, remember, as of this writing Travis CI does not support Deno out of the box.

Let’s take a look at how support for Deno is implemented.

Getting Travis CI to support Deno

As mentioned above, the way you get Travis CI to do what you want it to do in its CI/CD process is to construct a .travis.yml file according to the format defined in the Travis CI build config specification.

The way I got Travis CI to support Deno was to do a bit of a hack in my project’s .travis.yml file. The entire contents of .travis.yml are shown below in listing 1. Let’s take a look at the details on a line-by-line basis:

language: default
os: ubuntu
services:
    - docker
branches:
    only:
    - master
before_install:
    - pwd
    - curl -fsSL https://deno.land/x/install/install.sh | sh
    - ls -l $HOME/.deno
    - export DENO_INSTALL="$HOME/.deno"
    - export PATH="$DENO_INSTALL/bin:$PATH"
    - deno run https://deno.land/std/examples/welcome.ts   
script: 
    - cd ./simplecalc/test/
    - sh run_test.sh
    - cd ../../fortune_cookies/test
    - sh run_test.sh

Listing 1: .travis.yml

Travis CI allows you to define a language that gets implemented in the VM runner. It supports a variety of languages out of the box, but sadly, Deno isn’t one of them. Thus, at Line 1 in the travis.yml file above, I set the value of the language attribute to default. It really doesn’t matter which language I define, because I am going to install Deno anyway.

At line 2 at the os attribute, I set the runner VM’s operating system to Ubuntu. Travis CI supports the macOS and Windows operating system too, but I use Ubuntu because my code is intended to run on Linux.

Travis CI allows you to configure the runner VM with a variety of services. For example, I could apply values to the services attribute that will install MySQL, Redis and Memcached to the runner VM automatically. In my case, I need to have Docker installed because one of my unit tests requires that a web server programmed under Deno be up and running as a Docker container. So at Lines 3 and 4, I set docker as a service.

Lines 5 through 7 set the branches attribute to tell Travis CI to use the code in the master branch of the cloned repository. You can set Travis CI to process specific branches in a given repo, but in this case, the code I’m interested in unit testing is in the master branch.

Lines 8 through 14 are where Deno support is implemented. I add the command line tasks required to download Deno. Once downloaded, the task will install Deno and then check that Deno is running. This is all defined under the before_install attribute, which indicates the stage before installation in the Travis CI job lifecycle. The install stage is the time when all dependencies are installed, so in this case, I am telling Travis CI to install Deno before any other download and configuration happens.

Line 9 tells Travis CI to output the present working directory (pwd). I do this to provide debugging information that might be useful later.

Line 10 runs the curl command that downloads and executes the shell (sh) script that does the actual work of installing Deno.

Line 11 lists the contents of the .deno directory. I do this to provide debugging information. When the Deno installation is run, it creates a .deno directory in the user’s $HOME directory where the Deno binaries and dependencies are stored.

Line 12 creates an environment variable, DENO_INSTALL which describes the location of the Deno binary and dependencies.

Line 14 runs the simple “Hello World” Deno application, which is downloaded directly from the Deno home site. I do this just to confirm that Deno is up and running properly.

Lines 15 through19 run the unit tests for the various subjects that are part of my main projects. (See figure 2, below.)

Figure 2: An excerpt from the console output created with travis.yml running my project’s unit tests

That’s it. As you can see, I got Deno up and running under Travis CI using a trivial amount of code in the .travis.yml file. Being able to do so much with such a small amount of code attests to the power that Travis CI brings to the CI/CD experience.

You can view the code for this Deno project on GitHub here.

Join the initiative

Travis CI provides the power and flexibility to make just about any CI/CD workflow possible. The platform’s versatility is reason enough for enterprises to consider adopting it. But when it comes to running Deno under Travis CI, there’s more!

Travis CI supports implementing community-based languages. Thus, it’s entirely possible to get Deno incorporated into the array of languages that Travis CI supports once the infrastructure is in place. All that’s required is for three members of the Deno community to commit to becoming maintainers and keeping the language support active. That being said, I am actively looking for people who want to become maintainers.

Making Deno work out of the box with Travis CI is a win-win for all parties. If you’re interested, contact me on LinkedIn. It will be a great adventure, and the result of our endeavor will benefit many for years to come.

The Cookbook: Build Matrix

Montana Mendy — Thu, 11 Feb 2021 10:53:30 +0000

A build matrix in Travis is made up by several multiple jobs that run in parallel. This can be useful in many cases, but the two primary reasons we see people use matricies is for reducing the overall build execution time and running tests against different versions of runtimes or dependencies to get the best version of the build. Let's learn about Build Matricies in Travis CI.

Build Matrix and setting up the .travis.yml

As you know by now, you need to tell Travis which language environment to select for your project. You can do so using the language key option, in this Cookbook we'll be using PHP. You can specify which PHP versions will the tests be executed. Not introducing patch versions tells Travis to use the latest available, which is sometimes referred to edge, or if it's the stable version, you guessed it, it's called stable. So let's start building out our .travis.yml:

language: php

php:
  - 5.3.3
  - 5.4
  - 5.5
  - 5.6
  - hhvm

Env Vars

Let's tell Travis to do multiple runs/builds with different sets of env var values. To do so, add an env key. Each segment is understood as a different environment and tests are run separately as such. We are going to use PHP_SEGMENT_TEST later in the file to run tester with the argument of -p and hhvm option in HHVM environment, so in your .travis.yml file, add the env segment:

env:
  - PHP_SEGMENT_TESTRUN="php-cgi"
  - PHP_SEGMENT_TEST="hhvm"

Now simple math would have it, the combination of five PHP versions and the two env vars generates a total of 10 runs.

Dependency installation using Composer

You'll obviously have an install segment in your .travis.yml. Each sub-segment means one single command. Composer installs your dev dependencies by default. You should use --no-interaction so Composer doesn't ask questions Travis can't answer, and can continue the build.

You'll want the latest build, so in you .travis.yml make sure you run the update flag at before_install, so it would look like:

before_install:
  - composer self-update

install:
  - composer install --no-interaction

The Build Matrix

Depending on the configuration above (different segments and sub segments), a build matrix wil be generated. The matrix contains all combinations the environment settings. A single combination is called a job and is run separately. You can modify the matrix in matrix section.

If you want to exclude a job, use the exclude key. In our case, we don't want to use -p hhvm parameter for standard PHP versions and -p php-cgi for HHVM:

matrix:
  exclude:
    - php: 5.3.3
      env: PHP_SEGMENT_TEST="hhvm"

    - php: 5.4
      env: TPHP_SEGMENT_TEST="hhvm"

    - php: 5.5
      env: PHP_SEGMENT_TEST="hhvm"

    - php: 5.6
      env: PHP_SEGMENT_TEST="hhvm"

    - php: hhvm
      env: PHP_SEGMENT_TESTRUN="php-cgi"

To your pleasure, you can define jobs that are allowed to fail without causing the whole build to shown as failed. To do so, declare allow_failures. For our sake, we allow HHVM to fail, lets do this via:

matrix:
  allow_failures:
    - php: hhvm

I only specified the PHP version, not the environment variables. This means all jobs with HHVM version will be allowed to fail/retry (if that is an option you have in your .travis.yml. Anyway, nevermind the environment variables values. It works with exclude also.

Running the scripts/tests

Tests are run in script segment of the .travis.yml. Let's assume your tests are in scripts/ folder in your file tree and you provide your own php.ini in the same folder. Additionally, we can tell TESTER to display information about skipped tests with the -s flag and to use value of earlier declared TESTER_SEGMENT_TEST as PHP binary with the -p flag:

script:
  - ./vendor/bin/tester -p $TESTER_SEGMENT_TEST -s -c ./tests/php.ini ./tests

If the build fails we want to use after_failure to get the exact values via:

after_failure:
  # Prints *.actual files contents (if not revert) 
  - for i in $(find ./tests -name \*.actual); do echo "--- $i"; cat $i; echo; echo; done

Other services

Travis comes with multiple popular services and has no shortages of them, (e.g. MySQL, Redis, Docker) pre-installed. However, if you need to use for example Redis storage, you can tell Travis in services section via:

 services:
  - redis-server

DB Init

Depending on what database you chose to use, it can vary -- in this example we are using MySQL. MySQL runs on 127.0.0.1 and you can log in using travis or root as username. Say you have a DB setup script, and it's in tests/montana/testbase.sql., here's a sample segment:

 before_script:
  - mysql -u root -e 'CREATE DATABASE testbase;'
  - mysql -u root testbase < tests/montana/testbase.sql

.travis.yml in it's final form

After a lot of this segment config, this is what you're left with:

language: php

php:
  - 5.3.3
  - 5.4
  - 5.5
  - 5.6
  - hhvm

env:
  - TESTER_SEGMENT_TEST="php-cgi"
  - TESTER_SEGMENT_TESTRUN="hhvm"

matrix:
  allow_failures:
    - php: hhvm

  exclude:
    - php: 5.3.3
      env: TESTER_SEGMENT_TEST="hhvm"

    - php: 5.4
      env: TESTER_SEGMENT_TEST="hhvm"

    - php: 5.5
      env: TESTER_SEGMENT_TEST="hhvm"

    - php: 5.6
      env: TESTER_SEGMENT_TEST="hhvm"

    - php: hhvm
      env: TESTER_SEGMENT_TESTRUN="php-cgi"

services:
  - redis-server

before_install:
  - composer self-update

install:
  - composer install --no-interaction --prefer-source

before_script:
  - mysql -u root -e 'CREATE DATABASE testbase;'
  - mysql -u root testbase < tests/montana/testbase.sql

script:
  - ./vendor/bin/tester -p $TESTER_SEGMENT_TEST -c ./tests/php.ini -s ./tests/

after_failure: # to find actual variables if needed, a verbose esque way of doing things
  # Prints *.actual files content
  - for i in $(find ./tests -name \*.actual); do echo "--- $i"; cat $i; echo; echo; done

Activate webhook

Go to www.travis-ci.com, and flip switch to ON for all repositories you'd like to enable. Travis will now add your repository to queue after every pushed commit or created pull-request. After a short while, your repository will be tested. Now if you'd like the opposite, there's a bash script I've created, which I will attach right below this - as some people find this more efficient:

#!/usr/bin/env sh

echo "Hello from Montana at Travis"
echo $TRAVIS_PULL_REQUEST

if [ "$TRAVIS_PULL_REQUEST" = "false" ]; then
  surge --project ./dist --domain auto-deploy-test.surge.sh
else
  echo "This is a PR, not deploying"
fi

To know if your build is successful either check the GitHub status, or generate a status image.

Congratulations

This is Travis's Build Matrix in a nutshell, it can seem a bit confusing but after going over it a few times, it will start to make sense.

The Cookbook: Jekyll

Montana Mendy — Mon, 25 Jan 2021 08:24:54 +0000

GitHub Pages are an amazing way to host Jekyll pages, but in some cases, you might be interested in running your Jekyll page on a different host (like Azure Web Apps, Heroku, AWS). In this Cookbook let's setup a DevOps Build Pipeline that takes commits, run smoke screen testing, and actually deploying.

Getting started with Jekyll

Ideally with Jekyll, you want a compile script (obviously in bash) that traverses to an out/ directory. For sake of this example, you can call the bash script anything but I'm going to call it jekyll.sh. Your project may require something different, for example npm build or something to that extent.

When creating the file tree/structure, let's make sure the out/ directory contains almost or essentially everything you want deployed to gh-pages. In a case like this, you will 95% of the time have a index.html file.

Check this script into your project. Now this is just an example, now let's say the script just reads (remember, your script might just read npm build:

Using Bash


#!/bin/bash/env sh 

# only let the script proceed when started not by a pull request (PR)

if [ $TRAVIS_PULL_REQUEST == "true" ]; then # you can make this more strict via "==="
  echo "this is a PR, exiting now"
  exit 0
fi

# enable error reporting to the console & log that into a file entitled "log.txt" 

set -e

# build site with jekyll, by default to `_site' folder

bundle exec jekyll build

# find ./_site -name "*.html" -exec bundle exec htmlbeautifier {} \;

bundle exec htmlproof ./_site --disable-external --check-html --verbose

# cleanup using `rm -rf`

rm -rf ../Montana.github.io.master

Travis and adding Travis to your project

We assume you have one, but if not get a Travis account at https://travis-ci.com/. Turn on Travis for the repo in question, using the Travis GUI.

Encrypted credentials

If you read the Cookbook Series: Encryption, you can most likely skip over this, but if you haven't - when you deploy using Travis, ideally you want to deploy gh-pages without checking in the necessary credentials to your repo, this can be done using encryption.

Next up, you'll want to generate a SSH key. The advice I always give is to never reuse SSH keys. To generate a new SSH key, you'll want to open a terminal and run:

ssh-keygen -t rsa -b 4096 -C "your_email@example.com" # in my case "ssh-keygen -t rsa -b 4096 -C "montana@travis-ci.org"

You'll be prompted to do a few things, one thing you want to remember is you do not want to include a password, just press enter when prompted, then follow the next prompt and continue. You'll now want to add the deploy key to your repo, and remember do not reuse SSH keys. You can do this via:

https://github.com/<github_username>/<your_repo>/settings/keys # add the deploy key to your repo

Now it's time to encrypt the key that was generated using the Travis CLI client. So now let's demonstrate on how we would add your deploy key to your specified repo.

Adding your encrypted vars

First, let's open an editor so we can view the dpl key, so let's run:

cat github_deploy_key.pub

You can also use touch, so instead of cat, it would be:

touch github_deploy_key.pub

Now that you have the key, make sure the proper repo is conditioned:

https://github.com/<github_username>/<your_repo>/settings/keys # add the deploy key to your repo

Now let's use Travis again from the CLI again to login:

travis login --org --auto

You'll then want to run:

travis encrypt-file 'github_deploy_key'

Make sure you add your PUBLIC key as well:

git add 'github_deploy_key.enc'

The bash script I've coded below automates the above process essentially, I did this to make it a bit easier on the Travis CI user, for the sake of time lets say this bash script I created is called dpl_key:

#!/bin/bash/env sh 

set -e

if [ -z "$GITHUB_TOKEN" ]; then
  echo ""
  echo "GITHUB_TOKEN environment variable is missing"
  echo ""
  echo "Travis dpl_key generator" 
  echo ""
  echo "This generates a public/private dpl_key, add the public key as a deploy key
  with write access to the origin remote github repo, encrypt the private key as
  github_deploy_key.enc and add the configuration necessary to use it in your .travis.yml file"
  echo ""
  echo "Give it a shot:"
  echo ""
  echo "  GITHUB_TOKEN=\`cat ~/secret/GITHUB_TOKEN\` ./generate_travis_deploy_key"
  echo ""
  echo "where ~/secret/GITHUB_TOKEN is a file containing a github token with write access to the current repository : (origin)"
  echo ""
  echo "You must have the Travis executable installed on your system and available in the PATH"
  echo ""
  exit
fi

url="$(git config --get remote.origin.url)"
reponame="$(echo $url | cut -d/ -f2 | cut -d. -f1)"

# generate a new private and public key

ssh-keygen -t rsa -b 4096 -f github_deploy_key -N '' -C $url -q 1>/dev/null

pubkey="$(cat github_deploy_key.pub)"

# add the PUBLIC key to the github repository as a deploy key with write access

curl https://github.com/<github_username>/<your_repo>/settings/keys -H "Authorization: token $GITHUB_TOKEN" --data @- << EOF
{
  "title": "travis deploy key",
  "key": "$pubkey",
  "read_only": false
}
EOF

# use travis to encrypt the private key as github_deploy_key.enc and remove the private key

travis encrypt-file github_deploy_key --add --no-interactive -w /tmp/github_deploy_key -f --pro
git add github_deploy_key.enc

# cleaning

rm github_deploy_key
rm github_deploy_key.pub

# bash script by montana mendy

.travis.yml

So, this is a .travis.yml file I've created, as you can see if you use my config it will automate the deployment for you, if you'd like to deploy manually, then just remove the line:

language: generic # this doesn't install any environment at all 

before_install:
- bash chmod +x ./dpl_key.sh # will give proper permissions via chmod to both scripts 
- bash chmod +x ./jekyll.sh

branches:
  only:
  - master

before_deploy:
- openssl aes-256-cbc -K $encrypted_0a644212b3ae3_key -iv $encrypted_0a644212b3ae3_key -in deploy_key.enc -out deploy_key -d


deploy:
  provider: pages
  local_dir: out
  deploy_key: deploy_key
  edge:
    branch: master

Now you should have 3 files, the ones we have talked about, jekyll.sh, deploy_key.enc and your .travis.yml. Make sure you've logged into Travis, let Travis know about the repo via syncing, once you push to GitHub it will compile and deploy your source, either using my bash script or doing it step by step.

Conclusion

I hope you learned some niche ways on deploying source while using Travis, until next time!

The Build, Test, Nuke Pattern

Montana Mendy — Thu, 21 Jan 2021 03:22:46 +0000

The Build, Test, Nuke Pattern

A Tutorial on Testing Etiquette in the Age of Ephemeral Computing

Paying only for what you use makes sense technically and financially. IaC offers a lot of benefits, but it does take some getting used to, particularly with regard to resource management.

There’s a certain etiquette involved when a lot of people are using a common set of resources. In a way, being part of an IaC environment is similar to using the laundry room in an apartment building. You need to be aware that other people are using the washers, dryers and folding tables. Thus, you need to keep moving your clothes along from washer to dryer in a timely manner, and you need to clean up after yourself, always.

The same is true in ephemeral computing. There is nothing more irritating than testing on a virtual machine that is all clogged up with useless applications and components left behind from previous tests. Not only do such remnants take up valuable computing resources, but they might also compromise the outcome of your testing.

When running tests in an ephemeral, IaC environment, a good pattern to follow is one that I call the Build, Test, Nuke pattern. Let’s explore the pattern using an example from a set of unit tests I created in my Deno demonstration project and ran on Travis CI.

Understanding Build, Test, Nuke

The structure of the Build, Test, Nuke pattern is to divide the testing event into three stages. In the first stage, Build, your testing script gets the source code from a source control management service, such as GitHub, and builds the required dependencies into a deployment unit, such as a Docker container. In the second stage, Test, the script tests the artifact. Then, in the third stage, Nuke, your script destroys the deployment artifacts used in the testing, removing them from both memory and disk. (See figure 1 below.)

Figure 1: The Build, Test, Nuke pattern

The benefit of the pattern is apparent. Your scripts are using only the resources needed, when they’re needed. For a small application, using the pattern might seem trivial. But, when you get into an application that has a dozen dependencies running in separate processes that put pressure on both memory and network IO, leaving them up and running after testing is done can be a significant risk. Destroying everything when testing is over addresses this risk head-on.

Now that we’ve covered the concept behind the Build, Test, and Nuke pattern, let’s take a look at how it’s implemented when running a set of unit tests under Travis CI.

Putting Build, Nuke, Test into Action

As mentioned above, I use the Build, Nuke and Test pattern in a set of HTTP tests I created for an application in my Deno demonstration project, which you can find [here].(https://github.com/reselbob/denodemo The application is called SimpleCalc. SimpleCalc is an API that exposes the math operations add, subtract, multiply and divide. The API runs as a web application over HTTP.

What is Deno?

Deno is a next-generation programming framework developed by Ryan Dahl, the creator of Node.js. Deno runs using the V8 JavaScript, as does Node.js, but Deno uses the Typescript language to program. Deno also improves upon Node.js in other ways, such as the way dependencies are stored and runtime permissions are granted. For more information about Deno, visit the website at: https://deno.land/.

The way the HTTP test works is the script creates the web server that hosts the SimpleCalc API in a Docker container. Then, once the SimpleCalc API server is up and running, a set of Deno tests are run. After the tests are run, the test script destroys both the container in which the SimpleCalc API server is running and the container image that’s used to create the container for the SimpleCalc API. (You can think of a container image as the template that describes the container that will run in memory.)

Listing 1 below shows the Bash script that creates the container, runs the HTTP tests, and then destroys the container and container image.

#!/bin/bash
cd ..

docker build -t mydenoserver .
docker run -d --name simplecalc -p 7700:7700 mydenoserver

deno test  --allow-net --allow-env --allow-read --allow-write --reload

docker rm -f simplecalc
docker rmi -f mydenoserver

Listing 1: The Bash script that implements the Build, Test and Nuke pattern

Notice that the Build phase takes place at lines 4 and 5. The script uses the Docker build command to create a container image named mydenoserver using the default Dockerfile stored along with the source code. (See line 4 above.) Then, at line 5, the script executes docker run to run a container named simplecalc using the container image mydenoserver.

Line 7 in listing 1 above invokes the unit tests that are stored in the test directory of the SimpleCalc app source code files. (The Deno test looks for test files in a directory named test by default.)

Listing 2 below shows an excerpt from the test file api_tests.ts. Notice at line 9 that the test accesses the SimpleCalc API’s add operation by way of the webserver running at http://0.0.0.0:7700/sum/8,4,3. The IP address 0.0.0.0 indicates the source address for the local computer on which the test is running. The SimpleCalc API is listening for traffic on port 7700. The /sum/8,4.3 segment of the URL is special to the SimpleCalc API. It describes the sum operation and an array of integers that will be summed up by the operation.

import { assertEquals } from "https://deno.land/std/testing/asserts.ts";

// The test, Can add from API, is an excerpt from a set of many tests
Deno.test({
    name: "Can add from API",
    ignore: false,
    async fn() {
        const url = `http://0.0.0.0:7700/sum/8,4,3`
        await fetch(url)
        .then((response) => {
            return response.text()
        })
        .then(data => {
            assertEquals(Number(data), 15);
        })
    },
  });

Listing 2: An excerpt from the test of HTTP tests that exercise the SimpleCalc API.

The important thing to know about listing 2 above is that it describes a test that is run against a container that hosts the web server, and that container was created on demand as part of the testing process.
After the test is completed, the Bash script controlling the entire process will issue these two commands as shown below, as well as above in listing 1 at lines 9 and 10:

docker rm -f simplecalc
docker rmi -f mydenoserver

These are the commands that remove both the container and the container image from the host environment. These two lines fulfill the last part of the Build, Test, Nuke pattern. As you can see, the container and the container image are removed completely from the virtual machine host.

Listing 3 below shows the .travis.yml file that gets run when new or adjusted code is committed to the demonstration Deno project.

language: default
os: ubuntu
services:
    - docker
branches:
    only:
    - master
before_install:
    - pwd
    - curl -fsSL https://deno.land/x/install/install.sh | sh
    - ls -l $HOME/.deno
    - export DENO_INSTALL="$HOME/.deno"
    - export PATH="$DENO_INSTALL/bin:$PATH"
    - deno run https://deno.land/std/examples/welcome.ts

script: 
    - cd ./simplecalc/test/
    - sh run_test.sh

Listing 3: The Travis.yml file that installs Deno executes the script run_test.sh, which implements the Build, Test, Nuke pattern in the Travis runtime environment.

One of the nice features of Travis CI is that I can register the Deno demonstration project that’s hosted on GitHub with Travis. Once registered, Travis will install a webhook in the GitHub repo that makes it so when code is committed to the repo, the repo will be cloned automatically into a Travis CI virtual machine dedicated to my testing session. Travis looks for the .travis.yml in the source code and then, once identified, will configure the virtual machine according to settings in the .travis.yml file.

In this case, the instructions to install Deno are at lines 8 through 14 in listing 3 above. The instruction to navigate to the directory that has the Bash script that controls the testing process is at line 17. The instruction to execute the Bash script is at line 18.

Remember, the Bash script contains all the instructions to execute the Build, Test, and Nuke pattern. Mission accomplished!

Putting It All Together

The Build, Test, and Nuke pattern is simple and powerful. Using it consistently as part of your CI/CD process will help you run your tests as a reliable, good neighbor in the host runtime environment, whether they’re running on your local machine, under Travis or anywhere else in the cloud.

The Cookbook: Fable

Montana Mendy — Mon, 18 Jan 2021 11:03:44 +0000

Fable produces readable JavaScript code compatible with ES2015 standards and popular tooling like Webpack, which you've probably heard of if you've ever used React. Let's start this Fable.

Build Fable with Travis CI

Hey builders let's get to building Fable with Travis CI! Let's start out with your .travis.yml file:

language: minimal

services:
  - docker

sudo: required

before_install:
  - docker pull vbfox/fable-build

jobs:
  include:
    - stage: "CI"
      script: docker run -v "${PWD}:/app" -w "/app" vbfox/fable-build bash -c "yarn && yarn postinstall && yarn build"

There's sometimes issues with Fable for whatever reason with syntax errors, so you may want to take the time and lint your .travis.yml file, you can do this via running travis lint.

The `package.json`

The package.json contains something like the following:

    "scripts": {
    "start": "webpack-dev-server",
    "build": "webpack -p",
    "postinstall": "paket restore && paket generate-load-scripts -f netstandard2.0 -t fsx",
    "predeploy": "npm run build",
    "deploy": "gh-pages -d docs"
  },

As you can see webpack is being triggered there as well. That's it - you've now built Fable in Travis CI! Phew, that was easier than we thought.

The Cookbook: Branch Flow

Montana Mendy — Tue, 12 Jan 2021 03:29:35 +0000

It's important when using multiple branches in your repository to define which branches to build to, and which branches not to build to. Let's get started.

Branch Flow

Travis CI uses the .travis.yml file from your project structure, once you've configured your flow via the .travis.yml file, you can trigger a build by pushing changes through the CLI using:

git init
git add . 
git commit -m "Travis build branch" 
git remote add origin remote repository URL
git remote -v 
git push -u origin master

You can tell Travis to build multiple branches using blacklists or whitelists. More specifically you can define which branches to build using a whitelist, or on the flip side you can use blacklist. You can blacklist branches that you do not want to be built via the following:

# blacklist (branches you don't want to be built) 
branches:
  except:
    - legacy
    - edge

# whitelist
branches:
  only:
    - master
    - stable

If you specify both whitelist and blacklist, only takes precedence over except. By default, let's say you have a gh-pages branch, that branch is not built unless you add it to the whitelist explicitly. When deploying to your version control, or if you're using Apache SVN (Subversion), in some cases, it may reset your working directory, it does this via:

git stash --all

As a cursory cautionary move, you can add skip_cleanup to your .travis.yml file. So after, your yml file might look like this:

deploy:
  skip_cleanup: true

If there's anything you want to add before you deploy, you can obviously add before_deploy. This is also true for an inverse method, for after deployment via after_deploy.

Deploying and Using WebAssembly Under Deno on the Server Side Using Travis CI

Montana Mendy — Thu, 07 Jan 2021 11:05:23 +0000

Containers are a game changer in software development. They provide the operational isolation found in virtual machines without the overhead. Whereas it can take a virtual machine minutes to spin up, you can have a container up and running in seconds, or even in some cases milliseconds.

Getting started

Containers have become central to modern distributed application architecture, particularly now that container orchestration technologies such as Kubernetes and Docker Swarm have become commonplace in the enterprise. However, as popular as container technology is for implementing isolated processes for use in a distributed application, there is an alternative. That alternative is WebAssembly.

WebAssembly is a compiled binary that operates inside a runtime unit called a WebAssembly VM, or virtual machine. WebAssembly has been around for a while on the client side, and all the major web browsers support it. But you can also run it on the server side. All you need is to have a programming framework that supports server-side WebAssembly VM.

Deno — an emerging language from Ryan Stahl, the creator of Node.js — supports WebAssembly. Deno uses the V8 JavaScript runtime found in Chrome on the server side, as does Node.js. The WebAssembly VM is built right into V8, so Deno supports WebAssembly.

In this tutorial, I am going to show how to use Travis CI to build a WebAssembly binary written in Rust, and then run a demonstration application I wrote in Deno that uses the WebAssembly as an internal component.

This tutorial is high-level, and you don’t need to know the details of Rust or Deno to get benefit from reading it. The hope is that the tutorial will spur your interest to delve further into the technologies.

Understanding the demonstration application

The demonstration project that accompanies this article is a Deno application that uses a WebAssembly binary. The Deno code uses logic in the WebAssembly to do a calculation. The calculation that the WebAssembly binary provides is encapsulated into a function named cube(). The function cubes a number that is provided as a parameter. For example, when the number 3 is passed to the function, as cube(3), the function returns 27.

As shown in figure 1 below, the WebAssembly (WASM) binary is written in the programming language Rust. Rust is a compiled language. A programmer writes some code in text and stores it in a file named lib.rs. The code is passed to the Rust WASM compiler. The result is a WebAssembly binary that’s used by the Deno code.

Figure 1: Deno uses a WebAssembly binary programmed in Rust.

That’s the high-level view. Let’s take a look at some of the detail involved in creating the WebAssembly binary in Rust and then consuming the binary in Deno.

Examining the WebAssembly code

The work required to create a WebAssembly binary in Rust is consolidated using Rust’s crate packaging technology.

The Rust programming community provides a packaging ecosystem similar to npm for Node.js and to Java’s Maven. Packages under Rust are called crates. The standard file that describes a crate is named Cargo.toml. It’s similar to the way Node.js uses package.json to describe a Node.js package. (I’ll explain the Cargo.toml file for the demonstration Rust binary in a moment.)

The way you install Rust packages from an external Rust package repository is to use the command cargo install .

Creating a WebAssembly in Rust can be a detail-laden undertaking. To simplify the process, we’re going to use an external package called wasm-pack. wasm-pack will execute against the Cargo.toml file that defines the WebAssembly binary we’re going to create. (Keep in mind that the standalone WebAssembly binary is considered a package.)

Listing 1 below shows the Cargo.toml file that describes the Rust project.

[package]
name = "cube_it"
version = "0.9.0"
edition = "2018"
authors = ["Bob Reselman <reselbob@gmail.com>"]
description = "A small project to demonstrate how to run to create a WebAssembly binary"
license = "MIT/Apache-2.0"
repository = "https://github.com/reselbob/deno-wasm"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
wasm-bindgen = "0.2"

[lib]
crate-type = ["cdylib", "rlib"]

Listing 1: The Cargo.toml file

Lines 1 through 12 above provide typical metadata about the project. Notice in line 1 that the arbitrary name of the package is cube_it. This makes sense because the WebAssembly binary that this package definition creates has a function that will cube a number.

Lines 13 and 14 indicate that the Rust project will use an additional package, wasm-bindgen. The purpose of wasm-bindgen is to provide interoperability between the Rust binary and JavaScript code. Deno programming is done in TypeScript. However, at runtime TypeScript transpiles into JavaScript. Using wasm-bindgen allows the Rust code to access JavaScript functions such as logging and the JavaScript DOM model, so wasm-bindgen is defined in Cargo.toml.

Lines 16 and 17 assign the library cdylib to crate-type. This provides interoperability with C code. In addition, crate-type is assigned the value rlib to indicate that a Rust library will be produced.

Listing 2 below shows the Rust code for the function cube(x: u32). The function will reside in the file lib.rs.

#[no_mangle]
pub extern "C" fn cube(x: u32) -> u32 {
    x * x * x
}

Listing 2: The Rust code in lib.rs that publishes the cube function

The logic in the function cube(x: u32) will be consumed by the Deno code.

Once the Rust package is declared in Cargo.toml and the cube() function is defined in lib.rs, we’ll need to create the Deno code to consume the WebAssembly binary that’s created.

Running WebAssembly under Deno

Listing 3 below shows the contents of the file main.ts. This file contains the Deno code that consumes the WebAssembly binary that contains the function cube().

const wasmCode = await       Deno.readFile("./rust_for_deno/target/wasm32-unknown-unknown/release/cube_it.wasm");
const wasmModule = new WebAssembly.Module(wasmCode);
const wasmInstance = new WebAssembly.Instance(wasmModule);
const {cube} = wasmInstance.exports;

console.log(cube(1));
console.log(cube(2));
console.log(cube(3));
console.log(cube(4));

Listing 3: The Deno code that consumes the WebAssembly binary

Lines 1 through 5 do the work of loading the WebAssembly into the V8 virtual machine and making the cube() function accessible to the Deno code. Lines 7 through 10 execute the cube() function four times, each time with a different parameter value, and then write the output of each call to the console.

Now that we have both the Rust WebAssembly and Deno projects coded and ready to go, we need to create a travis.yml file that will build the WebAssembly binary, as well as install the Deno runtime environment and run the Deno code.

Working with the `.travis.yml` file

One of the nice features of Travis CI is that it will execute a run script in response to an event generated from a webhook installed in a GitHub repository that is bound to Travis CI. Under Travis CI, the run script has a standard name, .travis.yml.

The deno-wasm demonstration project for this article stored on GitHub has a webhook installed that sends an event notification to Travis CI whenever code is committed to the deno-wasm repository. Travis CI receives the event notification and clones the code into a runtime virtual machine created just for the project. Then, it runs the instructions in the expected .travis.yml file.

Listing 4 below shows the contents of the .travis.yml file.

language: rust
os: ubuntu
branches:
    only:
    - master
before_install:
    - pwd
    - rustc --version
    - sudo apt-get update
    - sudo apt-get -y install libssl-dev pkg-config 
    - rustup target add wasm32-unknown-unknown
    - cargo install wasm-pack
    - export PATH="$HOME/.cargo/bin:$PATH"
    - cd ./rust_for_deno
    - wasm-pack build
    - cd ..
    - curl -fsSL https://deno.land/x/install/install.sh | sh
    - ls -l $HOME/.deno
    - export DENO_INSTALL="$HOME/.deno"
    - export PATH="$DENO_INSTALL/bin:$PATH"
    - deno run https://deno.land/std/examples/welcome.ts

script: 
    - deno run --allow-read main.ts

Listing 4: The travis.yml file for building and running the deployment on Travis CI

The first thing to notice about the .travis.yml file in listing 4 above is that Travis CI supports the Rust programming language right out of the box. All you need to do is set the language definition, as shown at line 1.

The work of creating the WebAssembly binary and installing Deno is done in the before_install section, starting at line 5.

Lines 7 and 8 are convenience instructions that report the present working directory (pwd) and the current version of the Rust compiler installed (rustc --version). This is done just to make sure that the .travis.yml script is talking to the Travis CI runtime in a predictable manner.

Lines 9 and 10 install some added packages that Ubuntu needs in order to execute the wasm-pack instruction that will come.

Lines 11 through 15 do the work of installing the wasm-pack Rust package and creating the WebAssembly binary using wasm-pack.

Deno is not installed in the Travis CI runtime environment by default. Thus, lines 17 through 20 do the work of installing it and configuring the DENO_INSTALL environment variable. Also, the PATH environment variable is updated to accommodate the Deno binaries.

Line 21 runs a Deno Hello World program downloaded from the internet, to verify that Deno is up and running.

As mentioned earlier, the code that uses the cube() function in the WebAssembly binary is stored in the file main.ts. main.ts is called within the script section of the .travis.yml file at line 24 above.

The output of the call to the main.ts file is shown below in figure 2.

Figure 2: The output from the demonstration application running under Travis CI

So there you have it. We’ve just used the Travis CI runtime environment to create a WebAssembly binary and run it in a Deno program. Granted, this is just a small demonstration program, but it does reveal the critical aspects of using WebAssembly to provide services to interested consumers in an isolated manner.

Putting it all together

It’s interesting that Solomon Hykes, the co-founder of Docker, said this about WebAssembly:

"“If WASM+WASI [WebAssembly and its system interface] existed in 2008, we wouldn’t have needed to create Docker."

Server-side WebAssembly has significant potential. Its elegance and power make it an attractive technology for the fast execution of complex algorithms on the server side. Consequently, there are a growing number of projects that are publishing tools for WebAssembly development. You can already create WASM binaries in Go, Rust, C# and C/C++. In addition, there are initiatives to bring support for WebAssembly to a wider array of languages. For example, AssemblyScript is intended to allow developers to create WebAssembly binaries from JavaScript code.

The opportunities at hand are significant. Of course, in order to make WebAssembly a viable addition to your programming toolbox, you will need to take the time to master the details of the technology. But, as current trends reveal, WebAssembly is becoming a mainstay on the server side, and realizing its value will be an investment that will yield dramatic returns.

WebAssembly is here. The time to take advantage of the opportunities at hand is now.

DEV Community: Montana Mendy

Getting started with an open source NSA tool to construct distributed graphs

Getting started with NSA's Datawave

What you'll need

Installing Datawave through the CLI in 5 commands

Overriding your default binaries

Let's try Datawave out finally

Edges

Datawave Architecture

Conclusion

Paralleling Code Coverage using Coveralls and Travis

Creating a base build of Snapcraft in Travis CI

Migrating between bases

Enforcing Snap Policies

Conclusion

Start Maxwell with Namespaced Topic Kafka Producer (Look for Idle Listeners in Kafka)

Travis CI Pipelines: 2 Approaches to Source Control Branching

Branching from repo

Branching from fork

Advantages and tradeoffs

Branching from repo is well suited to private governance

Branching from fork is well suited to open-source development

Putting it all together

Configuring Travis CI to Run a Deno Project

Getting started

Travis CI 101

Getting Travis CI to support Deno

Join the initiative

The Cookbook: Build Matrix

Build Matrix and setting up the .travis.yml

Env Vars

Dependency installation using Composer

The Build Matrix

Running the scripts/tests

Other services

DB Init

.travis.yml in it's final form

Activate webhook

Congratulations

The Cookbook: Jekyll

Getting started with Jekyll

Using Bash

Travis and adding Travis to your project

Encrypted credentials

Adding your encrypted vars

.travis.yml

Conclusion

The Build, Test, Nuke Pattern

The Build, Test, Nuke Pattern

Understanding Build, Test, Nuke

Putting Build, Nuke, Test into Action

What is Deno?

Putting It All Together

The Cookbook: Fable

Build Fable with Travis CI

The package.json

The Cookbook: Branch Flow

Branch Flow

Deploying and Using WebAssembly Under Deno on the Server Side Using Travis CI

Getting started

Understanding the demonstration application

Examining the WebAssembly code

Running WebAssembly under Deno

Working with the .travis.yml file

Putting it all together

The `package.json`

Working with the `.travis.yml` file