DEV Community: Tyson Lawrie

Creating a GitHub stale repositories cleanup bot with Boomerang Flow

Tyson Lawrie — Tue, 01 Jun 2021 01:53:02 +0000

Why Boomerang Flow?

Boomerang Flow is a no-code event-driven cloud native workflow automation tool and is an open-source project.

There are a number of tools that you can use to create a GitHub bot; GitHub Actions, Netlify, Heroku, or any functions hosting cloud, with various skill levels required.

The following will give you a view on doing the same with no-code automation.

Why a repositories bot?

This particular bot is useful to identify if new repositories are being created and you want to make sure to keep a clean GitHub Organization. Its possible that repositories get created for testing purposes and forgotten about, or are created and now need to be managed with issue templates or a license.

Creating the Workflow

The default landing page is the Workflows screen and in your space will be the ability to create a new workflow.

Wiring together the bot

On the Workflow Editor screen you can drag and drop a number of tasks from the palette providing a great end user experience. For advanced management these tasks are Tekton defined YAML and can be managed via the Task Manager.

Find Repositories in Org

Drag on the Find Repositories in Org Task and add the required specifics.

Custom Task Name: Find Repositories.
This will be used later on to get the result parameters.

Parameter	Value
URL	`https://github.com/api/v3`
Token	You can get this value on github.com from your profile under Settings > Developer settings > Personal Access Tokens.
Org	The org you want to scan, in this demo I chose the boomerang-io org
Visibility	The types of repositories you want returned. Default: All
Repositories to Skip	This is a newline delimited list of repositories to ignore based on repository name
Number of Repositories to Retrieve	Helpful to retrieve more or less based on the GitHub API paging.

Send Simple Slack Message

Drag on the Send Simple Slack Message Task and add the required specifics.

We make use of the prior tasks result parameters to output the repositories of note.

Parameter	Value
Webhook URL	You create this in Slack App for your workspace as an Incoming Webhook and will be similar to `https://hooks.slack.com/services/<id>/<id>/<id>`
Channel	This can be a channel or direct message
Icon	I usually go for a bot or Github logo
Username	`Boomerang GitHub Bot`
Message	This is a weekly reminder announcement from your friendly GitHub Bot\n\nThe following repositories are currently in the Boomerang org and need to be cleaned up:\n$(task.Find Repositories.results.repositoriesPrettyPrint)

Wire the Tasks together

Drag and drop the lines between the tasks. You can make use of the Task state as well by changing it to only proceed on success.

Find out more information about Link and Tasks states.

Configure Schedule Trigger

We want this to run on a weekly or monthly schedule. On the Configure tab under Triggers, enable the Scheduler and then click Change Schedule.

In the basic view you can set a weekly schedule or you can switch to the advanced view and set any cron based schedule.

The Activity

Boomerang Flow provides an activity view where you can see whats running, dynamically tail logs, and view a Tasks Result Parameters.

Result Parameters are important in this Workflow as it is how we passed information from the repositories listing Task that connected to GitHub, to the Slack messaging Task.

The Result

And as you can see, the workflow resulted in a message post to a Slack channel. This is a really easy, no-code way of creating a bot.

What did we achieve

A no-code workflow
No configuration of setting up a function running in a cloud
No YAML!
Easy to use interface for providing information
A bot that can keep on running and be refined with a filterable list.

Future Steps

If you are confident that the bot is now finding the correct repositories, you can implement a repository deletion mechanism or raise a GitHub issue to close the repository.

Show your support

Show your support if this helped you:

⭐️ the GitHub repository
💬 Follow us on Twitter
Install the project on your own Kubernetes cluster
🙏 Join us and help contribute

Canceling a Tekton TaskRun with Fabric8 Kubernetes Java Client

Tyson Lawrie — Thu, 13 May 2021 00:16:04 +0000

This work is being done as part of my open-source contribution to Boomerang Flow, a low-code cloud-native workflow automation project.

The Fabric8 Java Kubernetes Client helps you integrate with Kubernetes and it has an extension for integrating with Tekton.

It makes building out Java code extremely easy, however, you still need to know what or how to update the objects. This is where examples and references really help and is a great place to start as a contributor!

The Code

As the name says, this article is all about how to Cancel TaskRuns in Tekton using the Fabric8 Tekton client. You can see the latest example code here

But there is no Cancel action

Unlike create(), list(), delete(), etc there is no corresponding action for cancel instead you need to use updateStatus().

This comes down to the fact that Kubernetes itself doesn't understand Cancel, and in fact you are inserting a new Status for the Kubernetes object.

Let's update the Status

In reading the Tekton TaskRun documentation all you have to do is stick "TaskRunCancelled" in the Status object.

apiVersion: tekton.dev/v1alpha1
kind: TaskRun
metadata:
  name: go-example-git
spec:
  # […]
  status: "TaskRunCancelled"

Seems easy enough, right?

🚧 ...not so fast!

This doesn't actually work...or at least I could not get it to work.

So what does work?

What worked for me was updating the Condition block in the Status.

Retrieve the particular TaskRun
Build up a Condition block and set the type, status, reason, and message.
Replace all existing Conditions. If you don't replace the existing conditions, you will receive a message back saying "Not all Steps in the Task have finished executing".

TaskRun taskRun = client.v1beta1().taskRuns().inNamespace("default").list().getItems().get(0);

List<Condition> taskRunConditions = new ArrayList<>();
Condition taskRunCancelCondition = new Condition();
taskRunCancelCondition.setType("Succeeded");
taskRunCancelCondition.setStatus("False");
taskRunCancelCondition.setReason("TaskRunCancelled");
taskRunCancelCondition.setMessage("The TaskRun was cancelled successfully.");
taskRunConditions.add(taskRunCancelCondition);

taskRun.getStatus().setConditions(taskRunConditions);

client.v1beta1().taskRuns().updateStatus(taskRun);

There is a really great table in the Tekton TaskRun documentation for the different execution statuses you can expect.

Show your support

Do not hesitate to support us if this helped you:

⭐️ the GitHub repository
💬 Follow us on Twitter
Install the project on your own Kubernetes cluster
🙏 Join us and help contribute

Converting to Tekton in Boomerang Flow using Fabric8

Tyson Lawrie — Fri, 07 May 2021 05:32:07 +0000

Recently, as part of work on version 3 of Boomerang Flow, we undertook a decision to migrate from Kubernetes Jobs to Tekton Tasks and TaskRuns.

As we stepped back and looked at the how and after many deep conversations late at night (thanks Marcus), we decided that it was also the right time to migrate from the existing Kubernetes Java Client to the Fabric8 Kubernetes Java Client.

What is Boomerang Flow

Boomerang Flow is an open-source event-driven cloud-native workflow automation tool that runs on Kubernetes and provides a no-code drag and drop editor for workflow creation. We focus first on the end user experience and then the technical features.

Why Tekton

It was a big decision to make, one that had been in the works for a while with over 9 months of healthy discussions, and so at the start of 2021 we commenced the changes.

Whilst we had done a lot of great work in our internal controller to orchestrate and manage Kubernetes Jobs, there were a number of areas that we lacked features or had alternative ways of achieving the same result, but without the broader community. We were also starting to see cracks in our error handling

We also wanted to make sure that by adopting Tekton we would not be sacrificing any end-user experience. In the following comparisons you'll see that in majority of the cases we could actually build out new functionality to help our users.

For now, we are only adopting the Tekton Task/TaskRun and working in with our existing Workflow DAG, it opens up a huge range of benefits;

Leverages the community of a popular open-source system
Provides additional Task functionality with steps and script blocks
Provides a wider range of Tasks built by community
Allows us to focus less on adding new functionality to the engine, and spending this time on enhancements with the community.

Tasks

We had Tasks, Tekton has Tasks. The models were relatively the same. Fantastic! Alignment would not be a pain.

We defined our Tasks in JSON, Tekton Tasks are in YAML. The great JSON vs YAML debate took place and we settled on YAML as this is only for the advanced user and does not take away from the end-user experience. Plus, everything is YAML!

We did however have metadata and elements needed for the UI and as such we have followed the Kubernetes approach and stored this metadata in the annotations and default it when importing a straight Tekton Task YAML without the optional annotations.

We did not support multi-step or script blocks so it was a win-win to be able to bring this to our users. Whilst the typical user does not see any of this, they drag and drop a task from a Palette onto the Workflow Editor and provide inputs, it allows us to provide more powerful and complex tasks and task management, abstracted by a powerful experience.

Outcomes

New YAML editor as part of Task Management
Experience metadata stored in the annotations
Additional Task features with script, working directory, environment variables, result parameters, and multi-step (coming soon).

Parameters

We had Properties, Tekton has Parameters. We had ${p:property} as the syntax, Tekton has $(params.parameter). We passed in all properties to a task, with Tekton you have to explicitly define the parameters wanted.

The biggest difference was in how we approach the layering and substitution of Parameters. Boomerang Flow offers additional layers for the user to have greater control as to how to store and re-use information.

Boomerang Flow also provides powerful inheritance and full recursive substitution, this means that if you define a Parameter at the Team layer and then also at the Workflow level, the Workflow Parameter will win out, this is all resolved as part of using $(params.<parameter>). Or if you have a Workflow Parameter that references another Parameter this will also be resolved and substituted.

Boomerang Flow	Tekton
System	Content
Global	-
Team	-
Workflows	Pipelines
Tasks	Tasks

Outcomes

Retained the same great experience and powerful inheritance and recursive substitution with all existing layers kept.
Adjusted to the Tetkon $( ) syntax
Explicitly define the parameters that are sent through to a task.

Result Parameters

We had Output Properties, Tekton has Result Parameters.

We utilized a watcher side car to scrape a folder and turn all files into Output Properties by posting these back using a HTTP Post and then patch a ConfigMap.

Tekton on the other hand requires explicitly defining the result parameters and

scrapes a folder for files named that match the explicitly defined expected results
Stores them in the container termination message. This is thus limited to 4096 bytes
relies on alternate resources for larger outputs.

What we lose in this instance we gain in other areas, including in not having a custom implementation that required patching ConfigMaps that sometimes failed. We will also be submitting a TEP to the community on hopefully adjusting how parameters are stored to remove the size restriction.

We also were able to update the UI to now provide auto complete when reference the result parameter of another task. Isn't that neat!

Outcomes

Explicitly defined Result Parameters in Task Manager
Deprecation of our bespoke Watcher and related code
A more stable well tested implementation
Auto prompt reference of result parameters when in the Workflow Editor
New description when viewing results in the Workflow Activity.

Error handling

Another big win was solving a number of our error handling concerns. Tekton provided a structured Task error object and error handling, and watching for these errors was made a lot simpler with the Fabric8 client.

We could then capture and translate these to nice user based messages to return to the user and provide greater visibility and detail at both the Task and Workflow level. This was a powerful enhancement for our user experience.

Outcomes

More structured and detailed errors
Better ability to watch and handle the errors
New Task and Workflow error viewing in the Workflow Activity for greater visibility.

Why change Java clients

Technically we did not have to move from the vanilla Kubernetes Java Client to the Fabric8 Kubernetes Java client. We could have achieved the required CRDs (Custom Resource Definitions) in the existing client... that being said, it was much easier with the Fabric8 Tekton extension.

The Fabric8 community itself has also been very helpful and its these experiences that help provide the level of examples and references for advanced implementations such as a Custom Watcher for TaskRun execution.

Acknowledgments

This migration would not have been possible without the blogs and information from the community, in particular

And my fellow contributors: Ben, Marcus, and Costel.

Show your support

Do not hesitate to support us if this helped you:

⭐️ the GitHub repository
💬 Follow us on Twitter
Install the project on your own Kubernetes cluster
🙏 Join us and help contribute

CICD Evicted: exceeded ephemeral storage

Tyson Lawrie — Fri, 28 Aug 2020 02:53:10 +0000

I am a Software Engineer enjoying the journey to a Cloud Native landscape. You can find my previous posts on Medium

Pod ephemeral local storage usage exceeds the total limit of containers

Recently we hit a major issue with the builds for some of our AI based components in our DevOps pipeline. Due to either storage or memory limits. As with all the new modern DevOps tools, a task (essentially a Kubernetes Job) is spun up to complete the commands required to build and package.

In can be complex to pinpoint exactly what the issue is and what reaching these limits mean, at what point in the task are these limits reached.

ephemeral-storage = emptyDir && medium = memory

In kubernetes ephemeral-storage can be mapped to an emptyDir, essentially a scratch space for the duration of the pod which is then terminated. Unfortunately on our clusters, each Node only has a maximum disk space of 100Gb so it very quickly fills up.

Ok so let's use this cool medium feature and set the storage to use memory. Bonus the task sped up, we saw an increase in performance of about 20%. Unfortunately we still reached a maximum for the task with the memory resource limit.

Volumes:
  cicd-vol-data:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  Memory

Visibility = New Relic infrastructure

We really needed to get a realistic view of what all these CICD tasks were doing, we could no longer just sail through with blind trust in tasks and pipelines. Time for a real deep analysis.

With new relic infrastructure agents on our kubernetes nodes we put together a dashboard that would show us a breakdown of the memory, storage, and CPU thats being consumed by our tasks.

In this dashboard you can see the spikes specifically for the AI related worker are outliers compared to all the other tasks. As you can see they reach the limits previously applied for disk and memory (16Gb Ephemeral + 16Gb of Memory as Disk) and just abruptly terminate. CPU is fine.

It seems we have just transferred the issue from storage to memory.

Unpacking the layers

In analyzing the memory usage, the task would consume a total of 21.6Gb of memory. Essentially the working scratch directory in memory was ~18Gb by completion. Well thats HUGE!

In breaking down the 18GB, we have;

After Clone = 1.7Gb
After Image Build & Export we have = ~16Gb.

Image build includes the build context, tempfs, and output archive. The build context
**Whenever there is a Dockerfile command (COPY . . or RUN xyz .) its copying the context into different layers. Every command in the container forms layers.

The key was to understand the various layers being built to see we were starting to copy the file contents around the place making multiples of 1.7Gb

The solution

To get this going for our team, we increased the memory limit to 24Gb, which was easier for us than disk (plus we wanted the speed increase). Hooray, it built!

And now we focus on the Dockerfile and code structure to try and make it more efficient and bring this back down to within normal range.

Thanks go to my whole team, in particular, key members of that team: Glen Hickman and Ionut Palada for the help in solving this issue.