DEV Community: Datree

ArgoCD Best Practices You Should Know

Itamar Ben Yair — Thu, 07 Apr 2022 11:02:57 +0000

My DevOps journey kicked off when we started to develop Datree - an open-source CLI tool that aims to help DevOps engineers to prevent Kubernetes misconfigurations from reaching production. One year later, seeking best practices and more ways to prevent misconfigurations became my way of life.

This is why when I first learned about Argo CD the thought of using Argo without knowing its pitfalls and complications simply didn’t make sense to me. After all, it’s probable that configuring it incorrectly can easily cause the next production outage.

In this article, we’ll explore some of the best practices of Argo that I've found and learn how we can validate our custom resources against these best practices.

Argo Best Practices

1. Disallow providing an codepty retryStrategy (i.e.{})

Project: Argo Workflows

Best practice: The user can specify a retryStrategy that will dictate how failed or errored steps are retried in a workflow. Providing an codepty retryStrategy (i.e. retryStrategy: {}) will cause a container to retry until completion and eventually cause OOM issues.

Resources: read more

2. Ensure that Workflow pods are not configured to use the default service account

Project: Argo Workflows

Best practice: All pods in a workflow run with a service account which can be specified in the workflow.spec.serviceAccountName. If omitted, Argo will use the default service account of the workflow's namespace. This provides the workflow(i.e the pod) the ability to interact with the Kubernetes API server. This allows attackers with access to a single container to abuse Kubernetes by using the AutomountServiceAccountToken. If by any chance, the option for AutomountServiceAccountToken was disabled then the default service account that Argo will use won’t have any permissions, and the workflow will fail.

It’s recommended to create dedicated user-managed service accounts with the appropriate roles.

Resources: read more

3. Ensure label `part-of: argocd` exists for ConfigMaps

Project: Argo CD

Best practice: Related ConfigMap resources that aren’t labeled with app.kubernetes.io/part-of: argocd, won’t be used by Argo CD.

When installing Argo CD, its atomic configuration contains a few services and configMaps. For each specific kind of ConfigMap and Secret resource, there is only a single supported resource name (as listed in the above table) - if you need to merge things you need to do it before creating thcode. It’s important to annotate your ConfigMap resources using the label app.kubernetes.io/part-of: argocd, otherwise, Argo CD will not be able to use thcode.

Resources: read more

4. Disable with DAG to set FailFast=false

Project: Argo Workflows

Best practice: As an alternative to specifying sequences of steps in Workflow, you can define the workflow as a directed-acyclic graph (DAG) by specifying the dependencies of each task. The DAG logic has a built-in fail fast feature to stop scheduling new steps, as soon as it detects that one of the DAG nodes has failed. Then it waits until all DAG nodes are completed before failing the DAG itself. The FailFast flag default is true. If set to false, it will allow a DAG to run all branches of the DAG to completion (either success or failure), regardless of the failed outcomes of branches in the DAG.

Resources: more info and example about this feature here.

5. Ensure Rollout pause step has a configured duration

Project: Argo Rollouts

Best practice: For every Rollout, we can define a list of steps. Each step can have one of two fields: setWeight and pause. The setWeight field dictates the percentage of traffic that should be sent to the canary, and the pause literally instructs the rollout to pause.

Under the hood, the Argo controller uses these steps to manipulate the ReplicaSets during the rollout. When the controller reaches a pause step for a rollout, it will add a PauseCondition struct to the .status.PauseConditions field. If the duration field within the pause struct is set, the rollout will not progress to the next step until it has waited for the value of the duration field. However, if the duration field has been omitted, the rollout might wait indefinitely until the added pause condition will be rcodeoved.

Resources: read more

6. Specify Rollout’s `revisionHistoryLimit`

Project: Argo Rollouts

Best practice: the .spec.revisionHistoryLimit is an optional field that indicates the number of old ReplicaSets which should be retained in order to allow rollback. These old ReplicaSets consume resources in etcd and crowd the output of kubectl get rs. The configuration of each Deployment revision is stored in its ReplicaSets; therefore, once an old ReplicaSet is deleted, you lose the ability to roll back to that revision of Deployment.

By default, 10 old ReplicaSets will be kept, however, its ideal value depends on the frequency and stability of new Deployments. More specifically, setting this field to zero means that all old ReplicaSets with 0 replicas will be cleaned up. In this case, a new Deployment rollout cannot be undone, since its revision history is cleaned up.

Resources: read more

7. Set scaleDownDelaySeconds to 30s to ensure IP table propagation across the nodes in a cluster

Project: Argo Rollouts

Best practice: When the rollout changes the selector on service, there is a propagation delay before all the nodes update their IP tables to send traffic to the new pods instead of the old. Traffic will be directed to the old pods if the nodes have not been updated yet during this delay.In order to prevent the packets from being sent to a node that killed the old pod, the rollout uses the scaleDownDelaySeconds field to give nodes enough time to broadcast the IP table changes. If omitted, the Rollout waits 30 seconds before scaling down the previous ReplicaSet.

It’s recommended to set scaleDownDelaySeconds to a minimum of 30 seconds in order to ensure that the IP table propagation across the nodes in a cluster. The reason is that Kubernetes waits for a specified time called the termination grace period. By default, this is 30 seconds.

Resources: read more

8. Ensure retry on both Error and TransientError

Project: Argo Workflows

Best practice: retryStrategy is an optional field of the Workflow CRD, that provides controls for retrying a workflow step. One of the fields of retryStrategy is retryPolicy, which defines the policy of NodePhase statuses that will be retried (NodePhase is the condition of a node at the current time).The options for retryPolicy can be either: Always, OnError, or OnTransientError. In addition, the user can use an expression to control more of the retries.

What’s the catch?

retryPolicy=Always is too much. The user only wants to retry on systcode-level errors (eg, the node dying or being precodepted), but not on errors occurring in user-level code since these failures indicate a bug. In addition, this option is more suitable for long-running containers than workflows which are jobs.
retryPolicy=OnError doesn't handle precodeptions: retryPolicy=OnError handles some systcode-level errors like the node disappearing or the pod being deleted. However, during graceful Pod termination, the kubelet assigns a Failed status and a Shutdown reason to the terminated Pods. As a result, node precodeptions result in node status "Failure", not "Error" so precodeptions aren't retried.
retryPolicy=OnError doesn't handle transient errors classifying a precodeption failure message as a transient error is allowed however, this requires retryPolicy=OnTransientError. (see also, TRANSIENT_ERROR_PATTERN).

We recommend to set retryPolicy: "Always" and use the following expression: 'lastRetry.status == "Error" or (lastRetry.status == "Failed" and asInt(lastRetry.exitCode) not in [0])'

9. Ensure progressDeadlineAbort set to true, especially if progressDeadlineSeconds has been set

Project: Argo Rollouts

Best practice: A user can set progressDeadlineSeconds which states the maximum time in seconds in which a rollout must make progress during an update before it is considered to be failed.

If rollout pods get stuck in an error state (e.g. image pull back off), the rollout degrades after the progress deadline is exceeded but the bad replica set/pods aren't scaled down. The pods would keep retrying and eventually the rollout message would read ProgressDeadlineExceeded: The replicaset <name> has timed out progressing. To abort the rollout, the user should set both progressDeadlineSeconds and progressDeadlineAbort, with progressDeadlineAbort: true

Resources: read more

10. Ensure custom resources match the namespace of the ArgoCD instance

Project: Argo CD

Best practice: In each repository, all Application and AppProject manifests should match the same metadata.namespace. The reason why is actually dependent on how you installed Argo CD.

If you deployed Argo CD in the typical deployment, under the hood Argo CD creates two ClusterRoles and ClusterRoleBinding, that reference the argocd namespace by default. In this case, it’s recommended not only to ensure all Argo CD resources match the namespace of the Argo CD instance, but also to use the argocd namespace because otherwise, you need to make sure to update the namespace reference in all Argo CD internal resources.

However, if you deployed Argo CD for external clusters (in “Namespace Isolation Mode”) then instead of ClusterRole and ClusterRoleBinding, Argo creates Roles and associated RoleBindings in the namespace where Argo CD was deployed. The created service account is granted a limited level of access to manage, so for Argo CD to be able to function as desired, access to the namespace must be explicitly granted. In this case, it’s recommended to make sure all the resources, including the Application and AppProject, use the correct namespace of the ArgoCD instance.

Resources: read more

So...Now What?

I’m a GitOps believer, I believe that every Kubernetes resource should be handled exactly the same as your source code, especially if you are using helm/kustomize. So, the way I see it, we should automatically check our resources on every code change.

You can write your policies using languages like Rego or JSONSchcodea and use tools like OPA ConfTest or different validators (for example, ‣) to scan and validate our resources on every change. Additionally, if you have one GitOps repository then Argo plays a great role in providing a centralized repository for you to develop and version control your policies.

However, writing policies might be a pretty challenging task on its own, especially with Rego.

Another way would be to look for tools like ‣ which already comes with predefined policies, YAML schcodea validation, and best practices for Kubernetes and Argo.

How Datree works

The Datree CLI runs automatic checks on every resource that exists in a given path. After the check is completed, Datree displays a detailed output of any violation or misconfiguration it finds, with guidelines on how to fix it:

How to build a Helm plugin in minutes

Roman Labunsky — Wed, 16 Jun 2021 13:39:58 +0000

Introduction

Helm is a great addition to the Kubernetes ecosystem; it simplifies complex Kubernetes manifests by separating them into charts and values.

Sharing charts has never been easier especially since all the customizable parameters are located separately (values.yaml). The downside of this is that there’s no single place to see the resulting manifest, as it’s usually compiled and installed in a single step - helm install.

Helm alleviates this with a plugin system that allows you to seamlessly integrate with the Helm flow and easily run custom code that’s not part of the Helm core code which, as we’ll see in the next section, can be written in any language, even Bash.

A simple example of how to create a Helm plugin

Let's build a simple plugin that prints out some useful information and environment variables that Helm provides.

A helm plugin consists of a single required file - plugin.yaml

That’s the entrypoint for Helm. By using this, Helm knows the settings for your plugin, the command it executes when the plugin is run and hooks for customizing the plugin lifecycle (more on that later).

This is the only required file for a complete plugin. You can set it up in 3 steps:

Create a plugin.yaml with the following content
Run the installation command from the directory of the file
```
helm plugin install .
```
Execute the plugin
```
helm myplugin
```

A complete example of a Helm plugin

The simple plugin example covers the most basic use cases, goals that can be achieved even with an alias. More complex cases require more points of integration with Helm and greater customizability of the plugin’s lifecycle. The following parts will give you all the necessary tools to execute any logic in the Helm flow.

Lifecycle customization - Install, update and delete hooks

Helm provides under-documented (the closest thing I found to documentation was here) capability for hooking into the plugin’s install, update or uninstall commands.

Each hook corresponds to a command and will execute the provided script when invoked.

Some plugins may require a more complex installation flow: Downloading a binary based on OS architecture, building or compiling code or simply installing system dependencies. The install script is the place to do it. A useful example can be found here.

Execution integration

There are two ways to specify what will be executed and when:

command

platformCommand

It's important to note that Helm has a hierarchy for choosing the correct command:

platformCommand(os+arch match)
platformCommand(os only match)
command

Helm plugins aren’t executed in a shell, so complex commands must be part of a script that will be executed every time the plugin is invoked.

A script can be used to run complex logic, handle parameter parsing, etc.

command: "$HELM_PLUGIN_DIR/scripts/run.sh"

A comprehensive example of a run script can be found here. A very useful thing that we can do is to render the chart and then execute logic on the resulting Kubernetes yaml file:

helm template "${HELM_OPTIONS[@]}" > ${TEMP_MANIFEST_NAME}

$HELM_PLUGIN_DIR/bin/myplugin ${TEMP_MANIFEST_NAME} "${MYPLUGIN_OPTIONS[@]}"

A very important flag in plugin.yaml is:

ignoreFlags: false

This flag specifies whether or not the command line params are passed to the plugin or not. If the plugin accepts parameters, this should be set to false.

Tips and caveats

Helm exposes many environment variables that can simplify a lot of the complex logic and provides much of the necessary information and context for the plugin’s execution.
The useTunnel flag in config.yaml is deprecated in Helm V3 and is no longer needed
To easily test the plugin during development, you can install the plugin from the dev directory by running helm plugin install PATH_TO_DIR
The plugin can then be uninstalled with helm plugin uninstall PLUGIN_NAME

Conclusion

Helm plugin writing is easy to learn but difficult to master. Lack of documentation makes writing complex plugins an arduous task.

This article aims to expose some of the lesser known abilities of the Helm plugin system and to provide tools and scaffolding that remove the limitations of the plugin system and allow execution of as complex a business logic as is necessary to extend Helm’s behavior.

A Deep Dive Into Kubernetes Schema Validation

Eyar Zilberman — Tue, 01 Jun 2021 10:24:16 +0000

Why run schema validation?

How do you ensure the stability of your Kubernetes clusters? How do you know that your manifests are syntactically valid? Are you sure you don’t have any invalid data types? Are any mandatory fields missing?

Most often, we only become aware of these misconfigurations at the worst time - when trying to deploy the new manifests.

Specialized tools and a “shift-left” approach make it possible to verify a Kubernetes schema before they’re applied to a cluster. In this article, I'll address how you can avoid misconfigurations and which tools are best to use.

TL;DR

Running schema validation tests is important, and the sooner the better.

If all machines (local developers environment, CI, etc.) have access to your Kubernetes cluster, run kubectl --dry-run in server mode on every code change. If this isn’t possible, and you want to perform schema validation tests offline, use kubeconform together with a policy enforcement tool to have optimal validation coverage.

Available tools

Verifying the state of Kubernetes manifests may seem like a trivial task, because the Kubernetes CLI (kubectl) has the ability to verify resources before they’re applied to a cluster. You can verify the schema by using the dry-run flag (--dry-run=client/server) when specifying the kubectl create or kubectl apply commands, which will perform the validation without applying Kubernetes resources to the cluster.

But I can assure you that it’s actually more complex. A running Kubernetes cluster is required to obtain the schema for the set of resources being validated. So, when incorporating manifest verification into a CI process, you must also manage connectivity and credentials to perform the validation. This becomes even more challenging when dealing with multiple microservices in several environments (prod, dev, etc.).

Kubeval and kubeconform are command-line tools that were developed with the intent to validate Kubernetes manifests without the requirement of having a running Kubernetes environment. Because kubeconform was inspired by kubeval, they operate similarly — verification is performed against pre-generated JSON schemas that are created from the OpenAPI specifications (swagger.json) for each particular Kubernetes version. All that remains to run the schema validation tests is to point the tool executable to a single manifest, directory or pattern.

Comparing

kubeval
kubeconform
kubectl dry-run in ‘client’ mode
kubectl dry-run in ‘server’ mode

Now that we covered the tools that are available for Kubernetes schema validation, let’s compare some core abilities (misconfigurations coverage, speed test, different versions support, CRD support and docs).

Misconfigurations coverage¹

I donned my QA hat and generated some (basic) Kubernetes manifest files with some intended misconfigurations, and then ran it against all four tools²:

Misconfig/Tool	kubeval / kubeconform	kubectl dry-run in ‘client’ mode	kubectl dry-run in ‘server’ mode
API deprecation	✅ Caught	✅ Caught	✅ Caught
Invalid kind value	✅ Caught	❌ Didn't catch	🚧 Caught³
Invalid label value	❌ Didn't catch	❌ Didn't catch	✅ Caught
Invalid protocol type	✅ Caught	❌ Didn't catch	✅ Caught
Invalid spec key	✅ Caught	✅ Caught	✅ Caught
Missing image	❌ Didn't catch	❌ Didn't catch	✅ Caught
Wrong K8s indentation	✅ Caught	✅ Caught	✅ Caught

Conclusion: Running kubectl dry-run in ‘server’ mode caught all misconfigurations, while kubeval/kubeconform missed two of them. It’s also interesting to see that running kubectl dry-run in ‘client’ mode is almost useless because it’s missing some obvious misconfigurations, and also requires a connection to a running Kubernetes environment.

Benchmark speed test

I used hyperfine to benchmark the execution time of each tool⁴. First I ran it against (1) all the files with misconfigurations (seven files in total), and then I ran it against (2) 100 Kubernetes files (all the files contain the same config).

(1) Results for running the tools against seven files with different Kubernetes schema misconfigurations:

(2) Results for running the tools against 100 files with valid Kubernetes schemas:

Conclusion: We can see that while kubeconform (#1), kubeval (#2) and kubectl --dry-run=client (#3) are providing fast results on both tests, while kubectl --dry-run=server (#4) is working slower, especially when it needs to evaluate 100 files — 60 seconds for generating a result is still a good outcome in my opinion.

Kubernetes versions support

Both kubeval and kubeconform accept the Kubernetes schema version as a flag. Although both tools are similar (as mentioned, kubeconfrom is based on kubeval), one of the key differences between them is that each tool relies on its own set of pre-generated JSON schemas:

Kubeval - instrumenta/kubernetes-json-schema (last commit: 133f848 on April 29, 2020)
Kubeconform - yannh/kubernetes-json-schema (last commit: a660f03 on May 15, 2021)

As of today (May 2021), kubeval only supports Kubernetes schema versions up to 1.18.1, while kubeconform supports the latest Kubernetes schema available today — 1.21.0. With kubectl, it’s a little bit trickier. I don’t know which version of kubectl introduced the dry-run, but I tried it with Kubernetes version 1.16.0 and it still worked, so I know it’s available in Kubernetes versions 1.16.0-1.18.0.

The variety of Kubernetes schemas support is especially important if you want to migrate to a new Kubernetes version. With kubeval and kubeconform you can set the version and start the process of evaluating which configurations must be changed to support the cluster upgrade.

Conclusion: The fact that kubeconform has all the schemas for all the different Kubernetes versions available — and also doesn’t require minikube setup (as kubectl does) — makes it a superior tool when comparing these capabilities to its alternatives.

Other things to consider

Custom Resource Definition (CRD) support
Both kubectl dry-run and kubeconform support resource type CRD, while kubeval does not. According to kubeval docs, you can pass a flag to kubeval to ignore missing schemas, so it will not fail when testing a bunch of manifests for which only some are resource type CRD.

Documentation
Kubeval is a more popular project than kubeconform, and therefore, its community and documentation are more extensive. Kubeconform doesn't have official docs but it does have a well-written README file that explains pretty well its capabilities. The interesting part is that although Kubernetes native tools, like kubectl, are usually well-documented, it was really hard to find the necessary information needed to understand how the dry-run flag actually works and its limitations.

Conclusion: Although it’s not as famous as kubeval, the CRD support and good-enough documentation make kubeconform the winner in my opinion.

Comparison summary

Item/Tool	kubeval	kubeconform	dry-run client	dry-run server
Misconfigurations coverage	+/-	+/-	-	+
Benchmark speed test	+/-	+	+/-	-
Kubernetes versions support	-	+	+/-	+/-
CRD support	-	+	+	+
Documentation	+	+/-	-	-

Now that you know the pros and cons associated with each tool, here are some best practices for how to best leverage them within your Kubernetes production-scale development flow.

Strategies for validating Kubernetes schema using these tools

⬅️ Shift-left: When possible, the best setup is if you can run kubectl --dry-run=server on every code change, but you probably can’t do it because you can’t allow every developer or CI machine in your organization to have a connection to your cluster. So, the second-best effort is to run kubeconform.
🚔 Because kubeconform doesn’t cover all common misconfigurations, it’s recommended to run it with a policy enforcement tool on every code change to fill the coverage gap.
💸 Buy vs. build: If you enjoy the engineering overhead, then kubeconform + conftest is a great combination of tools to get good coverage. Alternatively, there are tools that can provide you with an out-of-the-box experience to help you save time and resources, such as Datree⁵ (whose schema validation is powered by kubeconform).
🚀 During the CD step, it shouldn’t be a problem to have a connection with your cluster, so you should always run kubectl --dry-run=server before deploying your new code changes.
👯 Another option for using kubectl dry-run in server mode, without having a connection to your Kubernetes environment, is to run minikube + kubectl --dry-run=server. The downside of this hack is that it’s also required to set up the minikube cluster like prod (same volumes, namespace, etc.) or you’ll encounter errors when trying to validate your Kubernetes manifests.

GRATITUDE

Thank you to Yann Hamon for creating kubeconform - it’s awesome!
This article wouldn’t be possible without you. Thank you for all of your guidance.

All the schemas validation tests performed against Kubernetes version 1.18.0 ↩
Because kubeconform is based on kubeval, they provide the same result and run them against the files with the misconfigurations. kubectl is one tool but each mode (client or server) produces a different result as you can see from the table ↩
Server mode didn’t mark the file as valid (exit code 1) but the error message is wrong: Kind=pod doesn't support dry-run ↩
All benchmark test performed on my MacBook Pro with a 2.3 GHz Quad-Core Intel Core i7 processor ↩
Disclaimer - self-promotion here :) ↩

10 insanely useful Git commands you wish existed – and their alternatives

Eyar Zilberman — Tue, 23 Apr 2019 13:43:50 +0000

There’s a git command for that

Git commands aren’t always intuitive. If they were, we would have these 10 commands at our disposal. They would be super useful for accomplishing common tasks like creating or renaming a git branch, removing files, and undoing changes.

For each git command in our wishlist, we’ll show you the commands that actually exist and you can use to accomplish the same tasks. If you’re still learning Git, this list reads like a tutorial and is worth keeping as a cheatsheet.

# 9 – git create branch: create a new branch with git checkout

The fastest way to create a new branch is to actually do it from the git terminal. This way you don’t have to use GitHub UI, for example, if you use GitHub for version control.

This command actually exists in git, only in a different name – $ git checkout.

How to create a branch with git checkout:

One-line command: $ git checkout -b <branch-name> master

-> commit-test git:(master) $ git checkout -b feature-branch master
-> commit-test git:(feature-branch) $

Git tip: Just like with commit messages, having a naming convention for git branches is a good best practice to adopt.

# 8 – git force pull: overwrite local with git pull

You find out you’ve made changes that seemingly conflict with the upstream changes. At this point, you decide to overwrite your changes instead of keeping them, so you do a $ git pull and you get this error message:

-> commit-test git:(master) $ git pull
Updating db40e41..2958dc6
error: Your local changes to the following files would be overwritten by merge:
README.md
hint: Please, commit your changes before merging.
fatal: Exiting because of unfinished merge.

How to overwrite local changes with git pull:

Stash local changes: $ git stash
Pull changes from remote: $ git pull

-> commit-test git:(master) $ git stash
Updating db40e41..2958dc6
Saved working directory and index state WIP on master: d8fde76 fix(API): remove ‘test’ end-point
-> commit-test git:(master) $ git pull
Auto-merging README.md
Merge made by the ‘recurive’ strategy.
README.md     | 1 +
ENDPOINT.js    | 3 ++–
2 files changes, 3 insertions(+), 1 deletions(-)

Git tip: If you want to retrieve your changes just do: $ git stash apply

# 7 – git remove untracked files: delete untracked files from working tree

When having unnecessary files and dirs in your own local copy of a repository, and you want to delete those files, in opposed to just ignore them (with .gitignore), you can use git clean to remove all files which are not tracked by git.

How to remove untracked files and dirs:

Start with a dry-run to see what will be deleted: $ git clean -n -d
After you are sure, run the git clean command with “-f” flag: $ git clean -f -d

-> commit-test git:(master) $ git clean -n -d
Would remove dontTrackDir/untracked_file1.py
Would remove untracked_file2.py
-> commit-test git:(master) $ git clean -f -d
Removing dontTrackDir/untracked_file1.py
Removing untracked_file2.py

Git tip: Instead of untracking files, a good practice is to prevent those files from being tracked in the first place by using .gitignore file.

# 6 – git unstage: unstage file(s) from index

When you’re adding files ($ git add) to the working tree, you are adding them to the staging area, meaning you are staging them. If you want Git to stop tracking specific files on the working tree, you need to remove them from your stage files (.git/index).

How to unstage file(s) from index:

Keep the file but remove it from the index: $ git rm --cached <file-name>

-> commit-test git:(master) $ git rm –cached unstageMe.js
rm unstageMe.js

To leave the entire working tree untouched, unstage all files (clear your index): $ git reset

-> commit-test git:(master) $ git reset

Git tip: you can also untrack files which already added to git repository based on .gitignore.

# 5 – git undo merge: abort (cancel) a merge after it happened

Sometimes you get in a situation (we’ve all been there) where you merged branches and realize you need to undo the merge because you don’t want to release the code you just merged.

How to abort (cancel) a merge and maintain all committed history:

Checkout to the master branch: $ git checkout master
Run git log and get the id of the merge commit: $ git log --oneline
Revert merge by commit id: $ git revert -m 1 <merge-commit-id>
Commit the revert and push changes to the remote repo. You can start putting on your poker face and pretend “nothing’s happened”.

-> commit-test git:(master) $ git log –oneline
812d761 Merge pull request #524 from datreeio/DAT-1332-resolve-installation-id
b06dee0 feat: added installation event support
8471b2b fix: get organization details from repository object

-> commit-test git:(master) $ git revert -m 1 812d761
Revert “Merge pull request #524 from datreeio/DAT-1332-resolve-installation-id”
[master 75b85db] Revert “Merge pull request #524 from datreeio/DAT-1332-resolve-installation-id”
1 file changed, 1 deletion(-)
-> commit-test git:(master) $ git commit -m “revert merge #524”
-> commit-test git:(master) $ git push

Git tip: Instead of reverting merge, working with pull requests and setting up or improving your code review process can lower the possibility of a faulty merge.

# 4 – git remove file: remove file(s) from a commit on remote

You wish to delete a file (or files) on remote, maybe because it is deprecated or because this file not supposed to be there in the first place. So, you wonder, what is the protocol to delete files from a remote git repository?

How to remove file(s) from commit:

Remove your file(s): $ git rm <file-A> <file-B> <file-C>
Commit your changes: $ git commit -m "removing files"
Push your changes to git: $ git push

-> commit-test git:(delete-files) $ git rm deleteMe.js
rm ‘deleteMe.js’
-> commit-test git:(delete-files) $ git commit -m “removing files”
[delete-files 75e998e] removing files
1 file changed, 2 deletions(-)
delete mode 100644 deleteMe.js
-> commit-test git:(delete-files) $ git push

Git tip: When a file is removed from Git, it doesn’t mean it is removed from history. The file will keep “living” in the repository history until the file will be completely deleted.

# 3 – git uncommit: undo the last commit

You made a commit but now you regret it. Maybe you committed secrets by accident – not a good idea – or maybe you want to add more tests to your code changes. These are all legit reasons to undo your last commit.

How to uncommit (undo) the last commit:

To keep the changes from the commit you want to undo: $ git reset --soft HEAD^
To destroy the changes from the commit you want to undo: $ git reset --hard HEAD^

-> commit-test git:(undo-commit) $ git commit -m “I will regret this commit”
[undo-commit a7d8ed4] I will regret this commit
1 file changed, 1 insertion(+)
-> commit-test git:(undo-commit) $ git reset –soft HEAD^
-> commit-test git:(undo-commit) $ git status
On branch undo-commit
Changes to be committed:
(use “git reset HEAD <file>…” to unstage)

    modified: README.md

Git tip: Git pre-commit hook is a built-in feature that lets you define scripts that will run automatically before each commit. Use it to reduce the need to cancel commits.

# 2 – git diff between branches

When you are working with multiple git branches, it’s important to be able to compare and contrast the differences between two different branches on the same repository. You can do this using the $ git diff command.

How to get the diff between two branches:

Find the diff between the tips of the two branches: $ git diff branch_1..branch_2
Produce the diff between two branches from common ancestor commit: $ git diff branch_1...branch_2
Comparing files between branches: $ git diff branch1:file branch2:file

-> commit-test git:(diff-me) $ git diff master..diff-me
diff –git a/README.md b/README.md
index b74512d..da1e423 100644
— a/README.md
+++ b/README.md
@@ -1,2 +1,3 @@
# commit-test
-Text on “master” branch
+Text on “diff-me” branch

Git tip: diff-so-fancy is a great open source solution to make your diffs human readable.

# 1 – git delete tag: remove a tag from branch

In the case of a “buggy” release, you probably don’t want someone to accidentally use the release linked to this tag. The best solution is to delete the tag and remove the connection between a release and its co-related tag.

How to delete tag by removing it from branch:

If you have a remote tag to delete, and your remote is origin, then simply: $ git push origin :refs/tags/<tag-name>
If you also need to delete the tag locally: $ git tag -d <tag-name>

-> commit-test git:(delete-tag) $ git push origin :refs/tags/v1.0.0
To github.com:datreeio/commit-test.git
– [deleted]         v1.0.0
-> commit-test git:(delete-tag) $ git tag -d v1.0.0
Deleted tag ‘v1.0.0’ (was af4d0ea)

Git tip: Not sure when or why to use tags? Read here to learn more (TL;DR: automatic releasing)

# 0 – git rename branch: change branch name

As I mentioned, having a branch naming convention a good practice and should be adopted as part of your coding standards, and it is especially useful in supporting automation of git workflows. But what to do when you find out your branch name is not aligned with the convention, after already pushing code to the branch? Don’t worry, you can still rename your branch.

How to rename branch name after it was created:

Checkout to the branch you need to rename: $ git checkout <old-name>
Rename branch name locally: $ git branch -m <new-name>
Delete old branch from remote: $ git push origin :<old-name> <new-name>
Reset the upstream branch for the new branch name: $ git push origin -u <new-name>

-> commit-test git:(old-name) $ git branch -m new-name
-> commit-test git:(new-name) $ git push origin :old-name new-name
Total 0 (delta 0), reused 0 (delta 0)
To github.com:datreeio/commit-test.git
–  [deleted]             old-name
* [new branch]      new-name -> new-name
-> commit-test git:(new-name) $ git push origin -u new-name
Branch new-name set up to track remote branch new-name from origin.
Everything up-to-date

Git tip: Want to make sure all branch names will always follow your convention? Set a Git-enforced branch naming policy.

What's next?

Found some useful commands? you can also set alias commands for them!
Relevant alias to this blog post are provided by @mfrata in his comment - you can thank him :)

Originally posted on https://datree.io/resources/git-commands

Top 10 GitHub Best Practices

Eyar Zilberman — Mon, 15 Apr 2019 13:28:03 +0000

After scanning thousands of repositories and interviewing hundreds of GitHub, I created a list of common best practices which are strongly recommended to be adopted in every modern software development organization which is using GitHub to store their code:

9. 🚧 Protect the main branches from direct commits

Anything in the master branch should always be deployable, that’s why you should never commit to the default branches directly and why Gitflow workflow has become the standard. Using branch protection can help you prevent direct commits and of course, everything should be managed via pull requests.

8. 👻 Avoid unrecognized committers

Maybe you are working on a new environment, or you didn’t notice that your Git configuration is incorrect which causes the user to commit code with the wrong email address. Now, their commit is not associated with the right user and makes it nearly impossible to trace back who wrote what.

Make sure that your Git client is configured with the correct email address and linked to your GitHub user. Check your pull requests during code review for unrecognized commits.

7. 🎩 Define CODEOWNERS for each repository

Using the CODEOWNERS feature allows you to define which teams and people are automatically selected as reviewers for the repository. This ability automatically requests a review from the repository owners. Nowadays organizations have dozens if not hundreds of repositories and CODEOWNERS gives the option to define who the repo maintainers are across your organization.

6. 🙊 Separate secret credentials from source code

When building a Cloud Native app, there are many secrets — account passwords, API keys, private tokens, and SSH keys — that we safeguard. NEVER! commit any secrets into your code. Instead, use environment variables that are injected externally from a secure store.

You can use tools like Vault and AWS Secrets Manager to help with your secret management in production.

There are lots of great tools to identify existing secrets in your code and prevent new ones. For example, Git-secrets can help you to identify passwords in your code. With Git Hooks you can build a pre-commit hook and check every pull request for secrets.

5. ⛔ Avoid committing dependencies into your project

Pushing dependencies into your remote origin will increase repository size. Remove any projects dependencies included in your repositories and let your package manager download them in each build. if you are afraid of “dependencies availability” you should consider using a binary repository manager solution like Jfrog or Nexus Repository. Also, check out Git-Sizer by GitHub.

4. 🔧 Separate configuration files from source code

We strongly recommend against committing your local config files to version control. Usually, those are private configuration files which you don’t want to push to remote because they are holding secrets, personal preferences, history or general information which should stay only in your local environment.

📤 3. Create a meaningful .gitignore file for your projects

A .gitignore file is a must in each repository to ignore predefined files and directories. It will help you to prevent secret keys, dependencies and many other possible discrepancies in your code. You can choose a relevant template from Gitignore.io.

2. 💀 Archive dead repositories

Over time, for various reasons, we find ourself with unmaintained repositories. Maybe you opened a new repository for an ad hoc use case (or you wanted to POC a new tech) or you have some repositories with old and irrelevant code. The problem is the same – those repositories are not actively developed anymore after they served their purpose, so you don’t want to maintain them or other people will rely on/use them. The best practice will always be to archive those repositories which will make them “read-only” to everyone.

1. 🔒 Lock package version

Your manifest file holds the information about all of your packages versions to maintain consistent results without breaking your code every time you install your app dependencies. The best practice is to use a manifest lock file to avoid any discrepancies and confirm that you are getting the same packages version each time. The opposite being that you leave your code component version imprecise, are uncertain which version will be installed on the next build and your code may break.

0. ♻️ Align packages versioning

Although you are using the same package, a different version distribution will make it harder to reuse code and tests in various projects.

If you have a package which is used in multiple projects, try at a minimum to use the same major version across the different repositories.

🚀 What’s next?

All that’s left for you to do is check off each of the aforementioned best practices, on each of your repositories, one by one.

Or, save your sanity and connect with Datree’s GitHub app (it's even free!) to scan your repositories and generate your free status report to assess if your repositories align with the listed best practices.

A guide to GitHub Actions using Node.js for Git workflow automation

Roman Labunsky — Sun, 31 Mar 2019 18:09:26 +0000

What is Github Actions?

GitHub Actions, a feature announced in October 2018 during GitHub Universe, generated immense hype under the apt positioning as the “swiss army knife” of git workflow automation.

Github Actions allows developers to perform tasks automatically in a Github workflow, such as pushing commits to a repository, deploying a release to staging, running tests, removing feature flags, and so on, by way of a simple text file.

Chewy.com, for example, demoed an action that checks if a Jira ticket number is included in every pull request name among other things, to ensure the code being deployed to production is compliant with their policies and best practices.

“With GitHub Actions, you can automate your workflow from idea to production.”

– GitHub actions page

Write Github Actions in Node.js

I started tinkering with the feature as soon as I could get access to the private beta. I noticed that most Actions are written in shell script. GitHub itself promoted writing actions in shell script for simple actions. While I understand the motivation (so you can quickly and easily start writing Actions), I feel that shell scripts are limited in terms of writing full-fledged software.

Since I generally work in Node.js, I decided to ‘accept the challenge’ and write my first Action in Node. But, after struggling with the specifics of running Node.js actions and understanding the differences between a simple container and the GitHub execution environment, I decided to write this tutorial in the hope that it will help others who prefer to write an action in JavaScript.

Environment setup

The basic setup is straightforward – start by following GitHub’s tutorial all the way to the entrypoint.sh section. The main difference up to this point is in the chosen docker image. I suggest using the alpine image, it’s very lightweight compared to the regular node image. In any case, I suggest using an LTS variant, currently being node 10.

A very important tool, one that helped reduce the development cycle from 5 minutes per iteration to mere seconds is Act, a zero-config, easy to use tool to run actions locally. It doesn’t fully replicate the environment (for obvious reasons, it doesn’t provide a GitHub token, more on that later) but it’s close enough to speed up the development process and test your action locally.

At the end of this step, you should have a Dockerfile that looks like this:

Node-specific setup

This is the basic entrypoint.sh file that will work for a Node.js action:

I chose npm ci because it’s the easiest way to make sure you always get the same versions of the packages you want to install. It requires you to have package.json and package-lock.json in your project – but that’s a best practice anyway.

The installation script is in the entry point file and NOT in the Dockerfile (as is usual in classic container use cases) because it makes it much easier to use an npm token and install private packages. All you need to do is add an NPM_TOKEN secret and use it in the entry point file (above npm ci) by adding:

npm config set //registry.npmjs.org/:_authToken=$NPM_TOKEN
node script.js $* runs the script and passes the action args as arguments to the script.

Node script tips and basic structure

Over the years, I’ve developed a preferred structure for a node executable (CLI) script. I will share it here but for the purpose of this tutorial, this part is completely optional and at this stage, you’re more than ready to develop your own action in Node.js.

The script looks like this:

The important section is at the bottom: if (require.main === module)

It checks if the file was imported/required or if it’s the entrypoint into the program. This allows reusing the same module both programmatically and as a CLI tool.

If this is the entrypoint, I would then parse the command line arguments (using commander) passed in from entrypoint.sh. The arguments were injected into entrypoint.sh by GitHub from the workflow file through the container (more on that later).

I then invoke the main function. Since it’s an async function, I handle its return value with a then clause and handle failure with a catch clause.

It’s also useful to read the event, provided by GitHub, and use it in the script:

const event = JSON.parse(fs.readFileSync('/github/workflow/event.json', 'utf8'))

Secrets, Args, and the execution environment

The Actions environment takes some getting used to. Although GitHub provides great tutorials on all things workflow related, I wanted to mention:

GitHub provides many environment variables inside the container running the Action, but most of the information can also be retrieved from the event file (see section above).
Others are defined in the workflow file, while some are provided as part of the secrets mechanism in GitHub.

Secrets are pretty straightforward, you define them in the repo settings tab and then they’re exposed as environment variables inside the container.

Defining the secret in the repo settings tab:

Using the secret in your workflow file:

The only exception is the GitHub token, which you don’t need to define in the settings. The token is only exposed in the workflow file and GitHub will provide the token itself with these permissions.

Another important item to note is the mounted folder GitHub provides. It’s mounted under /github and provides a couple of useful things:

the event under /github/workflow/event.json and
the repo where the action runs under /github/workspace/REPO_NAME

More information on the mechanics of the mount can be found here.

This tutorial provides a good starting point for anyone who wants to create their first Node.js action. My action can be found here and the code for it here.

If you’re interested in learning more about how you can use Github Actions to automate git workflows, check out this webinar to watch how Shimon Tolts, Datree.io Co-founder, "built a CI/CD dev pipeline with Github Actions, Node.js, Docker, and AWS Fargate".

If you have any questions, corrections, or suggestions please comment below or contact me directly.

How to Get More Out of Your Git Commit Message

Eyar Zilberman — Thu, 28 Mar 2019 12:06:00 +0000

Git commit message

If you aren’t already, defining your project Git commit message convention is always on every developer’s “To-Do” list. Like flossing your teeth – everyone knows it’s necessary best practice for healthy gums and avoiding the dentist, but it ends up on the ‘I’ll move it to tomorrow’s to-do list’ aka, procrastination galore.

I’m going to break down the reasons why you really have NO excuse not to set a Git commit message convention (or floss), and the required steps in order to move this task from your “To-Do” list to “DONE” in a few simple steps!

I’ll leave your dentist to yell at you about not flossing 😁

Why use a commit message convention?

Better collaboration between potential and existing committers

It is important to communicate the nature of changes in projects in order to foster transparency to a slew of people: existing teammates, future contributors, and sometimes to the public and other stakeholders. It’s obvious why a well-formatted Git commit message convention is the best way to communicate context about a change to fellow developers (and their future selves) when requesting a peer code review. A commit message convention also makes it easier to explore a more structured commit history and to understand which notable changes have been made between each release (or version) of the project.

Squeeze the most out of git utilities

“$ git log” is a beautiful and useful snippet. A well-organized commit message history leads to more readable messages that are easy to follow when looking through the project history. Suddenly, navigating through the log output become a possible mission! Embracing a commit message convention will also help you properly use other git commands like git blame, git revert, git rebase, git shortlog and other git subcommands.

Support different automation tools

Automation, automation, automation. Once you know you can rely on a standardized Git commit message, you can start building a flow around it and leverage the power of automation to level-up your project development flow:

Automatic generation of CHANGELOG – keeps everyone up to date on what happened between releases.
Automatic bump ups to the correct version – determine a release semantic version based on the types of commits made per release.
Automatic triggers to other processes – you are only limited by your own imagination on this one. For example, you can decide that a predefined string in the commit message will trigger your CI.

Choosing the right commit message convention

Now that we know that having a commit message convention is useful whether you’re working on an open source project, working on your own, or working with your team on a single project, standardizing the Git commit message is the only right way to commit!

We covered the “why” part and now we will move to the “how” part – in my opinion, there are pretty much only two ways to go:

A. Adopt defacto best practices

This approach is a simple and easy guideline, good for getting used to the idea of having a convention/have a majority of coders or junior developers on the team. It’s the top 5 best practices to implement TODAY:

Have a commit message – white space or no characters at all can’t be a good description for any code change.
Keep a short subject line – long subjects won’t look good when executing some git commands. Limit the subject line to 50 characters.
Don’t end the subject line with a period – it’s unnecessary. Especially when you are trying to keep the commit title to under 50 characters.
Start with a capital letter – straight from the source: “this is as simple as it sounds. Begin all subject lines with a capital letter”.
Link to a planning system – if you are working with a planning system (like Jira), it is important to create a logical link between the planning ticket number and the subsequent code change.

B. Adopt an existing conventions framework

This approach is relevant for advanced/engaged teams, the key benefit of this approach is that you can also use the supporting tools in the ecosystem of the chosen conventions. There are plenty of different conventions so I will focus on the top two:

Angular Git commit message guidelines – well known and proven Git commit message convention which was introduced by the Angular project (A.K.A. Google).
Emoji Git commit message convention – I’m not kidding, it’s a thing. Incorporating emoji in the commit message is an easy way of identifying the purpose or intention of a commit at a glance, and of course, emoji are fun 😜. Because this convention is a philosophy and not a method, if chosen, I would recommend “Emoji-Log” commit message convention (by Ahmad Awais).

How to enforce Git commit message?

If you got this far, you probably agree with my opinion that every project should have a defined commit message convention. Now, the question is how can I make sure all the project committers (me, outside contributors and teammates) are aligned about the chosen convention and apply to it? My top two solutions for that are:

🔧 1. Git hooks

🚔 2. Server-side policy enforcement

Both options explained with a guide on my original blog post (I didn't add it here to keep this post short).

DEV Community: Datree

ArgoCD Best Practices You Should Know

Argo Best Practices

1. Disallow providing an codepty retryStrategy (i.e.{})

2. Ensure that Workflow pods are not configured to use the default service account

3. Ensure label part-of: argocd exists for ConfigMaps

4. Disable with DAG to set FailFast=false

5. Ensure Rollout pause step has a configured duration

6. Specify Rollout’s revisionHistoryLimit

7. Set scaleDownDelaySeconds to 30s to ensure IP table propagation across the nodes in a cluster

8. Ensure retry on both Error and TransientError

9. Ensure progressDeadlineAbort set to true, especially if progressDeadlineSeconds has been set

10. Ensure custom resources match the namespace of the ArgoCD instance

So...Now What?

How Datree works

How to build a Helm plugin in minutes

Introduction

A simple example of how to create a Helm plugin

A complete example of a Helm plugin

Lifecycle customization - Install, update and delete hooks

Execution integration

Tips and caveats

Conclusion

A Deep Dive Into Kubernetes Schema Validation

Why run schema validation?

TL;DR

Available tools

Comparing

Misconfigurations coverage1

Benchmark speed test

Kubernetes versions support

Other things to consider

Comparison summary

Strategies for validating Kubernetes schema using these tools

GRATITUDE

10 insanely useful Git commands you wish existed – and their alternatives

There’s a git command for that

# 9 – git create branch: create a new branch with git checkout

How to create a branch with git checkout:

# 8 – git force pull: overwrite local with git pull

How to overwrite local changes with git pull:

# 7 – git remove untracked files: delete untracked files from working tree

How to remove untracked files and dirs:

# 6 – git unstage: unstage file(s) from index

How to unstage file(s) from index:

# 5 – git undo merge: abort (cancel) a merge after it happened

How to abort (cancel) a merge and maintain all committed history:

# 4 – git remove file: remove file(s) from a commit on remote

How to remove file(s) from commit:

# 3 – git uncommit: undo the last commit

How to uncommit (undo) the last commit:

# 2 – git diff between branches

How to get the diff between two branches:

# 1 – git delete tag: remove a tag from branch

How to delete tag by removing it from branch:

# 0 – git rename branch: change branch name

How to rename branch name after it was created:

What's next?

Top 10 GitHub Best Practices

9. 🚧 Protect the main branches from direct commits

8. 👻 Avoid unrecognized committers

7. 🎩 Define CODEOWNERS for each repository

6. 🙊 Separate secret credentials from source code

5. ⛔ Avoid committing dependencies into your project

4. 🔧 Separate configuration files from source code

📤 3. Create a meaningful .gitignore file for your projects

2. 💀 Archive dead repositories

1. 🔒 Lock package version

0. ♻️ Align packages versioning

🚀 What’s next?

A guide to GitHub Actions using Node.js for Git workflow automation

What is Github Actions?

“With GitHub Actions, you can automate your workflow from idea to production.”

Write Github Actions in Node.js

Environment setup

Node-specific setup

Node script tips and basic structure

Secrets, Args, and the execution environment

How to Get More Out of Your Git Commit Message

Git commit message

3. Ensure label `part-of: argocd` exists for ConfigMaps

6. Specify Rollout’s `revisionHistoryLimit`

Misconfigurations coverage¹