DEV Community: langyizhao

Kotlin Scala Differences Cheat Sheet

langyizhao — Thu, 02 Dec 2021 15:57:02 +0000

Feature	Kotlin	Scala
Function Keyword	fun	def
Nullability	Default not nullable except by adding ? at the end of variable type	Null is discouraged, but no hard check
Null Check	Use ?. and/or ?:	Use Option
Lambda/Anonymous Functions	Need to be surrounded with {}	Can be used anywhere
Covariant	[out T]	[+T]
Contravariant	[in T]	[-T]
Pattern Matching	(partial) when statement	(full) match/case
Default Case	`else ->`	`case _ =>`
Loop a Range	i in 1..10 step 2 (closed) i in 1 until 10 step 2 (half-closed)	i <- 1 to 10 by 2 (closed) i in 1 until 10 by 2 (half-closed)
Read File into Strings	File(fileName).useLines { it.toList() }	scala.io.Source.fromFile("file.txt").mkString
For Comprehension (for/yield)	No Equivalent	Supported
Companion Object	`companion object` keyword inside class	`object` keyword besides class
New Immutable Sequence	listOf("one", "two", "one")	Seq("one", "two", "one")
New Immutable Set	setOf("one", "two", "three")	Set("one", "two", "one")
New Immutable Map	mapOf("key1" to 1, "key2" to 2, "key3" to 3, "key4" to 1)	Map("key1" -> 1, "key2" -> 2, "key3" -> 3, "key4" to 1)
Top Level Functions (Package Level)	Supported	Will be supported in Scala 3
Extension Functions	fun Any?.toString(): String { if (this == null) {return "null"} return toString()}	extension (c: Circle) def circumference: Double = c.radius * math.Pi * 2
Operator Overloading	with `operator infix` keyword	operators are just normal functions
Regex String	"[0-9]".toRegex()	"[0-9]".r
Tuple 2	Pair(a, b)	(a, b)
Tuple 3	Triple(a, b, c)	(a, b, c)
Destructuring Assignment	Objects with `componentN()` (data classes and collections)	Objects with `unapply()` (extractors like case classes)
Tail Recursive Function	`tailrec` keyword	`@tailrec` annotation
Multiple Inheritance Resolution	Must override function in question manually	Later trait wins
Class Inheritance	only with `open` or `abstract` keyword	allowed
The `sealed` keyword	sealed class is something like enum	sealed types are confined in its source file
Secondary/Auxiliary Constructors	`constructor` keyword	`this()` functions
Reference Equality (the Java `==`)	`===`	`eq`
Map Entry Literal	k `to` v	k `->` v

Helm Charts, with or without Kustomize

langyizhao — Thu, 22 Jul 2021 18:33:47 +0000

I've recently read an interesting blog about using Helm together with Kustomize.

Though it provides good information such as the difference between Templates (via Helm) vs Overlays (via Kustomize), I would argue that, in a real deployment, the same effect can be achieved with Helm alone without the extra complexity of Kustomize.

To avoid repeating the details in the original blog, I just provide the files to simulate the approach used by the original blog here: https://github.com/langyizhao/helm-charts-with-or-without-kustomize/tree/master/with-kustomize

The distilled bare minimum steps are:

kustomize build
helm template
kubectl apply -k

3 steps are actually not too bad. The major clutter is the introduction of the temporary folder and temporary template files and the easily-confusing order of applying overlays.

So how do we achieve the effect of overlays by just using templates? We will need some logic when drafting the template. Fortunately the Go template language under the hood of Helm charts easily support such trivial logic and some non-trivial ones.

One possible implementation is:

# from without-kustomize/templates/pod.yaml
{{- if and .Values.employeeName .Values.employeeDepartment }}
    args: ["/bin/echo My name is {{ .Values.employeeName }}. I work for {{ .Values.employeeDepartment }} department. Our company name is {{ .Values.companyName}}"]
{{ else }}
    args: ["/bin/echo Hello! My company name is {{ .Values.companyName }}"]
{{ end }}

In this case we will only render the complex form when the necessary values exists in the values.yaml file provided by our customer and safely default to the simple form when they are missing.

No pre or post rendering needed and Helm alone handles everything.

Feel free to check the simple examples out and try it yourself:

langyizhao / helm-charts-with-or-without-kustomize

This is the example code for https://dev.to/langyizhao/helm-charts-with-or-without-kustomize-2f1j

Example 1: helm template without-kustomize -f values1.yaml

Expected last line: args: ["/bin/echo Hello! My company name is ABC Company"]

Example 2: helm template without-kustomize -f values2.yaml

Expected last line: args: ["/bin/echo My name is Gary. I work for Marketing department. Our company name is ABC Company"]

View on GitHub

Troubleshooting K8s Ingress Issues

langyizhao — Tue, 01 Jun 2021 19:31:34 +0000

Most K8s resource creation failures are easy to debug, usually you can figure them out with either kubectl get or kubectl describe command on the failed resource.

However Ingress resources are often cluster-provider-specific, more often than not the errors are not explicit.

EKS Ingress Troubleshooting

`kubectl get ingress` doesn't show the ingress you just created

It might have been created in another namespace than you expected. Try again with --all-namespaces flag.

`kubectl get ingress` shows no ADDRESS after several minutes of resource creation

Try the AWS official example:

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.2.0/docs/examples/2048/2048_full.yaml
kubectl get ingress/ingress-2048 -n game-2048 --watch

If 1 works in a few minutes, the problem might be in your own manifest file.
If no ADDRESS is assigned for 2048 either, try kubectl logs -n kube-system deployment.apps/aws-load-balancer-controller
If the error message is Error from server (NotFound): deployments.apps "aws-load-balancer-controller" not found, follow https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html to deploy the controller.

(Will add more contents for other providers later)

References:

https://docs.aws.amazon.com/eks/latest/userguide/alb-ingress.html

K8s Ingress vs. OpenShift Route

langyizhao — Fri, 26 Mar 2021 00:07:17 +0000

This article aims to clear some confusion between Ingress and Route, and hopefully enable you to take advantage of the power of both.

Recently starting to deal with OpenShift clusters, I found most of the resource types in OCP are the native K8s resources I’m familiar with. Most of them, except for Route.

Ingress

Ingress is the native API object K8s defined to enable external access into services in the cluster.

It is THE resource type people usually expect to see in AKS, GKE, EKS, or in-house K8s installations when they connect their service to the vast internet.

A bare Ingress resource YAML manifest is like an interface, where the actual behavior heavily depends on how it is provisioned in your cluster - more specifically, the Ingress Conroller backing it.

For example, if your cluster uses ALB to back your Ingress (which is the most-likely case for AWS EKS), you will need some alb-specific annotations like this as part of the official example:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: IP

Normally when you try to use Ingress you would want to:

Find the Ingress Controller installed in your cluster.
If none-installed, look up one that's supported by your cloud provider (including private/hybrid) and install it.
Add the annotation based on the manual of that controller.

Route

Route, on the other hand, is not a native K8s thing, it only exists in the OpenShift world.

It depends on another Openshift concept - Router. Consider Router as plugins deployed on your cluster nodes that enables configurable routings. By routings, I mean traffic from external clients to in-cluster application services.

Note that a Router is installed on a node, so to reach the Router you need to reach the node first.

For example, if an HAProxy Router is used, external traffic is expected to go to the particular node hosting the HAProxy, then being routed to the service/pod.

According to the dev guide, there are 3 ways to create a Route.

The Openshift web console UI
oc CLI, such as oc expose svc/frontend --hostname=www.example.com
YAML manifest, such as

apiVersion: v1
kind: Route
metadata:
  name: frontend
spec:
  host: www.example.com
  path: "/test" 
  to:
    kind: Service
    name: frontend

Like Ingress, Route also supports annotations, for example router.openshift.io/cookie_name="my_cookie". But note that the annotations have different prefixes than those of Ingress resources, thus their annotations are not interchangeable.

However, the main method to control the behavior of a Route, rather than through annotations, is by changing the Router Environment Variables.

Ingress to Route

Starting from Openshift 3.10, Ingress objects created in Openshift can be translated into Route objects.

If you know the difference between Ingress and routes well, this is very convenient.

On the other hand, however, it can easily mislead people trying to take advantage of this translation.

As described in the section above, Routes:

Doesn't respect Ingress annotations. As a result, your fine-tuned annotations on your Ingress manifest file would NOT affect the true behavior of the created Route.
Needs traffic to first reach the node. So you will need to configure DNS records to Router nodes separately.

According to the dev guide:

DNS resolution for a host name is handled separately from routing; your administrator may have configured a cloud domain that will always correctly resolve to the OpenShift Container Platform router, or if using an unrelated host name you may need to modify its DNS records independently to resolve to the router.

This has to be taken into consideration and worked upon with the cluster admin for anyone familiar with K8s Ingress but not Openshift Route.

Infrastructure as Code: the 5 Questions to Ask before You Start

langyizhao — Mon, 15 Mar 2021 15:41:42 +0000

According to Wikipedia, infrastructure as code (IaC) is the process of managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools.

Although I don’t think I can define it any better, I believe that “as code” implies slightly more than those definition files. Especially since over the years, we as developers have accumulated so many best practices and principles dealing with code, there is no reason we can’t apply those experiences to the infrastructure if we make it as code.

Question 0: Should I consider Infrastructure as Code?

Consider this a bonus question, but an important one before embarking on any path towards IaC. Let’s assume that if you are reading this article, you are either considering IaC or more likely than not, already starting your journey.

Infrastructure as Code (IaC) is an inevitable phase of the DevOps movement. Automation is a cornerstone of DevOps culture [1], and after you finally automated your codebase with Version Control and CICD (or at least CI) pipelines, manual maneuvers of the underlying structure would feel so out of the place. People may call it with different names, like playbooks, cookbooks, manifests, and templates, but they are all the same as infrastructure code.

One common objection to Infrastructure as Code (or perhaps any automation) is "We don’t make changes often enough to justify automating them" [2].

Intuitively, it may feel true that changes happen much less often at the infrastructure level than at the code level.

But wait a moment, have you considered configuration changes? How about security patches (like updating to the latest AMI if you are using AWS)? Scaling up your cluster before Black Friday? Have you ever restored your database from a snapshot?

These are all changes related to infrastructure and can be made easier with IaC.

Even when infrastructure changes are infrequent, there are many additional benefits of stack created via IaC. For example, it takes much less effort to fully control your resources (e.g. cleaning it up completely or spinning it up again later), thus ideal for any POC type of work, especially in cloud environments. I've seen teams unknowingly kept huge orphan EC2 and RDS instances running forever, contributing steady revenues to Amazon, only because they started them manually and lost the due observability very quickly.

So from our perspective, the answer to whether you should consider IaC (even if you don’t adopt it) is most definitely yes.

Question 1: Infrastructure as Code or no Infrastructure at all (a.k.a. serverless)?

It is beneficial to have your infrastructure as code. But sometimes less is more, and you don't even need your own infrastructure - your cloud provider takes care of it for you under Function-as-a-Service (FaaS), or serverless.

I am a heavy user of AWS Lambda Functions because some tasks are so decoupled from other services that they are literally nothing but functions.

A simple scheduled job to send out notifications to either humans or external systems? A good candidate for a serverless function. Of course, you can have a cron job in an EC2 instance but that would be crazy to have its own instance (a waste of resource and a burden for maintenance), and probably too coupled to have the instance shared with many other unrelated jobs.

A filter to route important info via a webhook of your log aggregator to your Slack? Good for serverless too. You can write, test, and even deploy one in an hour and forget about it.

A popular cloud-agnostic choice of tooling is the Serverless Framework. Regarding languages, Go, Python, and Node.js all work very well. C#, Java (or JVM languages using the Java runtime) are not preferred for AWS Lambda because of the infamous cold-starts [3].

My own rule of thumb is if I can handle a task with no more than 10 files of source code (excluding external libraries) in Python or Node.js, I would consider using FaaS. Avoid being too obsessed: The tendency of splitting services small enough to fit into functions can drive microservices into nano services, which is considered an anti-pattern by many [4].

All being said, investing in serverless options for a subset of your services doesn't contradict the codification of your infrastructure in the big picture. You still can consider a not-so-standalone serverless function as a part of your infrastructure, connected with other parts of your system by API calls or streaming (SQS, Kinesis, Kafka, etc.)

Question 2: To Terraform or not to Terraform?

Among the Infrastructure as Code tools, Terraform popularity has undeniably grown in the last few years. Two of the four teams I most recently worked with were using Terraform as the primary tool (combining with other tools such as Kubernetes), with one other using plain CloudFormation and another completely going serverless. To decide whether Terraform is the right choice for your infrastructure, we can try to start with some comparisons.

Terraform vs. Cloud-specific Options

Every major cloud provider invented their own template format that you can use to draft your infrastructure code such as ARM Template from Azure and CloudFormation from AWS.

A very obvious advantage of using these formats is that no one can be more familiar with their cloud than the people developing them, thus you can expect that those templates reflect all of the latest cool features the cloud provider has added or is planning to add. If using cloud-agnostic options, you would sometimes have to wait for the community to come up with a solution to provision something only offered by a specific cloud provider. Tools with a larger community and ecosystem, like Terraform, would likely react faster than those with a smaller community.

On the other hand, using the template from a certain cloud provider would inevitably result in vendor lock-in. You can't run your CloudFormation on GCP or Azure, and it is usually harder than it looks to translate among templates.

Using cloud-agnostic tools like Terraform makes it theoretically easier to switch among cloud providers. Some companies run stacks on multi-cloud with intentional data redundancy in each cloud because, in reality, you can't escape vendor lock-in if that vendor holds all your data.

Besides the concern of vendor lock-in, Terraform is also useful for the polycloud strategy, which is slightly different from multi-cloud. Polycloud implies you leverage components from different cloud providers in the same infrastructure stack. As an extreme case, you want a GCP Compute Engine instance to run some Azure Machine Learning task and save the result into AWS S3. Using Terraform with 3 providers configured is a much saner choice than writing one template for each cloud.

Terraform vs. Other Cloud-agnostic Options

Excluding the cloud-specific options still leaves a lot of options out there. Big players including Chef, Puppet, and Ansible (though some may argue that they are more server configuration tools than infrastructure creating tools [2]), and my favorite is Kubernetes (commonly stylized as K8s).

Kubernetes Is incredibly powerful and has an even larger ecosystem than Terraform. However, it might be unfair to compare them because Kubernetes is a container orchestration system that happens to meet the need of Infrastructure as Code and Terraform are scoped much narrowly.

In Kubernetes, you define all resources in your infrastructure via manifest files (usually in the form of YAML), and the control plane manages them for you, almost magically. You need to know even fewer details about the underlying hardware than Terraform to provision new stacks, within an established K8s cluster. And it's much less likely to mess things up if role-based access control (RBAC) was configured correctly in that cluster.

However setting up a Kubernetes cluster, especially for the first time, can be very hard. It would be easier if your cloud provider gives you managed options such as AWS EKS or GCP GKE, but still pretty involved to get it configured properly.

OpenShift is a similar option to Kubernetes, and in fact, it is running K8s underneath and provides an extra abstraction layer. Some may find it useful, for me the vanilla K8s provides just the right level of abstraction. Yet another layer feels a little redundant and doesn't adjust the new moving parts introduced.

Besides these established options there have also emerged many newer, smaller players that you may be interested in. Pulumi, enables you to write your infrastructure code with the same languages you would usually use for your application code, such as Python, JS, TS, and Go, in case you are not a fan of the declarative DSL models used by most tools.

Other Pros and Cons of Terraform

The module system is a major strength of Terraform that doesn't exist in many cloud-agnostic or cloud-specific options natively. Not only it dramatically increases the reusability of your infrastructure code, but you also get the invaluable opportunity to distill the difference between the different environments in just a couple of files.

.
├── modules
│   ├── backend
│   │   ├── main.tf
│   │   ├── user_data.sh
│   │   └── variables.tf
│   ├── documentDB
│   │   ├── main.tf
│   │   └── variables.tf
│   ├── frontend
│   │   ├── install_docker_compose.sh
│   │   ├── main.tf
│   │   ├── routings.tf
│   │   └── variables.tf
│   ├── grafana
│   │   ├── main.tf
│   │   └── variables.tf
│   ├── postgreSQL
│   │   ├── main.tf
│   │   └── variables.tf
│   └── redis
│       ├── main.tf
│       └── variables.tf
├── production
│   ├── main.tf
│   └── variables.tf
├── qa
│   ├── main.tf
│   └── variables.tf
└── staging
    ├── main.tf
    └── variables.tf

In the above example, all the differences in configuration between each environment are strictly scoped in the stack level variables.tf files, and they would be very concise if your module-level variables.tf files have sensible defaults.

The biggest gripe about working with Terraform is the fragility of its "state file", which is terraform.tfstate by default. It is understandable that a state must be persisted somewhere to manage any dynamic infrastructure, but many tools abstract the trouble away from you. For example, CloudFormation provides drift detection as a service, through the console or API, and Kubernetes strives to be as stateless as possible.

Terraform leaves the duty of safekeeping the state of the stack to you, the applier of the infrastructure code. There are a few different ways of managing the state file, but the last thing you would want to do is to keep it publicly in your repo because anyone can open the file with a text editor and peek at the plaintext secrets inside. For AWS environments consider using the "s3" backend to store the state file. A properly secured private S3 bucket is required and a DynamoDB table is used for locking (the same bucket and lock table can be used for multiple stacks).

I highly recommend enabling the versioning feature on the cloud storage service containing your state file so you can restore your corrupted state file from the last working version. If you can't enable it or you store your state file with a local backend, you may want to back up the state file before applying anything nontrivial.

Another not-so-obvious issue of Terraform is the version compatibility. Terraform is not even 1.0 at the time of writing, which is more than 6 years after its initial release. Being sub-1.0, on the bright side, implies it's still rapidly evolving, but meanwhile, this means each version could break backward-compatibility and get forgiven. Hashicorp does provide you helper commands or tools to migrate, but from my experience upgrading many stacks either from 0.11 to 0.12 or 0.12 to 0.13, there is always a chance to miss something and corrupt your state file. Again, do keep a backup of your state file! For Kubernetes, I've had much fewer issues when it upgraded minor versions.

Question 3: When or how frequently should I apply my IaC changes?

After you finish coding your infrastructure, you will need to apply it to the real world to have your stack instantiated. You can choose the way to apply based on the requirements of how soon and how often changes will be made.

The fastest way to create your infrastructure from code or apply changes is to run it manually on CLI. Usually, the only other thing you need is the right access from your cloud provider. For AWS, it could be an STS token to assume an admin role or an ID/secret pair in the .aws folder or as environment variables.

If the idea of creating stacks directly from your laptop scares you or your security team, you can run the same commands after SSHing into a dedicated host with preset system roles just enough for managing your piece of infrastructure. This manual approach gives you more flexibility in the timing of applying stack changes, with the drawback of noncompliance in many companies, unless it's just a POC project.

A more widely accepted approach is to embed your infrastructure change into a CI pipeline. There are lots of plugins pairing CI platforms with IaC tools such as the Terraform Plugin on Jenkins. But more often than not, you don't even need any plugin to apply your infrastructure code in your CI. In Gitlab CI, for example, I only need to set the job image as hashicorp/terraform with the image tag matching my script to run any Terraform commands I would usually run in my CLI.

There are also other ways to apply your changes with third-party tools. Atlantis, for example, applies your changes when a Pull Request containing changes in Terraform files is submitted into Gitlab. Using the Github Pull Request page as the main interaction point is an increasingly popular choice for many modern tools (e.g. the static code analysis platform MuseDev) because it is one of the few points in the development lifecycle that draws the attention of the entire team.

Regardless of your choice, there is always the risk to build your stack into something monolithic, which slows you down in the long run. It is almost always helpful to make your changes small, modular, and independent of each other. This is especially true if you agree with the "Treat your servers like cattle, not pets" mindset [5], which means you should have the majority of your system disposable and reproducible.

For example, if you use Terraform, as we discussed above, the concept of modules would be your friend. You may want to start creating modules early and keep your root level main.tf file as small as possible: Ideally only a set of module blocks beside the mandatory terraform and provider blocks. Moreover, considering the risk of the state file corruption mentioned earlier, you may want to avoid having too many modules in the same stack sharing the same tfstate file.

Want to be even more confident to apply your change as frequently as you want? Consider testing your Infrastructure code, which is the next question to ask.

Question 4: How to Test my Infrastructure Code?

When you catch a colleague testing his code in production, will you recall this famous "I test in production" meme and laugh at him?

Well, how about when his code is in the context of "Infrastructure as Code"? Will your reaction change to "Oh, testing THAT in production is OK."? But is the infrastructure code really so different from the application code we are familiar with: that it is not testable in lower environments?

To answer the question let's first admit that infrastructure code IS harder to test. For one thing, you can hardly create a miniature version of the full infrastructure stack in your laptop, thus local testing is not always feasible. For another, mocking an upstream infrastructure is much harder than mocking an upstream service like in your application code because of the difference in the level of abstraction.

Fortunately, being unable to test something locally doesn't mean you have to test it in production. You should still have at least one other environment lower than production, whether you call it testing, QA, staging, or something else. Deploying a testing environment is a test itself, for your infrastructure code.

Assertions make tests possible, and for each IaC tool, there are companion tools to help you verify the actual end state of an infrastructure change against the desired. For Terraform, there is terratest, for CloudFormation there is taskcat, and inspec is for Chef. Even if you don't use these assertion tools, simply running your integration tests prepared for the application code would many expose potential problems in your infrastructure code. If you use CICD to apply your infrastructure changes, put the pipeline steps of applying changes to lower environment AND integration testing before the step to change production infrastructure.

However, testing and assertion in a lower environment only make sense when you run the same infrastructure code for your production environment. Take some time to modularize and reuse the infrastructure code you share among environments instead of copy/paste them around.

Sometimes, instead of full-fledged assertive testing, static code analysis could be a much more pragmatic target to automate. And yes, you can do static code analysis on your infrastructure. Most IaC tools included CLI commands to validate and lint your infrastructure code out of the box, and third-party tools can go beyond that and find certain misconfigurations and security-related issues as well. Below is an example of checkov scanning an IaC stack coded with Terraform, triggered via MuseDev.

Question 5: Will IaC mitigate my Tech Debt?

Infrastructure as Code, by itself, is just a tool like the Cloud or the DevOps or the AI, will not magically solve your problem. You have to decide if it is the right tool for your particular fix.

We usually consider technical debt as a combination of design debt and code debt [6], but here let's take either level apart and evaluate their relationships with IaC:

IaC and Design Debt

Here we limit the term "design" at the architectural level, rather than the application level.

Although in an ideal world a good architect should defer all the decisions at the infrastructure level as long as possible [7] to avoid premature constraints, the rubber has to meet the road eventually. To materialize the system, you have to decide on the exact type of DB, queue, server, logging, monitoring, alerting, and everything that you initially abstracted away from the developers of the application code. And some of the decisions are bound to be wrong the first time however experienced you are, because of the lack of information, which would be collected only after the initial decisions are made.

You will want to be able to refactor your infrastructure, continuously in some cases, after you start to get more information. You will want to be agile. And this is when you reap the benefits of Infrastructure as Code.

In places where IaC was not all adopted or implemented incompletely as an afterthought, I've seen infrastructure such as application servers drifted from their original state so much that no one dares to make changes on them. They are called by some "snowflake systems" (don't confuse with Snowflake Data Warehouse) because they are so unique that no one can reproduce them [2]. With the existence of such systems, any attempt to refactor your code would either fail or get dramatically slowed down, falling into the automation fear spiral [8].

Contrarily, proper implementation of IaC would result in disposable and replaceable modules of the infrastructure that you can quickly troubleshoot and resolve failures. The effort and risk of swapping new components in and old ones out would be finally recused to a level that you can experiment and refactor the details of your architectural design. And this is how IaC helps you solve your tech debt.

IaC and Code Debt

When you talk about spaghetti, smelly, inflexible or legacy code, you know you are talking about the code debt type of tech debt.

Unfortunately, Infrastructure as Code, however good it is, will not help you refactor your application code. In fact, these two types of code shouldn't even know the details of each other.

Ideally, thanks to the power of the cloud and containerization, your application layer should be decoupled from the runtime platform hosting it, and the runtime platform should be even further decoupled from the infrastructure platform supporting it. Your application code and infrastructure code should be in parallel dimensions.

As a result, You still need your good old red/green/refactor cycles to gradually improve the quality of your application code. Even more, you may want to start refactoring your infrastructure code as well from now on, since they are not only "real" code, but we also have ways to test them (see Question 4).

Fortunately, nowadays we have plenty of tools to help us fight code debt. For example, static code analysis tools with properly configured ruleset could easily give you actionable refactoring suggestions for source code of almost any language [9], and modern Continuous Assurance
platforms such as MuseDev are all you need to integrate those tools into your own repo.

Summary or TLDR

Q0: Should I consider Infrastructure as Code?

Suggestion: Yes if you have a cloud environment, public, hybrid or private.
Q1: Infrastructure as Code or Serverless?

Suggestion: For small standalone services, try serverless. For complex architecture, IaC or both.
Q2: To Terraform or not to Terraform?

Suggestion: Use Terraform if you worry about vendor lock-in. Be aware of other options (such as K8S) if you worry about getting locked into Terraform itself.
Q3: When or how frequently should I apply my IaC changes?

Suggestion: The smaller the unit of your infrastructure stack is, the more frequent you can change it.
Q4: How to Test my Infrastructure Code?

Suggestion: Lint your infrastructure code before applying your changes. Optionally use assertions to verify the deployment.
Q5: Will IaC mitigate my Tech Debt?

Suggestion: IaC helps you fixing your design debt, but you need other tools to fix your code debt.

[1] DevOpsCulture
[2] Infrastructure as Code, 2nd Edition - Kief Morris
[3] New for AWS Lambda – Predictable start-up times with Provisioned Concurrency | AWS Compute Blog
[4] Microservices, Nanoservices, Teraservices, and Serverless - DZone Microservices
[5] The History of Pets vs Cattle and How to Use the Analogy Properly | Cloudscaling
[6] Technical debt - Wikipedia
[7] A Little Architecture
[8] Infrastructure as Code: The Automation Fear Spiral | ThoughtWorks
[9] List of tools for static code analysis - Wikipedia

My 3 Different Attempts in Creating an EKS K8s Cluster

langyizhao — Mon, 15 Mar 2021 14:39:36 +0000

For the past few years, I was just using Kubernetes. Only until recently, I had to do some experimental work in a sandboxed Kubernetes environment. The first thing I need to do is to build the sandbox myself.

There was no hard requirement about how to build it, but since our team was already using AWS, EKS is the obvious choice at least to me: Rather than rolling my own I almost always prefer to use a managed service so I don't have to worry about the maintenance, some of the testing, and most of the security issues.

For me, the trade-offs such as vendor lock-in and marginally more costs are acceptable.

A consideration after deciding to use EKS is whether to use Fargate to back the nodes or the plain auto-scaling EC2 instances. Fargate is attractive to me because of its serverless vibe: I won't need the POC to be up all day long, so on-demand is not bad. But again, this is a soft requirement because I would only need a few micro to small nodes thus the cost would be negligible.

[Edit] Eventually I went back to plain EC2 from Fargate because:

Fargate requires more steps to set up in some of the cluster building options I list below.
Fargate requires more steps to add additional controllers (such as the AWS Load Balancer Controller ) into an established cluster.

Without further ado, here are the three ways I experimented in creating a new EKS cluster:

Experiment 1: Terraform with official AWS resources types

When using Terraform, my usual preference is to use as many official resources as possible before resorting to 3rd-party modules.

For EKS, I noticed the official AWS resource is https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_cluster

I created a local module starting from this aws_eks_cluster resource:

resource "aws_eks_cluster" "fargate-eks-cluster" {
  name     = "fargate-eks-cluster-${var.env}"
  role_arn = aws_iam_role.cluster-role.arn

  # Refer to https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_cluster#basic-usage
  ...
}

Then I realized it depends on a few other resources such as aws_iam_role and aws_iam_role_policy_attachment.

Fortunately, I can find the basic examples in the links above.

Trying to use Fargate, I referred to https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_fargate_profile and created three Fargate profiles for the default namespace, our working namespace and one specifically for CoreDNS as instructed.

resource "aws_eks_fargate_profile" "fargate-profile-default-ns" {
  cluster_name           = aws_eks_cluster.fargate-eks-cluster.name
  fargate_profile_name   = "fargate-profile-default-ns-${var.env}"
  pod_execution_role_arn = aws_iam_role.fargate-role.arn
  subnet_ids             = var.private_subnets

  selector {
    namespace = "default"
  }
}

resource "aws_eks_fargate_profile" "fargate-profile-coredns" {
  cluster_name           = aws_eks_cluster.fargate-eks-cluster.name
  fargate_profile_name   = "fargate-profile-coredns-${var.env}"
  pod_execution_role_arn = aws_iam_role.fargate-role.arn
  subnet_ids             = var.private_subnets

  selector {
    namespace = "kube-system"
    labels    = { "k8s-app": "kube-dns" }
  }
}

resource "aws_eks_fargate_profile" "fargate-profile-muse" {
  cluster_name           = aws_eks_cluster.fargate-eks-cluster.name
  fargate_profile_name   = "fargate-profile-muse-${var.env}"
  pod_execution_role_arn = aws_iam_role.fargate-role.arn
  subnet_ids             = var.private_subnets

  selector {
    namespace = "muse"
  }
}

Then I created the aws_iam_role resource for all of them.

And these are all the resources I needed in this local module. I used this module in my root module and did terraform apply.

The Terraform steps looked successful and after the cluster creation, I was able to verify that kubectl commands do connect to the right server.

However, when I tried to create K8s resources with my pre-existing manifest files, it didn't work as expected: the Overview Tab in the AWS console didn't show anything node alive.

Experiment 2: With `eksctl` CLI commands

I was initially following the official Getting started with eksctl guide in creating my cluster, with the --fargate option.

At the end of the eksctl create cluster --fargate command, a Fargate profile was created and several configurations were done under the hood. This is at least a couple of steps ahead of what I have achieved at the end of experiment 1.

Unfortunately, the out-of-box Fargate profile apparently doesn't work for non-default namespaces, and the networking wasn't quite right.

Despite [following the troubleshooting steps as well as creating custom Fargate Profiles, I still couldn't get any of my testing pods into running status. I believe I wasn't far from getting everything working but decided to try something else with the remainder of time.

Experiment 3: Experiment 3: Terraform with community modules

In seek of a more turnkey-ish solution, I looked into this community Terraform EKS module: https://github.com/terraform-aws-modules/terraform-aws-eks

As mentioned in its own issue, the complexity of this module is getting very high. I would usually avoid any heavy module like this and stay with plain resources if I hadn't failed in Experiment 1.

My main.tf file was just a combination of its working examples especially https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/examples/basic, except for that the module source has to be

module "eks" {
  source          = "terraform-aws-modules/eks/aws"

Instead of the ../.. in the example.

The end result turned out to be really good and the generated cluster met all our needs for my sandbox (Though I still had to install AWS Load Balancer Controller separately for Ingress)

My Conclusion

As it turned out, my smoothest cluster creating experience was to use the community Terraform EKS module following its examples. It works almost out-of-the-box.

If Terraform is not your thing, using eksctl would also save you a lot of initial legwork by creating the default underlying resources behind the scene with its subcommand. Just keep in mind that you still need to tinker with the cluster after the creation is done.