DEV Community: Andrew M

Deploy Keycloak Clients and Roles with Code

Andrew M — Fri, 09 Dec 2022 23:27:27 +0000

We last discussed how to structure our Keycloak role organization within a microservice deployment context. This discussion provides the foundation for what we will discuss today: leveraging Infrastructure as Code (IAC) to deploy Keycloak clients and roles easily. The goal for this project is to easily add additional microservice integrations to Keycloak so that they can be set up with minimal configuration. The tool will come as a lightweight CLI that interacts with a Keycloak server deployment to perform the client and role creation. THe tool will also provide smart defaults so new users can take advantage of this service without a full understanding of each setting.

Proposed IaC Schema

Here is a link to the JSON schema
Here is a link to the schema documentation

The proposed schema for this tool breaks down the Keycloak client configuration into more manageable and modular segments to ease template creation. For example, in this simple example, the hierarchical nature of the template enables users to specify their client features more easily.

Usage

In order to run the kraxen CLI, you must have NPM installed.
Install the CLI tool, aka Kraxen, by cloning the kraxen repository and installing it globally:

git clone https://github.com/elmsln/kraxen && cd kraxen
npm install -g .

Firstly, a configuration file for kraxen (.kraxen) must be created in either a user's home directory, or the current working directory. The configuration file must specify three things: KEYCLOAK_HOST, KEYCLOAK_ADMIN_USERNAME, and KEYCLOAK_ADMIN_PASSWORD:

KEYCLOAK_HOST=https://keycloak-host.org
KEYCLOAK_ADMIN_USERNAME=user
KEYCLOAK_ADMIN_PASSWORD=password

These values can also be set via environment variables.

Next, the configuration file for the client must be created. These configuration files are YAML files that define attributes of a Keycloak client that kraxen will create. Kraxen is configured to set smart defaults for a decoupled application that implements OIDC authentication. The only required values are the clientId, protocol, realm settings. Most of the other settings are set as smart defaults and can be overridden via the YAML configuration file. See an example of a template here.

Lastly, we can run the kraxen CLI and watch our template create!

kraxen -t template-file.yaml -c kraxen-config

The -c flag can be omitted if the .kraxen configuration is present in the user's home directory.

Final Notes

This initial executable is just a simple proof-of-concept model to demonstrate how Keycloak APIs can be leveraged to programmatically create clients and roles for use within an application. In the future, the CLI will scaffold out the template based on prompts that a user enters. It will also include sub-commands such as kraxen init to begin CLI setup, kraxen create to create a new set of Keycloak clients and roles, and kraxen sync to update the client configuration to match the current template. I hope to use the great example of elmsln's wcfactory tool to build a more robust CLI that can integrate even more features.

Lastly, I would like to shout out coveooss and their JSON Schema For Humans tool, which allowed me to produce markdown documentation for the Infrastructure as Code schema that I defined above. Kudos!

Integrating Keycloak Identity Management in the Frontend and Backend

Andrew M — Fri, 09 Dec 2022 22:52:33 +0000

Now that our authentication system is in place, client is created, and frontend assets are in motion, we will explore how authentication can be implemented into a micro-frontend component. The intended use case would be that this micro frontend could operate alone, or inside of a larger content system with other components that require authentication.

For the purposes of this example, let's consider a "discussion" microservice with a frontend web component and backend API.

Frontend Component

For the micro-frontend use case and other decoupled applications (i.e., applications with a distinctly separate frontend and backend service that communicate via API calls), a "public" Keycloak client is required. Public clients are used with applications that expose the frontend logic of an application and make it impossible to store Keycloak secrets securely. The alternative, a "confidential" client utilizes a secret key to create a trust relationship between the application and the Keycloak server. These types of clients are applicable for monolithic or server-rendered applications. Public clients enable applications to conduct authentication flows with Keycloak to return a signed token that contains information about the user. Security controls are implemented via origin policies and well defined redirect URIs. By controlling the endpoints that Keycloak is able to redirect users to, we are able to prevent external or rogue services from hijacking our Keycloak client and receiving tokens on behalf of our users.

In order for the frontend component to execute the authentication flow, there must be support within the front end of the application to prompt the user to log in before application content can be displayed. This portion of the front end should be modular, so that it follows a singleton pattern on a page. For example, if the discussion component was placed on a page by itself, the component should prompt the user to log in. However, if the discussion component was placed in the context of a larger application that requires authentication, the individual components within that application should defer to the wider application context without handling authentication individually.

Authentication Flow

Based on the requirement outlined above, the Open ID Connect (OIDC) Authorization Code Flow should be implemented. This flow is implemented by the keycloak-js frontend adapter by default.

First, the user will visit the page containing the web component. At this point, the micro-frontend is "disabled". A login button will be displayed to prompt the user to log in to the application.
When the user clicks on the login button, a second window will appear, directing the user to the Keycloak sign-in page. With this request, the application includes a redirect URL that defines where the user will be redirected after the authentication process.
The user selects an identity provider and is redirected to their log-in page.
The user authenticates with their identity provider of choice. Upon a successful authentication, the user is redirected from the identity provider back to Keycloak.
The user is then redirected from Keycloak back to the application using the redirect URI specified during the original authentication request. The response includes an "authorization code" that is used to obtain the Access, Identity, and Refresh tokens from Keycloak.
The application requests the Access, Identity, and Refresh tokens from Keycloak.
Keycloak validates that the requesting application is able to request the tokens.
Keycloak returns the tokens to the application. At this point, the application is "enabled" and can make API requests to resources that require authentication.
The application makes requests that load the initial data from the API. The Identity token is included in this API call.
The API verifies that the token is valid and that the user has the appropriate permissions to execute the request. The application then issues a response to the application.

A benefit to using the keycloak-js library is the abstraction of the token exchanges from steps 5-8 in the flow above. The keycloak-js adapter also builds the requests and URLs that are requires to begin the authentication flow. This enables developers to simply focus on the visual and system requirements for their application without focusing on the cumbersome request flow and security requirements.

Backend Implementation

A few functions must be defined on the backend API in order to enable it to participate in the flow defined above. Firstly, it should verify the validity of the token provided by the user. The Open ID Connect specification defines that JSON Web Tokens (JWTs) will be provided as the Identity, Access, and Refresh tokens in this case, thus developers can use this specification to validate user requests without interfacing directly with the keycloak server.

Firstly, the backend API should check if the identity token is valid. A valid token is one that is signed properly and is not expired. The jsonwebtoken package contains functions that can be called to verify that a token is valid. In order to execute the jwt.verify function, the public key for the Keycloak realm should be defined. This public key is required to verify the token signature.

In cases where keycloak roles are utilized, the role claim within the token should be evaluated against the list of applicable roles for the given API. Since the token has been declared cryptographically valid at this point, the role claim can be trusted and evaluated independent of the Keycloak server. This requires the API to be aware of the applicable keycloak roles for each resource thus centralizing the authorization decision. A benefit of this is that other identity management tools with role definition capabilities can be used in place of keycloak as a systems needs change. This reduces the amount of time required to process each API request.

In cases where processing related to the requesting user is required, the API should verify that the user record exists and create it if it does not. Keycloak returns the sub claim as a unique identifier for the user, following the OIDC specification. An example of this requirement for the discussion application would be if a list of users is maintained with a unique id. Tracking the user's ID would allow them to edit or delete their posts without granting the user admin permissions.

Considerations for Creating Keycloak Roles

Andrew M — Mon, 10 Oct 2022 22:11:33 +0000

Now that we have Keycloak up and running, we are ready to start configuring it for our microservices. A large part of keycloak is the concept of roles

What are roles exactly? Keycloak defines roles as the following:

"Roles are type or category of user. Applications often assign access and permissions to specific roles rather than individual users as dealing with users can be too fine grained and hard to manage.

"A user role mapping defines a mapping between a role and a user. A user can be associated with zero or more roles. This role mapping information can be encapsulated into tokens and assertions so that applications can decide access permissions on various resources they manage."

One important thing to get started with Keycloak is to understand that it does NOT link roles to application specific permissions. Instead, permissions must be assigned within the application (or microservice). With that being said, we should consider how to organize groups of users based on the level of permissions that they require, and also decide any global roles that have implications across all applications (e.g., Admin, default role, etc.)

There are some additional constructs that Keycloak provides to make role management easier. For example:

Groups manage groups of users. Attributes can be defined for a group. You can map roles to a group as well. Users that become members of a group inherit the attributes and role mappings that group defines.

A composite role is a role that can be associated with other roles. For example a superuser composite role could be associated with the sales-admin and order-entry-admin roles. If a user is mapped to the superuser role they also inherit the sales-admin and order-entry-admin roles.

Client scopes define protocol mappers (configuration to pass specific user attributes) and role scope mappings for that client.

With client roles, clients can define roles that are specific to them. This is basically a role namespace dedicated to the client.

Proposed Role Structure

Typically, Keycloak role structures mimic organizational structure, where each role is related to a specific job title. This makes sense in cases which job titles and roles do not change frequently. In our case, we must consider the educational environment, where there are 3-5 job titles: Administrator, Professor, Student, etc. Clearly, we need to have more granular control.

Global Roles

Global Roles apply to all applications within the realm. These are more applicable to Administrative permissions and developers.

Examples:
Site Admin: global.admin (i.e., access to global application configuration)
Site Developer: global.dev (i.e., access to test and development resources)

Additionally, there should be global roles for authenticated users to differentiate them from unauthenticated users, even if the user does not match the required roles to perform a specific action.

Application Specific Roles

Application Specific Roles broadly limit the types of actions that users may take in a specific application. For example, these roles could be used to grant user access to create new content within the application or adjust global settings. General Application Specific Roles are not granular and are not meant to control access to content.

Additionally, Keycloak client roles define smaller sets of roles to pass to each client. Role scopes will ensure that permission grants are efficient and roles which are not required are not passed to the client.

Examples:
Application User: .user (i.e., create new application resources and manage their own)
Application Admin: .admin (i.e., access to application specific configuration)

Site Specific Roles

Application Specific roles are a subset of Application Specific Roles. They grant user access to individual sites within an application, and make it possible to provide different permissions for different microservices. For example, A user could have read access to site content, but write access to create comments on that site.

Examples:
Site Admin: ..admin (i.e., manage site settings and content)
Service User: ..user (i.e., view/edit/update created service content)

How to Deal With Roles

Now that we have a role structure outlined, we must consider the following:

How roles should be created
How roles should be assigned to users
How permissions should delegated to users

Questions 1 and 2 will be answered in later parts of the series, as we plan to automate role creation and assignment. The Keycloak REST API makes it easy for administrators to automate the creation roles as new applications are registered. Additionally, the creation of Keycloak clients (or application registrations within Keycloak) should be automated within the deployment lifecycle.

References

Keycloak Server Administration Guide

Installing Nessus Agents on AWS EC2

Andrew M — Mon, 03 Oct 2022 18:07:34 +0000

Public cloud services like AWS make it easy to create scalable resources that expand and contract with load. While this solves many operational IT problems, it fails to address them securely.

The threat landscape is constantly evolving, so IT teams must be ahead of the curve when it comes to patching. Nessus is one vulnerability scanning and remediation tool that many organizations utilize to monitor the security of their assets. Nessus is deployed via an agent that runs on each asset. It runs automated vulnerability scans and reports the results to a centralized cloud platform which can be accessed by security analysts.

Nessus is a great solution for security on premises- but it also can protect cloud resources. As part of my centralized microservice strategy, I implemented Keycloak Authentication as a cloud instance. To comply with organizational policy, all servers utilizing the company's domain name must have the Nessus agent installed. This posed a challenge as I originally planned to deploy Keycloak with ECS Fargate - a serverless container solution. ECS also works with EC2 to expose the underlying infrastructure and enable administrators to have greater control over the host.

As with many of my deployments, I leveraged AWS CloudFormation to deploy my resources. This enables me to leverage Infrastructure as Code to maintain visibility and consistency into my infrastructure configurations. See my CloudFormation template for this deployment here. NOTE: You may experience some problems with this CloudFormation template if you do not complete the below configuration steps. Please read below before deploying the template.

ECS on EC2 requires a few additional resources to operate:

Auto Scaling Group: Defines a set of EC2 instances that can dynamically grow and shrink with application demand. ECS will schedule containers to run on these instances.
EC2 Launch Configuration: Defines the configuration of EC2 instances that will be launched into the autoscaling group. Some configuration options include: AMI ID, Security Groups, and Instance Type. Another configuration item is EC2 User Data, which is a bootstrap script that is run when the instance is launched.

In our use case, user data is the primary mechanism for configuring the Nessus agent on our instances. Because User Data is defined in the launch configuration, we can be sure that each instance launched in our ECS Auto Scaling Group will include the Nessus Agent.

Configuration Steps

Download the Nessus Agent Installer: Nessus packages are available for download directly from the nessus website. In fact, Nessus requires users to download it right from their website.
Create an Amazon S3 Bucket to store the Nessus installers: Next, create an S3 bucket to store the Nessus agent installer. This bucket should block all public access. Upload the installer to this bucket and note the filename.
Create a Role that enables EC2 instance to access files from the S3 bucket: Create a role that allows s3:GetObject access to the agent storage s3 bucket. Note: If using the provided CloudFormation template, the role and instance profile are configured, BUT they reference a User managed policy called S3NessusAgentAccess. This policy should be created before deploying the template. You should be sure to replace the bucket name nessus-agents in the policy to reflect your bucket name.
Use the following UserData to install the agent:

#!/bin/bash
yum install -y aws-cfn-bootstrap
yum install -y unzip
echo ECS_CLUSTER=${ECSCluster} >> /etc/ecs/ecs.config
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install 
/usr/local/bin/aws s3 cp ${NessusAgentUri} .
rpm -ivh NessusAgent-*.rpm
/opt/nessus_agent/sbin/nessuscli agent link --key=${NessusAgentKey} --groups=${NessusAgentGrp} --cloud
service nessusagent stop
/opt/nessus_agent/sbin/nessusd -R
service nessusagent start
/opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --resource ECSAutoScalingGroup --region ${AWS::Region}

This user data is formatted for CloudFormation with a few required parameters:

NessusAgentUri: The path of the Nessus agent installer
NessusAgentKey: The tenant key for Nessus
NessusAgentGrp: The name of the group to link the agent to
ECSCluster: [Optional] The name of the ECS Cluster that the instance should be available to.

Another note for this UserData is that it is designed to work within ECS configured via CloudFormation. This script would also work with regular EC2 instances after brief modifications.

If not using this script with AWS ECS, omit the following line:
echo ECS_CLUSTER=${ECSCluster} >> /etc/ecs/ecs.config

If not using this script with AWS CloudFormation, replace all CloudFormation parameters, and omit the following lines:
yum install -y aws-cfn-bootstrap
/opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --resource ECSAutoScalingGroup --region ${AWS::Region}

Lastly, if you are using this script with an AMI that includes the AWS CLI (e.g., Amazon Linux 2), omit the installaiton of the CLI package:

yum install -y unzip
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

Architecting a Keycloak Deployment in AWS

Andrew M — Thu, 22 Sep 2022 22:31:03 +0000

Now that we've defined our use case for Keycloak, let's dive in to how we will actually deploy it. Keycloak is designed to support a variety of environments including Docker, Kubernetes, and OpenShift. For our use case, we will be utilizing Amazon Web Services Elastic Container Service (ECS). ECS is a contianer orchestration playform that enables users to deploy their continers to the cloud with minimal overhead. If you've been baffled by Kubernetes, I assure you that ECS will be a much better experience.

Architecture Overview

First, let's take a look at the various services at play in this deployment:

Take a look at the CloudFormation Template for this deployment here

VPC Overview

Virtual Private Clouds (VPCs) are logically isolated sections of the AWS cloud that act are assigned to one customer. Think of a VPC like a virtual data center in the cloud. Each VPC is divided into subnets, each with their own route table and set of security groups. Subnets are blocks of IP address space within the larger VPC block. Route tables determine how each subnet handles traffic, and security groups act as a host based firewall, but security groups can be applied to multiple hosts. For more information about the structure of VPCs, check out this article by BMC's Joe Hertvik.

How is Traffic Routed?

Firstly, traffic destined for the Keycloak server will be sent to the resolved DNS CNAME that points to the Application Load Balancer.

Next, after entering the Amazon Web Services network, the traffic traverses to the Internet Gateway Internet Gateways are common constructs of the VPC, and they are required to route traffic from the internet to VPC based hosts and services.

After entering the VPC, requests are routed to the first public subnet where the Application Load Balancer is located. Elastic Load Balancers (the umbrella term for all load balancers) distribute traffic across a group of hosts to maintain high availability and scalability. Application Load balancers enable specific route based forwarding through a set of rules. In our use case, we have a single rule that simply forwards traffic to the Keycloak instance.

Additionally, the Application Load Balancer performs TLS termination for our Keyloak deployment, and enables users to connect securely to the Keycloak server. A TLS Certificate is defined in Amazon Certificate Manager, which is referenced by the Application Load Balancer. As requests arrive at the load balancer, the certificate is returned to begin the TLS encryption process.

Next, traffic is routed into the private subnet where our application resources live. We opted to utilize ECS on EC2 for this deployment, as it affords us the additional control required to meet compliance requirements. In this deployment strategy, containers are scheduled to run on a defined EC2 instance that is managed by the user. Even though this responsibility is on the user, container configuration and management is still defined by the ECS service. One requirement of Application Load Balancers is the definition of multiple subnets across multiple availability zones to schedule containers in. In the event that a container exits or fails a health check, ECS will deploy an instance in another availability zone, and start the container there. Thus, high availability is built into the deployment by default.

Lastly, the deployment reaches the Keycloak container, running the latest Keycloak Docker image.

ECS deployments are broken into Clusters, which can have many services that contain a set of tasks. Clusters typically contain all of the services required for a given application. For example, an application with multiple backend services would be grouped into the same cluster. A Task Definition is a document that outlines the configuration for an ECS Task. Tasks are essentially containers that are running instances of the task definition. Task definitions can define a variety of configuration items such as the container image URI, environment variables, port mappings, and desired container counts.

Because Keycloak requires a database to function, we've created an EC2 instance to run the database instance. We've chosen MariaDB, but many other options are available per the Keycloak documentation. In the future we may upgrade this to a true RDS instance, but for now, EC2 suits our needs.

Configuration Overview

Much of the configuration for this deployment occurs within the Keycloak container's environment variables. In the event that the container needs to be restarted, all of the configuration can be referenced via the container's task definition.

Introduction to User Authentication for Microservices

Andrew M — Fri, 09 Sep 2022 00:48:12 +0000

Do you have a collection of microservices that you've been painstakingly integrating into an authentication service one by one? Maybe you're running these services in a federated user environment, or you need a way to control authentication from a single point. Regardless of your situation, it is important to consider how to design an authentication system that is robust, extensible, and meets business requirements.

Today, we'll be looking at a few standards to integrate authentication into microservices through a centralized identity and access management system. We'll be reviewing Security Assertion Markup Language (SAML) OpenIDConnect, and Oauth2. We'll also begin to discuss how to integrate these services into applications at a high level. This article will begin a series that journeys through the process of integrating authentication at the system, backend, and frontend level. Our application model will follow btopro's microservice design process, with the goal of adding authentication to the threaded-discussion web component.

Without further ado, lets get started!

Authentication standards

Key Terms

Authentication: The process of confirming a user's identity
Authorization: The use of authentication data to grant access based on a given user's permissions.

SAML

SAML is a standard that passes credentials from Identity Providers (IdP) to Service Providers (SP) aka applications (https://www.okta.com/blog/2020/09/what-is-saml/). IdPs Authenticate users (e.g., Microsoft Active Directory) and SPs authorize users.

SAML passes informaiton between providers (and the user browser) via SAML Assertions. Assertions are simply XML Documents that provide informaiton about a user.

Authentication assertions prove a user’s identity.
Attribution assertions pass SAML attributes—the pieces of data that provide information about the user—to the service provider.
Authorization assertions confirm whether the user is authorized to use a service—and what degree of authorization they have

OAuth2

OAuth2 is a standard for "secure delegated access", or Authorization between applications. It provides claims for a smaller service (e.g., mycoolapp.com) to access data on another service, (e.g., Facebook). OAuth makes use of tokens (usually JWTs) to delegate access to applications for a set of resources.

An important thing to remember is:
>"OAuth is NOT about authenticating the user, and this is key. OAuth 2.0 alone says absolutely nothing about the user. You just >have a token to get access to a resource." Matt Raible, What the heck is OAuth2?

OpenIDConnect

Building on top of the OAuth2 Framework, OpenIDConnect (OIDC) provides authentication via an ID Token. This enables systems to share authentication state and user profile information in a predictable format. Without OpenIDConnect, developers are left to find an arbitrary authentication framework to combine with OAuth, which is complex and challenging.

In regards to OpenIDConnect, there are a view parties that need to communicate:

Resource Owner: A user
Client: A service (e.g. mycoolapp.com) that needs information
Authorization Server: The identity provider of the resource server
Resource Server: System containing information that the client needs

OpenIDConnect provides a set of standards that developers can use to connect various systems predictably:

Scopes: While OAuth defines scopes in an abstract sense, OIDC defines a standard set of scopes (openid, profile, email, and address).
Claims: Similarly, OIDC explicitly defines a set of claims that the scopes provide. These claims contain sets of user information.
ID Token: The ID Token contains information used to federate identity across applications.
userinfo Endpoint: OIDC defines a standard set of endpoints. The /userinfo endpoint provides identity metadata verification to provide interoperability.
JWTs: OIDC utilizes the JSON Web Token (JWT) standard to sign the ID token, ensuring a cryptographically strong identity statement.

Centralized Identity Providers and SSO

Authorization and Authentication protocols, such as those that we discussed previously, allow developers to separate their identity management systems from their applications. In this case, a single authentication source and user base can be used for a wide variety of applications. This is particularly useful in the enterprise environment, where users must access many resources but should not have independent user accounts for each service. Similarly, Authentication and Authorization can link many microservice to a single identity provider that is independent from the service itself, leading to a more loosely coupled application.

In this series, we will be implementing Keyloak as the bridge between a federated identity provider and our microservice applications. This will simplify the integration of additional microservices into our environment, as we will only need to interface with the identity provider once.

Additionally, we will make use of a feature in Keycloak called roles. Roles are a specific tag assigned to users within Keycloak, and are passed to the application. Roles enable developers to define permission sets in their application for a given role, without considering the individual users that will assume that role.

High-Level Authentication Flow

The image above shows a high-level view of the authentication and authorization flow that we will build. Firstly, we will deploy the Keycloak server which will act as the SAML Service Provider and the OIDC Authentication Server. Next, we will configure the Keycloak server to use Azure Active Directory as the SAML IdP. This will enable users to log in to our applications. I will configure the microservice applications as OIDC clients. Because of the Single Sign On capability within Keycloak, authenticating once with the server will enable users to access all microservices on the page.

As we progress in this series we will develop a pattern to integrate Keycloak into a micro-frontend service, with the goal of expanding authentiation to 10s of services on a single page. Follow along to learn more!

References

What’s the Difference Between OAuth, OpenID Connect, and SAML?
What Is SAML and How Does It Work?
What is OpenID Connect?
What the Heck is OAuth?
OAuth2

Using Vercel to instantly deploy web applications

Andrew M — Mon, 14 Mar 2022 00:59:36 +0000

Have you built a new web project recently and wanted to show it off to the world? Most likely, you run into this desire as a developer and discovered that the deployment process is long and laborious. Even simple automation tools like GitHub Actions are riddled with complexity and can take a while to understand.

Enter Vercel, a platform as a service product that enables developers to showcase their projects with just a few clicks. Oh, and its free too, by the way. Today, I'll show you how you to leverage Vercel to deploy two styles of applications: a NodeJS application and an 11ty static site.

Why use Vercel?

In the world of application deployment, you may need to deal with servers, VMs, containers, Docker, networking, Terraform, AWS, Jenkins, Ansible...

The list could go on forever.

Vercel seeks to avoid complexity by abstracting away infrastructure entirely. It uses a "Functions as a Service" or FaaS approach, which essentially means that you can upload your code (the functions) and Vercel will build an environment around that. If this sounds like AWS Lambda, that's because Vercel uses Lambda's engine under the hood to provide this functionality to its users. Vercel further abstracts Lambda's configuration process to provide an approachable environment to serverless compute.

Getting started with Vercel

The first logical step would be to create an account. As I mentioned previously, Vercel is incredibly easy to work with. Most developers use some type of version control (i.e., GitHub) during the development process, and Vercel integrates with many to connect your repositories right to their build pipeline. You can use GitHub, GitLab, or BitBucket to set up your account.

After setting up your account, you're ready to start deploying your projects. From the Vercel dashboard, select the New Project button.

From there, you can take one of two paths:

Import a Git repository: This is ideal for finished projects, or projects that you've already created a GitHub repository for. During the creation process, you'll create the first deployment of your app using the main branch of your repository. You don't need to use a specific framework for this method. Another consideration: when connecting your account to version control, you are able to install Vercel across all public repositories on your account, or limit the installation to specific repositories. In the Vercel dashboard, You are only able to access projects that you have installed Vercel into.
Clone Template: This is ideal for new projects, after you've decided on a framework to use. Some available options are Next.js, Nuxt, 11ty, and Remix, amongst other options. Cloning a template will clone the starter project repository to your version control of choice, then install Vercel into that repository.

After selecting your path, you will be given some configuration options for the build/deployment of your application. When you're ready, select the blue Deploy button, and you're off to the races! You'll see deployment progress, and a live output of the build right from the Vercel site.

Environment variables: Often, environment variables are required to store API Keys, database connection information, or other sensitive data. These values can be saved to Vercel via the "Environment Variables" section. If you need to add them after deployment, you can do so under ${Project} > Settings > Environment Variables. These values can then be accessed in code via process.env.${variableName}.

11ty on Vercel

Example 1 - Hello Horses!
Example 2 - 11ty Base Blog
Example 3 - hax11ty

Developers often leverage static site generators (like 11ty) to spin up quick sites from template files, like markdown, that are easy to write. Left with these files, it can be a hassle to set up a web server, configure Nginx, copy the files, set up HTTPS, and configure a domain for the world to see. GitHub Pages helps to take away some of this configuration by automating the deployment process, but the configuration of GitHub Actions is still required, and some nice features like parallel preview builds are complex or missing.

Vercel runs your eleventy build command and completes all of the grunt work required to generate a public URL for the world to see with near zero configuration. It even allows users to deploy the same application from different branches so that new features can be test driven before production.

One issue that I ran into was a configured pathPrefix variable in my .11ty.js configuration. This generated non-existent paths for my assets and links on the site. If you've deployed a site to GitHub pages, you likely have this configured. To solve this issue, I modified my .11ty.js configuration to include the following in the module.exports function:

  //support for deployment on both Vercel and GitHub Pages
  let deployPath = '/';
  if (process.env.deployEndpoint != 'vercel'){
      deployPath = '/11ty-base-blog/';
  }

Then, I set module.exports to return pathPrefix: deployPath

I also included the deployEndpoint environment variable and set it equal to 'vercel' in my deployment settings on Vercel.

NodeJS Applications on Vercel

Example

In addition to serving static content, Vercel provides server-side processing via Functions as a Service (FaaS). In the example above, the site utilizes micro-frontend components that process data via Vercel's serverless functions.

Serverless functions in Vercel are created via files in the /api directory of a project. There are a number of supported backend languages, including JavaScript, which we used for this project. Within the file, a "handler" function is defined via the syntax below:

export default function handler(request, response) {
  const { name } = request.query;
  response.status(200).send(`Hello ${name}!`);
}

The above function defines a handler

The image above shows a JavaScript function that defines a Serverless Function within Vercel. This function creates a HTTP endpoint at domain.com/api/hello. This Function takes a query parameter of name, and returns Hello ${name} to the requesting client.

No additional configuration was required to acheive this functionality, simply creating handler functions in the /api/ directory created the defined functions when the project was built and deployed.

Case Study: Shoelace CSS on Vercel

Mock Client
Codebase

Vercel enables users to utilize "preview branches" of their applications to test new features or application updates without pushing them to production. This is a feature not possible in simple deployment platforms like GitHub pages. Through this case study, I will demonstrate how to use preview branches when testing out changes to a web component library.

Create a new branch in version control Vercel automatically identifies your main branch in your version control platform. In this example, my main branch is called main and my preview branch is called mayormaier/future. This page shows components that are sourced from the main (production) branch, served at https://shoelace-mayormaier.vercel.app. Vercel treats all other branches in version control as preview branches.
Push changes to the new branch When changes are pushed to non-main branches, mayormaier/future in this case, a preview deployment is created.

Because any non-main branch can trigger a preview deployment, one application can have seemingly endless numbers of preview deployments, each to test out new features. In this example, the preview deployment is found here. It is also referenced on this page, where the script tags in the <head> of the page reference the preview deployment. Notice the underlined text in the components, the "new feature" that we wanted to test drive.

<link
    rel="stylesheet"
    href="https://shoelace-git-mayormaier-future-mayormaier.vercel.app/themes/dark.css"
    onload="document.documentElement.classList.add('sl-theme-dark');"
    />
    <script type="module" src="https://shoelace-git-mayormaier-future-mayormaier.vercel.app/shoelace.js"></script>

Conclusion

Ship applications at blazing speeds with Vercel, a service that abstracts much of the configuration details away for their Serverless Functions. Use of vercel is free for most "Hobby" use cases, however it is subject to certain limits on execution time and bandwidth. Enterprise applications may need additional firepower or control. Vercel has a paid tier that unlocks additional performance, but for fine grained control configuring AWS Lambda yourself may be necessary.

Architecting Portable Applications with Docker

Andrew M — Mon, 28 Feb 2022 07:34:10 +0000

"But it works on my machine!"

Thats the endless cry of developers working on a team and developing in different environments. There's always one that just can't get the program to run, no matter what they try.

Sure, this problem is unfortunate, but what's worse is deploying applications across inconsistent environments. The more that an application must scale, the larger chance there is for inconsistency.

Enter Docker- a containerization platform that obliterates the problem of inconsistent environments and also provides other benefits like a smaller footprint compared to VM deployments.

What is a container?

A container is a standardized compute environment (like a Virtual Machine) that only contains the libraries and components that an application needs to run. Containers have a smaller footprint because they do not run their own kernel, but instead connect to the host's kernel through the containerd runtime.

How are containers so portable?

Containers can be hardened into images that can be replicated across machines, but it is the automated build process that truly makes containers rock.

When software is ready for deployment, it can be run through an image build pipeline, where a standardized set of commands (a Dockerfile) can be run to produce a container image.

This is an example of a Dockerfile used to build a NodeJS backend application:

FROM node:12

WORKDIR /home/node/app

COPY package.json ./
RUN yarn install

COPY . .

CMD [ "yarn", "start" ]

What do all of those commands mean?

FROM: This Dockerfile builds an application backend on top of a nodejs 12 base image
WORKDIR: This changes the working directory of the build process
COPY: This copies files from the host computer into the container as it is being built
RUN: This executes a command within the container itself (i.e., to install the required packages in this case)
CMD: This provides a command that will be run when the container is started

So, when building a container image through a Dockerfile, it is important to think about a few things:

What software do I need on my container? This will determine your base image (node, mysql, python, etc.)
What files do I need to copy to my container?
Do I need to run any commands to build my software?
What command or process do I want to start when the container runs?

After considering these questions, you will have enough information to build your container through a Dockerfile. Regardless of the environment or application, these principals can be applied to produce a standardized image that will run anywhere.

How to build powerful sites from scratch with 11ty

Andrew M — Mon, 28 Feb 2022 02:38:11 +0000

If you've been interested in the web space recently, you've heard of static site generators like 11ty and Jekyll. These pieces of software take easily written markup such as markdown, nunjucks, and liquid and turn them into content that is ready for the web. Many developers want to author blog posts for their own site, and 11ty provides a way to do just that.

While on the surface it seems simple, theres a lot under the hood that makes 11ty a machine.

Terms

Before starting, it's important to know terms that 11ty uses to describe their files. The names sound similar, but the differentiations are important.

template - a file that contains content (written in a language like markdown)
layout - pre-built wrappers for templates. Used to define a common structure for a set of templates/pages
front matter - key-value pairs at the top of a template that define specific variables that are used when rendering pages in 11ty
data file - purpose built files that define variables to be inherited by all templates in a defined directory.
- Ex. in the directory /posts, /posts/posts.json would be the directory data file, and would define variables that all other template files /posts/* should inherit.

Getting Started

11ty can be intimidating at first, especially if you are new to command line tooling. To start, I followed the great getting started guide at

Here, I will demonstrate three ways to build an 11ty site, both using templates and from scratch.

Method 1: From scratch

Shameless plug, I heavily leaned on the great tutorial by Alex Trost and Ben Meyers when completing this project.

One nice thing about 11ty is that you can start out VERY basic, then build up in complexity.
To start,

initialize a new npm package npm init and install 11ty npm install @11ty/eleventy
create some template files somethingCool.md and throw them in the project root
run npx eleventy --serve and watch your templates turn into HTML in the _site directory

Now lets kick it up a notch.
First lets provide some organization to our project.

create an .eleventy.js file
add return { dir: { input: "src" } }; in a new module.exports() function

const embeds = require("eleventy-plugin-embed-everything");

module.exports = function(eleventyConfig) {
    eleventyConfig.addPlugin(embeds);
    eleventyConfig.addWatchTarget('_site/assets/*.css');
    eleventyConfig.setBrowserSyncConfig({
    files: ['_site/assets/*.css']
    });

    return { dir: { input: "src" } };
}

An example .eleventy.js file. NOTE: this file contains additional plug in modules that you probably don't have installed yet. Only use this configuration if you have the defined modules installed.

create a new directory called src
move your precious templates to the new src directory

Now, lets add some additional structure to our pages. When eleventy builds a page, it focuses on minimal HTML. No <head> information is included.

in your src directory, add a _includes subdirectory. This is where you will store your layout files.
create a new layout file. you can specify where the content should be inserted with {{content}}
specify this layout to be used in the front matter of a template using the layout key:

---
layout: layoutFilename
---

- cool
- markdown

Lastly, lets add a plugin to this site. We will be installing the eleventy-embed-everything plugin.

install the plugin with npm install eleventy-plugin-embed-everything --save
add the plugin to the .eleventy.js file

const embedEverything = require("eleventy-plugin-embed-everything");

module.exports = function(eleventyConfig) {
  eleventyConfig.addPlugin(embedEverything);
};

Now you can embed media into the page simply by dropping its url on to the page.

More information
Source repository

Method 2: Based on a template

The 11ty community has developed a plethora of templates that accomplish the initial configuration of 11ty quite well. Many of these sites contain a lot of useful tools and features, but on occasion this can bloat the final size of the project. The output files however remain flat and slender.

The first step that I took when configuring this template was to modify the metadata of the site to match my details (name, url, email, etc.)

Next, I modified the pathPrefix variable in the .eleventy.js file. This allows the site to be deployed to GitHub pages without explicitly defining the pathPrefix in page links (which would break the site in development).

Lastly, I added my articles to the /posts/ directory. Because much of the configuration has already been completed, it is pretty much plug and play for all of the articles. If additional subdirectories/categories are desired, more configuration is necessary.

After inserting my articles, I specified some template metadata for each article. These vary slightly from the custom built template as different data variables are used, however they generally accomplish the same thing.

More Information
Source Repository

Method 3: hax11ty

hax11ty combines the powerful library of web components found in HAXCMS with the simplicity of static sites. Just like 11ty, it takes in template files, and outputs them as HTML. Just like in Method 2, utilizing a pre-built 11ty configuration saves a lot of time when you need to get a site off the ground quickly. With a clean theme and the addition of web components, hax11ty takes static sites to the next level.

For example, adding the <video-player> tag to our markdown yields a rendered video-player component on the page.:

- things
- and
- stuff

<video-player source="https://www.youtube.com/watch?v=dQw4w9WgXcQ"></video-player>

Data Structure

When using 11ty, there is a clear division between source content and static HTML site files. 11ty will locate the content for your site, written in any of the supported templating languages, then convert it into HTML and place it into the _site directory. Within the _site directory, there is an exact clone of the directory structure where all of the source files lived previously. This _site directory is key, because this is the directory that includes the HTML files needed to power a website. The _site directory can be zipped and uploaded to a website hosting platform.

These HTML files are generated through a process that 11ty calls the data cascade The data cascade dictates the order of priority for sources of layout and style data for the static HTML files. While extremely technical, the cascade basically begins by applying global data to the template files, and then overwrites these data files as it moves deeper into the directory. Data files with the same name and same directory as the templating files are given the highest priority.

Lastly, the YAML front matter in each templating file is applied, which overwrites values further up in the data cascade. This is helpful for applying specific properties, such as a permalink, to individual files.

When a website goes live, the folder structure that contains the markdown files transfers directly to the generated static HTML files. So, for example, if a particular markdown file is located at /src/welcome.md then the subsequent HTML file will appear at _site/src/welcome/index.html.

One caveat to 11ty is that it does not automatically copy non-supported file types (anything other than files written in the supported templating languages) into the _site directory. In order to request that these files are copied into the _site directory, use the --formats = flag on the command line to specify certain file types.

More information
Source Repo

Integrating APIs with JavaScript in the Front End

Andrew M — Sun, 30 Jan 2022 22:36:26 +0000

Fetch and the Power of APIs

fetch() is an asynchronous JavaScript function that allows client-side web applications to make HTTP Requests to a web endpoint. This is most commonly used to make API calls from the browser.

Asynchronous functions are known as "non-blocking". Rather than taking up a processing thread while waiting for a return value, non-blocking functions allow other operations to execute in the program. This results in much more responsive applications.

Fetch's asynchronous property is enables it to free the processing thread while waiting for an API response. This is a great way of handing API calls, as responses can vary in speed depending on the destination server and application.

fetch('http://example.com/movies.json')
  .then(response => response.json())
  .then(data => console.log(data));

the above example is courtesy of Mozilla

The fetch method is relatively simple. In its most basic form, fetch() has one parameter, the URL of the HTTP endpoint. Other parameters can be added to send data to an endpoint (i.e., JSON for a HTTP PUT request). This enables developers to fully leverage API requests in their front end applications.

In the example above, an HTTP GET request was made, which returns data from the server to the client. After the response returns successfully, the .then() functions parse the response as JSON, then print it to the console. However, console logging is not the only thing that can be accomplished in this function.

.then() clauses can also be used to pull data from the API response and set it as a variable. For example, in the application presented in this example, the responses from freegeoip.app/json are used to identify the location of a user at a specific IP address. The latitude longitude city and region_name fields are all variables that the API returns and are tracked by the application. Here's an example of the JSON data returned by the API:

{
"ip": "104.38.28.100",
"country_code": "US",
"country_name": "United States",
"region_code": "PA",
"region_name": "Pennsylvania",
"city": "University Park",
"zip_code": "16802",
"time_zone": "America/New_York",
"latitude": 40,
"longitude": -77,
"metro_code": 574
}

This JSON blob is sample API response from the freegeoip.app/json API.

Element Variables and Rendering

Variable assignment in the then() method calls enables stateful updating of the application. Each time that the API is called and returns data successfully, the instance variables are updated, and the DOM is re-painted with the new data. The render() function determines how the page will be displayed each time that the DOM is painted. Not all variables in the application achieve this behavior- only variables defined in the static get properties() method trigger the DOM to be re-painted. Note: you can also generate new variables based on the variables that are returned by an API call. For example, I set location equal to $city, $region_name which is used many other times in the application.

static get properties() {
    return {
      lat: { type: Number, reflect: true },
      long: { type: Number, reflect: true },
      city: { type: String, reflect: true },
      region: { type: String, reflect: true },
      location: { type: String, reflect: true },
    };
  }

The properties defined in this method trigger the DOM to be re-painted

Let's discuss the <location-from-ip> component in more depth. Firstly, the properties listed above populate the component with the data that it needs to render. The data relies on APIs to populate. The getGEOIPData() function includes all of the logic to obtain these data points.

Firstly, a UserIP object instance is created to identify the IP address of the user. This relies on an API that returns the IP of the user making the request. This IP addres data is then fed into another API (freegeoip.app) that takes the IP address from the user and returns location data about that IP address. See the example API response above. After the response is retuned, the given variables are updated which triggers a repainting of the DOM. This update feeds those new variables into a number of services defined in the render() function:

const url = `https://maps.google.com/maps?q=${this.lat},${this.long}&t=&z=15&ie=UTF8&iwloc=&output=embed`;
    return html`
      <iframe title="Where you are" src="${url}"></iframe>
      <br /><a
        href="https://www.google.com/maps/@${this.lat},${this.long},15z"
        target="_blank"
        >Expand Map to ${this.location}</a
      >
      <br /><br />
      <wikipedia-query search="${this.location}"></wikipedia-query>
      <wikipedia-query search="${this.city}"></wikipedia-query>
      <wikipedia-query search="${this.region}"></wikipedia-query>
    `;

The lat and long variables are inserted into an Google Maps embed link that populates an <iframe>.
The lat, long and location variables are used to populate a hyperlink that opens the location in the full Google Maps site.
The <wikipedia-query> web component is leveraged to provide articles about the location determined by the GEOIP API. The compnent relies on a search property that defines the wikipedia page to display. There are three <wikipedia-query> tags total. One uses the location property as the search string, and the other two use city and region.

The <location-from-ip> element, visually

Ultimately, the use of APIs within web components can be achieved with ease and is a great way to add responsive elements to a static site.

For more information about the API used in this application, see freegeoip.app, wikipedia element, IPFast IP Address API

Check out the application repository here

Sources

General asynchronous programming concepts - MDN Web Docs
Using Fetch - MDN Web Docs

Getting Started with open-wc and Web Components

Andrew M — Sat, 15 Jan 2022 22:49:16 +0000

One month ago, I hadn't written a single line of JavaScript.

I knew that JavaScript was important, but I seemed to be scared away by the massive ecosystem. As I looked deeper into the powerful things that you can do with JavaScript, I knew I had to get my feet wet. If you're looking to get started with open-wc and Web Components, you're going to need to understand the fundamentals of plain vanilla JavaScript. That might sound daunting, but getting started with Javascript is easier than you think. Today I'll show you why.

JavaScript Basics

Before starting my journey with JavaScript I had intermediate programming experience. Most of my academic career I have used Java, and I've written my personal projects in python. If you've never written any code, thats ok. We all start somewhere. Most language fundamental tutorials are beginner friendly.

From a JavaScript newbie, I recommend the JavaScript Essential Training LinkedIn Learning Course. This course helped me to get an understanding od the language as a whole while also understanding the JavaScript ecosystem. I'm about 30% of the way through it, and plan on banging out some more after I finish this article.

One more thing- one of the biggest surprises for me is that JavaScript's native runtime is in the browser (Like Google Chrome). It took some getting used to, as I was more familiar with working completely in the command line. It is intuitive though, as a majority of JavaScript use cases involve the web in some capacity.

Preparing the JavaScript Environment

Unlike many languages (like Python, Java, and Go), you don't need to download a language interpreter to your machine. In fact, you run JavaScript code every time you visit most websites. So, if you have a modern web browser installed, then you're golden.

VS Code

I use Visual Studio Code to write my JavaScript Code. It is very lightweight bare bones out of the box, but it has a rich extension ecosystem that you can use to increase its functionality.

Head over to code.visualstudio.com to install VS Code. Click on the big blue installation button to download the installer and follow the prompts when running it.

One key extension that you will need when writing and testing vanilla JS is "Live Server". This allows users to start a local web server for their current VS Code project with just one click.

Node.js

The next step is to download Node.JS. You want to get the current LTS (Long Term Support) version for increased stability. To install, head to nodejs.org and click the nice big green "LTS" button. The site should recognize the OS that you are working with and give you the right installer.

Node.JS enables you to run server-side JavaScript applications on your machine. Some people think that Node.JS is a JavaScript library or a framework or its own separate language, but Node.JS is none of those. Node.JS is a runtime for JavaScript.

After running the installer, check that Node.js was installed properly with node -v

npm

Node comes with a package manager for JavaScript called npm. npm enables you to use other people's code in your projects without needing to go and write it yourself. Users can find npm packages on the npm registry, then use them in their code by using the Node require() function and defining them in their projects' package.json file, creating a dependency. All of this can sound confusing at first, but for now, just know that you can piggyback off of other projects with npm.

To verify that npm was installed correctly when you installed Node.js, you can run npm -v

Yarn

Yarn is another package manager for JavaScript. It is very similar to npm as it enables users to reuse code from other developers by helping them register dependencies in their project. It registers dependencies to the package.json just like npm.

With newer versions of Node, yarn comes pre-installed and can be installed without much hassle. Simply run corepack enable.

After doing this, you can verify that yarn is installed properly with yarn -v.

Initializing an open-wc starter project

Now that the basic software is installed and running, a new open-wc component can be created. This process is also relatively simple. First, create a new directory where your project will live and navigate to it. Then, run npm init @open-wc If the command runs successfully, that means that you've installed everything correctly and you are beginning to work on your first web component!

This is the screen that you will see with instructions to set up the project.

I've initialized my starter project with the following settings:

New project scaffold
Web Component
Linting, Testing, and Demoing enabled
No TypeScript
installed using yarn

Once these settings are selected, a browser window should open and connect to the Node web server that is serving the project. If that does not happen automatically, or you want to start up the server at a later date, simply run npm start. Boom! Now you can take a look at how web components work on the web.

Analyzing a Web Component

Web Components, even in their most simple form have numerous files that provide functionality. Each of these files has a unique purpose that enables web components to function as the easy to use, reusable HTML elements.

First, lets look at the index.html of this project, the file that the browser loads when this project is opened.



<body>
  <div id="demo"></div>

  <script type="module">
    import { html, render } from 'lit';
    import '../hello-world.js';

    const title = 'Hello World!';
    // takes elements with id=demo and replaces with hello-world element
    render(
      html`
        <hello-world .title=${title}>
          some light-dom
        </hello-world>
      `,
      document.querySelector('#demo')
    );
  </script>
</body>
</html>

In this file, there is a div with id=demo, that is then replaced by the <hello-world> element when the script is loaded. The <hello-world> element is hydrated with content referenced in hello-world.js, which is imported to the script.



// imports the HelloWorld class from the source files
import { HelloWorld } from './src/HelloWorld.js';

// defines the "<hello-world>" HTML tag from the HelloWorld class in the imported module
window.customElements.define('hello-world', HelloWorld);

hello-world.js defines the <hello-world> HTML tag with the HelloWorld Web Component.

The meat of the element is found in ./src/HelloWorld.js. This component defines the functions and properties of the HelloWorld web component, represented as a class that extends the base HelloWorld class. For example, one of the functions called __increment() increments the counter property of the HelloWorld object every time a button in the component is pressed.

Many of the other files that come with the base "hello-world" web component serve important purposes also. I've annotated many of the files found in this project and uploaded them to a GitHub repository that can be found here.

Vaccines for Common SQL Injection Bugs

Andrew M — Fri, 25 Sep 2020 19:06:15 +0000

I chose to connect SQL to the Web through option 2, a more SRA/ security-based approach

SQL Injection is one of the most common web hacking techniques (w3schools.com). In cases where SQL databases are queried by site visitors, these injections pose a big threat to the database. SQL injections allow a user to send malicious code to a database by inserting a SQL command into a web input field. Without proper defense against them, websites can be left vulnerable, and massive data leaks can occur as a result.

Below, I take a look at some of the most common SQL injection

1=1

One of the most common approaches to SQL injection is the 1=1 method. This is often used to gain unauthorized access to a user account. In this case, a false password is entered into the password field, followed by a quote mark and OR 1=1--

This allows the unauthorized user to bypass the password field of the login page, because the database reads that it is supposed to allow a login WHERE Password = ... OR 1=1 Because 1=1 is always true, the user will be allowed in and the rest of the line is commented out.

Batched Statements

Another common SQL Injection is the batched statement. These injections finish a legitimate statement with a semicolon ; before executing their malicious statement. For example, if a user enters details into a search box, and the search is selecting some information where a condition is true, after giving the parameter, a user can insert their malicious code.

Exploitation through URL

Lastly, if a user can see part of an SQL query via the URL for example http://widgetshop.com/widget/?id=1, they are able to understand how the query is formed in order to change manipulate the URL and create malicious queries. In this case, the user can easily append additional SQL statements to the end of the query for example ...OR 1=1 and see more data than intended.

Solutions

Clearly, using user input to craft SQL Queries is a big vulnerability. Thus, finding a way to protect your database from malicious requests is essential to maintaining data security. In order to avoid malicious input to your site, you need to use one (or many) abstractions so that only valid input is accepted into the SQL command.

One abstraction that can be used to solve this issue is to parameterize the SQL statements outbound for the database so that they are treated properly. In order to do this, the SQL statement string and its associated input parameters are passed to the database separately which allows the database to properly interpret the command. Each parameter has a defined variable type, so the input is sanitized. This is opposed to the full SQL command being constructed on the front end of the website, making it vulnerable to malicious parameters.

To see how these SQL injections can be used for malicious purposes, check out the video below.