DEV Community: Naseem Mohammed

Distributing API Authorization Policies using OPA Bundles

Naseem Mohammed — Tue, 05 Jul 2022 08:00:12 +0000

A typical organization will have several Applications & Deployments in multiple environments. Most of them would have some central Identity Provider like Keycloak or Azure AD/Okta...
While the identity provider issued OAuth2.0 token and solve the Identity problem, we didn't have a clean central way to handle API Authorization. This usually might be bundled along with the Application code.

For Authorization, OPA is an open-source CNCF graduated project that allows us to write Authorization logic in Declarative statements.

The goal today is to explore the OPA bundle feature. We can

build policies using Rego,
package them as bundles (tar files),
and distribute them to various Applications or Deployment Clusters and the from a central location.

The central policy portal can organize your policies as the below diagram shows. An individual bundle is created at the environment level and it will have all the policies of the projects that the environment hosts.

In my organization, we had a similar distribution of environment.
All the APIs were fronted by various API Gateways. What we want is for every call should be intercepted by API Gateway and forwarded to the Authorization engine. If the Authorization engine returns false then the request should be declined with HTTP status 403. Below diagram represents this.

The below table gives details on how this can be achieved with different API Gateways.

API Gateway	Details
AWS API Gateway	-
Traefik	Using Forward Auth Middleware
NGINX	Module ngx_http_auth_request_module

In our case, we have deployed OPA as a sidecar to a golang API project. The API receives a request from the API Gateway and then invokes the OPA. It parses the response received and if the Allow is set to false, then the API will return 403 to API Gateway.

Below is the yaml for the deployment of the API project and OPA. The configuration is from wsl2 desktop.



apiVersion: apps/v1
kind: Deployment
metadata:
  name: authorization
  namespace: default
  labels:
    app: authorization
spec:
  replicas: 1
  selector:
    matchLabels:
      app: authorization
  template:
    metadata:
      labels:
        app: authorization
    spec:
      containers:
      - name: auth-policy-manager
        image: naseemmohammed/policyengine:0.1.7
        imagePullPolicy: IfNotPresent
        env:
        - name: ENV_AUTH_SERVER
          value: ":8080"
        - name: ENV_PPSA
          value: "localhost"
        - name: ENV_OPA_PORT
          value: "8181"
        ports:
        - containerPort: 8080
      - name: opa
        image: openpolicyagent/opa:0.41.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8181
        env:
        # location to subscribe for new policy bundles
        - name: AWS_REGION 
          value: us-east-2
        - name: AWS_ACCESS_KEY_ID
          value: AKIA52XXXUIISSCC-FAKE
        - name: AWS_SECRET_ACCESS_KEY
          value: E/VLCUZJ2G-NOTEXISTING-aUhmsMcI33yS8O     
        args:
          - "run" 
          - "--ignore=.*"  # exclude hidden dirs created by Kubernetes
          - "--server"
          - "--set=decision_logs.console=true"          
          - "--config-file"
          - "/config/config.yaml"
        volumeMounts:        
        - readOnly: true  
          mountPath: /config
          name: config-volume
        livenessProbe:
          httpGet:
            scheme: HTTP              # assumes OPA listens on localhost:8181
            port: 8181
          initialDelaySeconds: 5      # tune these periods for your environemnt
          periodSeconds: 5000  # in prod reduce to 5
        readinessProbe:
          httpGet:
            path: /health?bundle=true  # Include bundle activation in readiness
            scheme: HTTP
            port: 8181
          initialDelaySeconds: 5
          periodSeconds: 5
      volumes:
      - name: config-volume
        hostPath:
          # directory location on host
          path: /run/desktop/mnt/host/c/Users/nmohammed/Downloads/cluster-files
          # this field is optional
          type: Directory
      imagePullSecrets:
        - name: topsecret

Now the OPA configuration file below



services:
  s3:
    url: https://zohoohio.s3.us-east-2.amazonaws.com
    credentials:
      s3_signing:
        environment_credentials: {}

bundles:
  authz:
    service: s3
    resource: Zoho-onprem/bundle.tar.gz
    persist: false
    polling:
      min_delay_seconds: 100
      max_delay_seconds: 200

IBM App Connect Professional CI/CD

Naseem Mohammed — Sun, 28 Jun 2020 14:52:45 +0000

IBM App Connect Professional is an iPaas. What is an iPaas?

Quoting from Mulesoft (a competitor of IBM App Connect Professional) *"In simplest terms, iPaaS is a platform for building and deploying integrations within the cloud and between the cloud and the enterprise. With iPaaS, users can develop integration flows that connect applications residing in the cloud or on-premises and then deploy them without installing or managing any hardware or middleware."

IBM App Connect Professional can be installed in multiple ways. IBM provides a Docker Image also. That lets us deploy the App Connect to the Kubernetes platform.

The goal of this article is to document the steps to build a CI/CD pipeline for IBM App Connect Professional. I am using the below tools to realize this. Also, the IBM App Connect Professional version I am using is 7.5.3.

Github for source code.
Codeship for CICD
Datadog for monitoring
Locust for Load testing.
Oracle Kubernetes Engine for hosting the Docker containers.

Here is the Deployment Architecture

IBM provides the App Connect Professional software (7.5.3.0-WS-ACP-_64-docker.tar.gz), which has the requisite software and Dockerfile for customizing.

System requirements

You can run the App Connect Professional Docker container in the following configurations:

Two CPUs with 4 GB RAM and a 100GB disk
Four CPUs with 8 or 16 GB RAM and a 100GB disk
Eight CPUs with 16, 24, or 32 GB RAM and a 100GB disk For production purposes, it’s recommended that you use four or more CPUs.

Below is the customized Dockerfile

FROM ubuntu:16.04

ARG ibm_file

ENV IRONHIDE_SOURCE /var/tmp/ironhide-setup

RUN echo "bing" && echo $ibm_file &&  apt-get update && apt-get install -y  openssh-server supervisor cron syslog-ng-core logrotate libapr1 libaprutil1 liblog4cxx10v5 libxml2 psmisc xsltproc ntp vim net-tools iputils-ping curl 

RUN curl -LO  $ibm_file --output ironhide-setup.tar.gz && tar -xzvf ironhide-setup.tar.gz

RUN ls -l &&  cd ironhide-setup &&  ls -l && cat supervisord.conf && cp supervisord.conf /etc/supervisor/conf.d/supervisord.conf  &&  sed -i -E 's/^(\s*)system\(\);/\1unix-stream("\/dev\/log");/' /etc/syslog-ng/syslog-ng.conf 

RUN sed -i 's/^su root syslog/su root adm/' /etc/logrotate.conf

RUN mkdir -p /var/log/supervisor & mkdir -p /opt/ibm/

#Directory to hold the artifacts which need to be loaded during docker launch/start
RUN mkdir -p /var/tmp/LoadArtifacts/projects & mkdir -p /var/tmp/LoadArtifacts/ThirdPartylibs & mkdir -p /var/tmp/LoadArtifacts/SecureConnectorConfig
RUN mkdir -p /var/tmp/LoadArtifacts/UsersAndGroups & mkdir -p /var/tmp/LoadArtifacts/CertificatesAndKeys

RUN cp /ironhide-setup/etc/cron.d/* /etc/cron.d/

#copy the configuration files inside docker container
COPY /projects /var/tmp/LoadArtifacts/projects
#nm 
#RUN cd \ && ls -l && pwd
#RUN cp -R /UsersAndGroups /var/tmp/LoadArtifacts/UsersAndGroups
#COPY /CertificatesAndKeys /var/tmp/LoadArtifacts/CertificatesAndKeys
#COPY /ThirdPartylibs /var/tmp/LoadArtifacts/ThirdPartylibs
#COPY /SecureConnectorConfig /var/tmp/LoadArtifacts/SecureConnectorConfig

RUN cp ironhide-setup/etc/logrotate.d/* /etc/logrotate.d/

RUN chmod 644 /etc/cron.d/*

RUN chmod -R 777 /var/tmp/ironhide-setup

ENV JAVA_HOME /usr/java/default

ENV PATH $JAVA_HOME/bin:$PATH

ENV IRONHIDE_ROOT /usr/ironhide

ENV LD_LIBRARY_PATH /usr/ironhide/lib

ENV IH_ROOT /usr/ironhide

ENV IRONHIDE_BACKUP_PATH /var/tmp/ironhide-backup

ENV PATH $IH_ROOT/bin:$PATH

ENV interface1=""

ENV interface2=""

RUN cp  ironhide-setup/scripts/liblog4cxx.so.10 /usr/lib/x86_64-linux-gnu/liblog4cxx.so.10.0.0

RUN echo 'PS1="[AppConnect-Container@\h \w]: "' >> ~/.bashrc

CMD ["/usr/bin/supervisord"]

Below is my codeship service file.


app:
  build:
    image: phx.ocir.io/axyp8vsk2dul/ibmappconnect
    dockerfile_path: Dockerfile
    encrypted_args_file: config_encrypted
oracle_dockercfg:
  image: codeship/myservice-dockercfg-generator
appkubectl:
  build:
    image: phx.ocir.io/oci_kubectl:0.0.4
    dockerfile_path: DockerfileOciKube
    encrypted_args_file: config_encrypted
    args:
      CommitID: "{{.CommitID }}"

And below is my Codeship steps file


# codeship-steps.yml
- name: Build and push to Oracle Docker Registry
  service: app
  tag: master
  image_name: phx.ocir.io/axyp8vsk2dul/ibmappconnect
  image_tag: "{{ .CommitID }}"
  registry: phx.ocir.io
  encrypted_dockercfg_path: dockercfg.encrypted
  type: push
- name: Check response to kubectl get nodes
  command: kubectl get nodes
  service: appkubectl
- name: Check OCI Version
  command: oci -v
  service: appkubectl  
- name: Deploy the IBM App Connect Image with flows and configs
  command: kubectl apply -f /config/.kube/ibmappconnect.yaml
  service: appkubectl
- name: Print out the environment varibales
  service: appkubectl
  command: printenv

I check in the par file to git.

This will trigger a build in Codeship.

Codeship packages the par files on top IBM App Connect Professional base image and deploys to Oracle Kubernetes Engine.

We can use Datadog to monitor the cluster and running pods (IBM App Connect Professional container).

We can also use IBM App Connect Professional WMC and confirm everything as expected.

We can run some load test and ensure the system performance will be acceptable to our end customers.

Using Istio for Ingress & Feature Flag Deployment in AKS

Naseem Mohammed — Sun, 28 Jun 2020 08:27:36 +0000

What is a Service Mesh?

According to Wikipedia, it is
"In software architecture, a service mesh is a dedicated infrastructure layer for facilitating service-to-service communications between microservices, often using a sidecar proxy."

What is Ingress (in Kubernetes)?

"Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource."

What is Istio?

Istio is all about traffic. Whether that traffic is between the Microservices within a Kubernetes* cluster (East-West) or traffic entering/leaving the Kubernetes* Cluster (Ingress traffic or North-South).
*"(Istio can be used in other Orchestration platforms also besides Kubernetes)."

So Istio is

Service Mesh (E-W) & Ingress Gateway (N-S)
Open Sourced by Google, IBM & Lyft in May 2017
Service Mesh designed to connect, secure and monitor microservices

Istio architecture from Istio Website.

Istio features

Traffic Management
1. Discovery.
2. Load balancing
3. Rate limiting
Resilience
1. Failure recovery
2. Fault Injection
3. Circuit Breaker
Observability
1. Metrics
2. Monitoring & Alerts
Deployment
1. Canary rollouts
2. Feature Flag Deployment
Security
1. Access control
2. End-to-end authentication
3. Security in transit- mTLS
Ingress Gateway
1. Prefix based traffic routing

For this article, I came up with a Made up use case. I have created 4 Microservice. Each is written in .NET Core and packaged as Docker.

Healthcare
1. Benefits
2. Insurance.
Hospital

The Healthcare microservice internally depends on benefits which depend on Insurance.
The Hospital microservice does not talk to any other microservice.
The Healthcare and Hospital microservice needs to be invoked from the outside world.
Below is the diagram of the microservices and their interaction.

So I installed Istio in my cluster. Enabled the namespace to inject Istio pods.

The installation had also installed Kiali.

What does Kiali provide? Well, it answers the below questions.

Which microservices are part of my service mesh?
How are they connected?
How are they performing?
How can I operate on them?

Now using Kiali I had a look at my cluster.

Couple of things you can do with Kiali is do Routing

Weighted Routing between different version of your Microservice
A more interesting and more useful part is the ability to Feature Flag release based on HTTP Header Routing.

Below are the configurations.

Weighted Routing

Flag release based on HTTP Header Routing

Setting up Ingress Gateway

Using Codeship to Deploy a Microservice onto Oracle Kubernetes Engine

Naseem Mohammed — Fri, 29 May 2020 05:30:13 +0000

So, I was looking at an alternative to Azure DevOps and Jenkins to build a CI CD pipeline for a new project. A friend had asked me for a recommendation. He wanted to host microservices in Oracle Kubernetes Service.

I had heard about Codeship and had wanted to give it a try for a while. So this was the nudge I needed and spend the weekend over it. And it was totally worth it.

There are two versions of Codeship. Basic and Pro. Pro is a bit more expensive. According to their faq below is the reason.

Why is CodeShip Pro more expensive than Codeship Basic?

"CodeShip Pro spawns single-tenant AWS instances for you whenever you push a build. You are not sharing your instance’s CPU, Memory, etc. with anyone else."

The flow

Developers checks in the code to a Github repo.
This triggers a build in Codeship. Codeship uses the Dockerfile checked in and tries to build a Docker image.
The resulting image is tagged by Codeship with the Github Commit Id and pushed to the Azure Container Registry (ACR). (Yeah, just mixing it up with Azure for fun.)
Deployment
1. Codeship now issues a kubectl command to Oracle Kubernetes Engine (OKE) Master API service. This will be to deploy the Microservice and the load balancer in front of it.
2. The Docker image that we build and pushed earlier to ACR will be the image in this deployment.

Prerequisites for achieving this

Github Account
Codeship Pro Account (There is a free tier which is what I am using).
Azure Account and Azure Container Registry (You can replace this with Oracle Container Registry).
Oracle Cloud account and a running instance of Oracle Kubernetes Engine. (I used a free 30 day Oracle Cloud Trial Account).

Codeship Structure and my Setup

In Codeship everything revolves around two configuration files. codeship-service & codeship-steps files

Correlation I build in my head about Codeship Services & Steps.

Services

Codeship Services provide the functionality to accomplish the CI CD pipeline’s steps (or tasks). Services provide these functionalities by using Dockers. Services at the end, are just a Docker.

Used for Build and Push Microservice
Two services are used to accomplish a Build & Push to registry functionality. That is two corresponding dockers are required. These two Services are actually mapped to a codeship step (or task) with an attribute called type(=Push). Below is Codeship documentation for that step which provides the Build and Push functionality.
https://documentation.codeship.com/pro/builds-and-configuration/steps/
Utility Service
These Services (Dockers) maybe prebuilt by Codeship (or us) and pulled from a registry like Docker Hub at the time of CICD execution.
codeship/azure-dockercfg-generator is an example of Codeship pre-built Dockers.
You can see more of Codeship prebuilt Dockers here. https://hub.docker.com/u/codeship. Interesting to see that the AWS Docker has been downloaded 500k+ times while the Azure one has only been downloaded 10K+ times.
Custom Service
A Service can also custom build a Docker at runtime. We provide the Codeship Service a Dockerfile. This is what I did with appkubectl Service. Custom build one with Oracle CLI (OCI) and kubectl packaged within it.
This is how my code-service file looks like

app:
  build:
    image: dockerstore.azurecr.io/aksistioinsurance
    dockerfile_path: Dockerfileweb
azure_dockercfg:
  image: codeship/azure-dockercfg-generator
  add_docker: true
  encrypted_env_file: az_config_encrypted
appkubectl:
  build:
    image: dockerstore.azurecr.io/oci_kubectl:0.0.4
    dockerfile_path: Dockerfile
    encrypted_args_file: config_encrypted
    #encrypted_env_file: config_encrypted
    args:
      CommitID: "{{.CommitID }}"

Steps (Or Tasks)

codeship-steps.yml file is where you specify the steps. Think of this as the task. Each step uses one of the Services. Usually, it is a 1: Many relationships between Service & Tasks. But some steps like the Building and Pushing Dockers to the repository has 2 Services mapped to it.

Example Steps (or tasks) can look like below. This we define and build the functionality as per our requirements.

Build Microservice & Push to Container Registry
Integration Test
Deploy to Oracle Kubernetes Engine

To accomplish these tasks, we use our custom Services (or Dockers) or Codeship provided Services. The below table shows the relationship I used in my Build pipeline.

# codeship-steps.yml
- name: Build and push to Azure Docker Registry
  service: app
  type: push
  tag: master
  image_name: dockerstore.azurecr.io/aksistioinsurance
  image_tag: "{{ .CommitID }}"
  registry: dockerstore.azurecr.io
  dockercfg_service: azure_dockercfg
- name: Check response to kubectl config
  command: kubectl get nodes
  service: appkubectl
- name: Check OCI Version
  command: oci -v
  service: appkubectl
- name: Deploy to Oracle Kubernetes Engine
  command: kubectl apply -f /config/.kube/insurance.yaml
  service: appkubectl
- name: Print out the environment varibales
  service: appkubectl
  command: printenv

The above one is my codeship-steps.yml file

Desktop utility

There is a nice command-line utility that Codeship ships called Jet. (Ship & Jet Hmm..) You can use it for

Encrypting/Decrypting your configuration files/variables. These may db passwords or Container Registry credentials.
Local Testing of your Codeship Steps (tasks) before pushing to Github.
Validation. (Didn’t use this much.).

Ok, let's look at the pipeline in action.

Flow

When I check in the code in Github, I expect a CommitId being provided by GitHub.

As you can see from above the Commit Id starts with characters => aa02a67

Now we expect this to have triggered a build in codeship engine.
Codeship will build and push a new Docker image to Azure Container Registry. This image will be tagged with the new GitHub Commit Id. The below screenshot confirms that the tagging has happened.
Next, we expect Codeship to issue a Kubectl command. Against the below yaml file. It contains a Kubernetes Deployment and related Service object.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: insurance-api
  namespace: nm
spec:
  replicas: 1
  selector:
    matchLabels:
      app:  insurance-api
      version: old
  template:
    metadata:
      labels:
        app:  insurance-api
        version: old
    spec:
      containers:
      - name: insurance-api
        image: dockerstore.azurecr.io/aksistioinsurance:##tag##
        resources:
          requests:
            memory: "32Mi"
            cpu: "25m"
          limits:
            memory: "64Mi"
            cpu: "100m"
        ports:
        - containerPort: 80
        livenessProbe:
          httpGet:
            path: health
            port: 80
            httpHeaders:
            - name: X-Custom-Header
              value: Awesome
          initialDelaySeconds: 90
          periodSeconds: 10
        env:              
        - name: "DeviceName" 
          value: "aStrangeDevice"        
      imagePullSecrets:
      - name: topsecretregistryconnection
---
kind: Service
apiVersion: v1
metadata:
  name: insurance-api-service
  namespace: nm
spec:
  type: ClusterIP
  ports:
  - name: http
    protocol: TCP
    port: 80      
  selector:
    app:  insurance-api

There are two things I want to bring to your attention about the above yaml file.

The imagepullsecret. This actually has the Docker username and password of the Azure Container Registry. This secret was set up initially at the time of the creation of Oracle Kubernetes Engine Cluster. The below command was used.

kubectl create secret docker
-registry topsecretregistryconnection 
connection --docker-server dockerstore.azurecr.io 
--docker-email "###" --docker-username="###" 
--docker-password "##$#####"

For details check Kubernetes documentation https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/?ref=hackernoon.com#registry-secret-existing-credentials

The image tag has a placeholder ##tag##. This placeholder will dynamically change in the Dockerfile.

cp insurance.yaml $HOME/.kube/insurance.yaml &&  
sed -i 's/##tag##/'$CommitID'/1' $HOME/.kube/insurance.yaml && 
cat $HOME/.kube/insurance.yaml && \

Once Codeship runs the kubectl apply command against the above yaml file we would expect this image to be deployed in the Kubernetes cluster within OKE. Let's check and find out.

As you can see from the above screenshot the image with the right tag has been picked up and deployed to OKE.

Codeship Dashboard

The codeship dashboard is minimal but has enough to help you in debugging.

And one last thing, be careful of Codeship build arguments and environment variables. I wish Codeship was a little more consistent in their naming. For instance, at places they got CI_Commit_ID and for build arguments, they got CommitID (no underscore). It would have been nice if it all were consistent.

Securing APIs Using Okta and Azure API Gateway

Naseem Mohammed — Thu, 28 May 2020 16:25:33 +0000

Traditionally in a .NET or Java Server application, the APIs have been secured using SessionId. After a user authenticates with the server; the server generates a unique sessionid and this sessionId is sent to the client in the Http Response. All further communication between the client and server will carry this sessionid in the payload. Though the HTTP protocol is stateless; using this sessionid helps the server to track the client and group all the client's requests as being part of one conversation. Now, this approach has worked for a long time; but it has some significant weaknesses.

Issues	Details
Session affinities	When there is just one server it works fine. But when 2 or more server behind a load balancer, the SessionId will need to be store outside the app server. Like a external State server or Database. This adds complications like serialization/deserialization and latency issues.
Session ID hacking or spoofing	Spoofing is hard if the session id is a Guid. But it does not prevent a stolen session id (or cookie) from being reused.
Multiple authentication	if the client needs to connect to a different service/, the client needs to authenticate again and use a different sessionid. This multiple authentications is cumbersome and also a security risk.
No access granularity	There is only id exchanged. No extra information, attributes are available. The server ends up making look ups into the database for enriching information about the user logged in,
Access propagation	if the server needs to talk another service on the client's behalf the sessionId will not suffice. A different mechanism needs to involved making this SessionId route more cumbersome and adding security risk.

So the industry guidance today is to avoid baking in API security into code. (read sessionid). Today in the world of cloud development; it is recommended to adopt the infrastructure for security instead of baking it in code.

Using infrastructure for API security has the following advantages.

Respects the separation of duties.
Makes the code simple and easy to maintain. (now that all that ugly & scary security stuff are removed.)
Less maintenance burden.
Reusable.
Transparent to the security team.

So what is the modern way of securing APIs in the cloud? The answer is Oauth2.0.
We won't go into the OAuth2.0 protocol. To learn more on the OAuth2.0 protocol you can use this as reference => https://www.oauth.com/.

The goal here is to look at how we can secure APIs hosted behind the API gateway. We will look at Okta (Identity as a service provider) as the OAuth2.0 token generator and API gateway ensuring each API request is authorized.

Just for analogy purpose let's see what roles Okta and Azure API Gateway play in securing our APIs.

As the above pic shows Okta issues the Access card (token to be exact) and the Gateway sits in front of each door (API to be exact) and validates whether the user has access to the said API.
Okta actually generates two tokens. ID Token and Access Token. We will focus on the Access Token. According to Wikipedia, an access token is defined as
"In computer systems, an access token contains the security credentials for a login session and identifies the user, the user's groups, the user's privileges, and, in some cases, a particular application."

Okta

So how does Okta know what token to issue a user?. Befire we answer that question; let's look at Okta a little more in detail.

The above screenshot shows that Okta stores

Users (and passwords),
Client Application information (SPA, Mobile Apps, etc..)
User Attributes (Prebuilt and Custom attributes based on client Application)
Authorization Server

Users

Okta stores users and their passwords. (In most enterprises this will actually be federated from the Enterprise Active Directory.)

Client Application

Okta configures the Client application (that will consume the APIs). The below screenshot shows an example Okta APP page with users assigned to it. Usually, you do not assign users directly to Apps, instead you would assign user groups to apps.

Only the users assigned to the applications can authenticate into the applications.

User Profiles

Okta allows us to store User attributes. Some of these are built-in. Admins can add custom attributes either at the Base Okta level or at the individual application level. Below screenshot shows; the IMS application's User Attribute configuration page. Here admins can add new or custom attributes about Users that are specific to the application. For instance, an application may add an attribute called "User Birthplace". Another Application may add an attribute called "User SchoolName". The point is that; each application is unique and the Admins of that application can customize attributes specific to that application.

Authorization Server

We talked earlier about the Access token. Authorization Server is the one that generates the Access Token. The below screenshot shows the configuration page of the Authorization Server. Here we can tell the Authorization server what all claims (user attributes) need to part of the Access token.

Okta provides an inbuilt utility where we can test the contents of the token.
Below is the screenshot for that.

SPA application receives this token after the user Authenticates against Okta.
This is the token received by my application =>

eyJraWQiOiJhQW85TktxdjMxZmQ0Q1RaRWtuQnpoZVZBMVcwLThhc0xsWC0wUG5uN0JNIiwiYWxnIjoiUlMyNTYifQ.eyJ2ZXIiOjEsImp0aSI6IkFULmhrckNOS092R3V4aE1IUGxSck9BbUhxYzRTZFpRdkN1c3hOelBhMng2MkkiLCJpc3MiOiJodHRwczovL2Rldi0yMTI5MDMub2t0YXByZXZpZXcuY29tL29hdXRoMi9kZWZhdWx0IiwiYXVkIjoiYXBpOi8vZGVmYXVsdCIsImlhdCI6MTU5MDY3NjQyNCwiZXhwIjoxNTkwNzEyNDI0LCJjaWQiOiIwb2FxNGY0N3hmU21JV2RxTjBoNyIsInVpZCI6IjAwdWhzYXN0NnlNcGdjVmZXMGg3Iiwic2NwIjpbIm9wZW5pZCJdLCJhcHB1c2VyIjoibS5uYXNlZW1Ab3V0bG9vay5jb20iLCJzdWIiOiJtLm5hc2VlbUBvdXRsb29rLmNvbSIsIkRlYWxlcnNoaXBMb2NhdGlvbiI6WyJTYW4gRnJhbmNpc2NvIiwiTG9zIEFuZ2VsZXMiXX0.OJmj-o0LHGtHtCtxLKtshwPK6IRQjDc6umZ_PBtGcZfXvE9afluOvfaqmLyYvyaA1uis3PK0jrZg8zSJFS-ryjYycbyJ8f7Mxhd5TyMxYpMRdPIEr3rG6KoByvhqLMJTnwKRV7sSp6af7R8DO7noFc1Wj7jolDRsb3iJmV7Z_g-IySqXKtU7BSpAI1nBoPG6SUsDfU18PoDT7z8_PPkvP7JpNWAzphcp1S3H0O_Z7RkaE04iqvboCjf3OHeBKFScx918bR4XtVh-dMdd84mL8xT7ez-IOpnoQvYCjXSTNmkJjEDUPYjgnA8VNgv3i9dBl4vxj7sDgQ5IXjslL5IYeQ

if you drop the above token at jwt.ms; you will see the below screenshot.

Now that we've got the token; we have to pass this in every API call we make to the Azure API Gateway. Because the Azure API Gateway checks each incoming request headers. It specifically looks for Header named "Authorization". It expects this header to have a valid Access token in it.

Azure API Gateway

Azure API Gateway sits in front of all our APIs. We have to configure our API Applications such that it allows traffic only from Azure API Gateway, whether the application is hosted in Azure Kubernetes, Azure WebApps, or Function Apps. Assuming we have done that; let us look at how Azure API Gateway is configured to allow only requests with the right Access Token.

Below is a screenshot of the Azure API Management Portal.

Every API has one more Operation within it. In the above screenshot, we can see that the "Vehicle Pricing" API has a single "GET" Pricing Operation in it. "GET" being the Http verb. Azure API Management allows us to place policies within it. Policies are nothing but a set of rules. These rules can be set at

Product Level (A logical grouping of multiple APIs)
API Level
API Operation Level A policy defined at the Product is applicable to all APIs and API operations under it and similarly, a policy defined at API level is applicable to Operations under it.

Policies can be for

Access Restriction
Authentication Policies
Caching Policies
Cross-Domain Policies
Transformation Polices.

More details can be found here => https://docs.microsoft.com/en-us/azure/api-management/api-management-policies

The policy we are interested in is the Validate JWT policy
https://docs.microsoft.com/en-us/azure/api-management/api-management-access-restriction-policies#ValidateJWT

JWT

JWT stands for JSON Web Token. This text is from Wikipedia "JSON Web Token (JWT, sometimes pronounced /dʒɒt/[1]) is an internet standard for creating data with optional signature and/or optional encryption whose payload holds JSON that asserts some number of claims. The tokens are signed either using a private secret or a public/private key. For example, a server could generate a token that has the claim "logged in as admin" and provide that to a client."
https://en.wikipedia.org/wiki/JSON_Web_Token

Validatge jwt policy

Now let's look at how we can configure the Validate JWT policy.

In the above screenshot, you can see a policy called Validate-jwt.

The policy first looks for the header named "Authorization". The policy says that if the jwt validation fails, an Http Response code 401 should be returned with the message "JWT Validation failed." to the caller.
The OpenId URL for the Okta Authorization Server keys are provided. These keys are used by Azure APIM to validate that the JWT Access Tokens is not a tampered or a fake token. That it really was issued by an Okta Authorization Server.
The Access token issuer is the default Authorization Server. (We can have custom Authorization server also within Okta).
If all the above passes; it knows the user has a valid token for the client App from Okta. So Authentication passed.
Next us the Authorization part. The policy now checks whether the incoming token has the custom claim DealershipLocation and also the values of this is Los Angeles and San Francisco. If the custom claims are present in the incoming request's Access Token the API request is forwarded to http://dummy.restapiexample.com/api/v1/employees. ###Note#### You would see some values hardcoded within the APIM Policies. As a best practice, these should be placed in API Management's "Named Value" facility.

Azure DevOps: How to Build, Test And Deploy to Azure Kubernetes Service

Naseem Mohammed — Wed, 27 May 2020 06:39:05 +0000

I have been using Azure DevOps for a while. Like most of the cloud products out there this is one which gets a constant refresh. My plan is to document the steps for building, testing, and deploying an app to Azure Kubernetes Service using Azure DevOps. So let's start.

Prerequisites

Github or Bitbucket
Azure Kubernetes Service
Azure Container Registry
Soap UI Pro (you need Pro edition for CICD).
Azure DevOps
Azure DevOps Agent hosted on your Windows VM. (needed for SoapUI Pro)

Flow Diagram

1) My code & Dockerfile

What I got is a simple .NET Web API project. And below is my Dockerfile.

2) My Github link to project

https://github.com/mohammednaseem/aksistio-hospital

3) Azure Container Registry ####

This is the repository where the Docker image is hosted. Below is a screenshot of the Azure Container Registry from Azure Portal.

4) Azure Kubernetes Service

Now that we have talked about the prerequisite required we will get right to it. We will walk through how we configure Azure DevOps to build and push Docker image to the registry and then deploying that image to AKS and running integration tests against it using SoapUI Pro

So we will create 2 pipelines in Azure DevOps. (Details below).

The first one is the build pipeline
and the second pipeline is the release pipeline.

Build pipeline

At the end of the build pipeline the expected output is to have a new Docker image in Azure Container registry. This will be the artifact we will deploying to AKS as part of the release pipeline.

Below is the build pipeline's YAML file.

# Dotnet
# Unit the dotnet project. xUnit and NSubtitute
# Docker
# Build and push an image to Azure Container Registry
# https://docs.microsoft.com/azure/devops/pipelines/languages/docker
# Publsih
# Now we get the tag of the published id and update the k8 Deployment yaml image
# https://docs.microsoft.com/azure/devops/pipelines/languages/docker

trigger:
- master

resources:
- repo: self


variables:
  # Container registry service connection established during pipeline creation
  dockerRegistryServiceConnection: '7c46ccde-aa97-4bd0-be94-abcd31bbe20b'
  containerRegistry: 'dockerstore'
  imageRepository: 'hospital'
  dockerfilePath: '**/Dockerfile'
  tag: '$(Build.BuildId)-$(Build.SourceVersion)'

  # Agent VM image name
  vmImageName: 'ubuntu-latest'

stages:
- stage: UnitTestBuildAndPublish
  displayName: Unit Test then Build and Push Docket to Register then Publish of release pipeline
  jobs:    
  - job: UnitTest
    displayName: Running Unit tests for the Hospital Microservice
    pool:
      vmImage: $(vmImageName)
    steps:    
    - task: DotNetCoreCLI@2
      inputs:
        command: 'test'
  - job: Build
    dependsOn: UnitTest
    displayName: Build and push to container registry
    pool:
      vmImage: $(vmImageName)
    steps:    
    - task: Docker@2
      displayName: Build and push an image to container registry
      inputs:
        command: buildAndPush
        repository: 'hospital'
        dockerfile: $(dockerfilePath)
        containerRegistry: $(containerRegistry)
        tags: |
          $(tag)
  - job: PreReleasePrepForhospitalMicroservice
    dependsOn: Build
    displayName: Pre Release Preparation (Bash build id and Publish for Release pipeline)
    pool:
      vmImage: $(vmImageName)
    steps:
    - task: Bash@3
      inputs:
        targetType: 'inline'
        script: |
          # Write your commands here            

            cat '$(Build.SourcesDirectory)/hospital.yaml'              
            value=`cat '$(Build.SourcesDirectory)/hospital.yaml'`              
            value=${value//##BUILD_ID##/$(tag)}            
            echo "$value" > '$(Build.SourcesDirectory)/hospital_build.yaml'             
            value1=`cat '$(Build.SourcesDirectory)/hospital_build.yaml'`             
            echo "$value1"
            mkdir '$(Pipeline.Workspace)/hospital'  
            echo 'after creation of hospital'  
    - task: PublishPipelineArtifact@1
      inputs:
        targetPath: '$(Pipeline.Workspace)'
        artifact: 'hospital'
        publishLocation: 'pipeline'

Before talking about what is happening in above pipeliene; it maybe better to look at Azure Build Pipeline hierachy first.

stages:
- stage: A
  jobs:
  - job: A1
    timeoutInMinutes: 10
    pool:
      vmImage: 'ubuntu-16.04'
    steps:
    - bash: echo "Hello world"
  - job: A2
    steps:
    - bash: echo "A"
- stage: B
  jobs:
  - job: B1
    steps:
    - bash: echo "B"
  - job: B2
    steps:
    - bash: echo "A"

So the hierarchy is like above. You can have a list of stages which is the top level. Underneath it you can have a list of jobs that can further be broken down into steps and then steps into tasks. Also you can assign the kind of Build agent you want at the job level. So you can have one job using Windows 10 agent another using Ubuntu.

OK now that the pipeline hierarchy is clear; let's go back to the original build pipeline I have above. In that we have only 1 Stage called UnitTestBuildAndPublish. But there are 3 jobs within it.

Job1 called UnitTest
Job2 called Build There is a task within this Job called Docker@2 which is for Build and pushing to Azure Container Registry.
Job3 called PreReleasePrepForhospitalMicroservice. There are two tasks withing this. In the first task I use the bash command to get hold of the K8 YAML file and replace the placeholder with the ImageTag. Then in the second task I use the PublishPipelineArtifact. This is kind of pushing the artifact so I can get hold of this in the release pipeline. There I need info on the ImageTag that was pushed to Azure Container Registry. Publish-Pipeline-Artifact

On whether the jobs run in sequence or in parallel. By default, they run parallel. But in the above YAML file you may notice the "dependsOn" tag under each job. You will note that the Build job "dependsOn" Unit test project to complete. And Prepublish job "dependsOn" Build Job. So in essence they are executed in a sequential manner.
So the Sequence is UnitTest > Build(&Push Docker Image to Registry > Prepublish

Variables in above build pipeline.

dockerRegistryServiceConnection: This is the Azure Container Registry's connection string. This is preconfigured.
containerRegistry: 'dockerstore' is the ACR service name,
imageRepository: This is an image repository (or microservice name).
dockerfilePath: Relative path to the Dockerfile in Github
tag: Docker Image tag. We are using the BuildId and the Github Commitid for traceability from Docker Image to Github code.

Release Pipeline

Below is the screenshot of my release pipeline tasks page. As you can see I am using two Agents.

Hosted Agent running Ubuntu 18.04
- Needed to build Linux Docker images for deployment to Linux Nodepool in AKS.
Custom Agent on Windows 10 VM
- SoapUI Pro requires a Windows 10 OS to run its tests.
- API Management tasks also require Windows 10 OS to run.

Download Pipeline Artifact (running on Ubuntu agent)

So the first step is to Download the Pipeline Artifact. We need this for the K8 YAML file. In the Build pipeline we have updated the ##BuildID## placeholder with the real tag of the image that was pushed to ACR.

Kubectl Apply (running on Ubuntu agent)

We will use this task to apply the above YAML file against the "NM" namespace of AKS. The namespace is a way to logically have multiple environments in Kubernetes. The above YAML shows that the pods and services will be deployed to "NM" namespace.

The above command confirms that the services and pods are deployed in that namespace.

Bash Script Task (running on Ubuntu agent)

This is more of a hack I put in there. This task just sleeps and holds the pipeline for a few seconds. I wanted to ensure that the Pods are given enough time to be up and running. This is required because if the Pods are not Up SoapUI Pro will fail all the tests.

Azure SQL Dacpac Task (running on Ubuntu agent)

This is for database deployment. I am not doing anything currently for this project. That is why it is in a disabled state. But this is the task you would use to run the DDL and DML statements against your Azure SQL.

API Management - Create/Update API (running on Windows 10 agent)

This is a task, I got from the marketplace which helps me deploying API definitions to Azure API Management. It accepts OAS 3.0 definition that I build using Swagger editor. Below is my API definition.

openapi: 3.0.0
# Added by API Auto Mocking Plugin
servers:
  - description: SwaggerHub API Auto Mocking
    url: https://virtserver.swaggerhub.com/BRB/Hospital/1.0.0
info:
  description: Demo API on Hospital. 
  version: "1.0.0"
  title: Hospital
  contact:
    email: m.naseem@outlook.com
  license:
    name: Apache 2.0
    url: 'http://www.apache.org/licenses/LICENSE-2.0.html'
tags:
  - name: Hospital
    description: Hospital related matters
paths:
  /hospital/{hospitalId}:
    get:
      tags:
        - Hospital
      summary: Finds hospital by id
      operationId: GetHospitalById
      description: |
        By passing in the valid id, you can search for
        the hospial details in the database
      parameters:
        - in: path
          name: hospitalId
          description: pass the hospitalId for looking up the database
          required: true
          schema:
            type: integer
          example: 415
      responses:
        '200':
          description: search result matching criteria
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Hospital'
        '400':
          description: bad input parameter
    delete:
      tags:
        - Hospital
      summary: deletes a hospital from list
      operationId: DeleteHospial
      description: Deletes a hospital from list
      parameters:
      - name: hospitalId
        in: path
        description: Hospital id to delete
        required: true
        schema:
          type: string
      responses:
        '400':
          description: "Invalid ID supplied"
        '404':
          description: "Hospital not found"
  /hospital:
    post:
      tags:
        - Hospital
      summary: adds a hospital to the list
      operationId: AddHospital
      description: Adds a hospital to the list
      parameters:
      - name: Authorization
        in: header
        required: true
        schema:
          type: string
      responses:
        '201':
          description: hospital added
        '400':
          description: invalid input, object invalid
        '409':
          description: hospital already exists
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Hospital'
        description: Add Hospital to list
    patch:
      tags:
        - Hospital
      summary: Updates a hospital in the list
      operationId: UpdateHospital
      description: Updates a hospital to the list
      responses:
        '200':
          description: hospital updated
        '400':
          description: invalid input, object invalid
        '404':
          description: hospital does not exists
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Hospital'
        description: Updates Hospital in the list
components:
  schemas:
    Hospital:
      type: object
      required:
        - Id
        - Name
        - Address
        - City
        - Pincode
      properties:
        id:
          type: integer
          example: 56
        name:
          type: string
          example: Epidemic Diseases Hospial
        Address:
          type: string
          example: MAJESTIC
        City:
          type: string
          example: Bangalore
        Pincode:
          type: integer
          example: 562110

SoapUI Pro for Azure DevOps (running on Win10 agent)

After all of the above steps are completed; it is time to do the integration tests. And make sure everything is complying. SLA will be met. We didn't break anything.
Once the tests are completed SoapUI exports some of the reports to Azure DevOps. Below is one such report.

Once all the test passes; we can promote the deployments to higher environments. This too can be automated nicely in Azure DevOps. Each environment is called stages and we can add Manual Approver too as part of environment promotion. Below is a screenshot of the stages graphic that Azure DevOps provides.

SQL Server container in Azure Kubernetes Services (AKS)

Naseem Mohammed — Mon, 25 May 2020 03:36:57 +0000

So recently I got involved with an ASP.NET project which was built over 10 years ago and over the years Developers and Change Requests came and went. And over the period the Application became quite cumbersome and quite hard to understand and manage, the Application became quite large in terms of functionality, codebase, and data. It was cumbersome and quite hard to understand for a new developer and manage for the Ops team. A lot of technical debts started accumulating because there was no real-time spend on optimizing or refactoring the systems. The database was slow and deadlocks were becoming normal. It was hosted on huge on-premise hardware making it a heavy and costly solution.

The management realized this and decided it was time for a refresh. The plan was made to look at utilizing cloud technologies. And I joined the project as the software architect for the upgrade/migration.

In this article the focus is on Data and I am putting down my thoughts on what would be the right Cloud Data architecture for this organization.

Let's start with categorizing Data, Use case, and the corresponding system.
Datastore can be categorized into two types of systems.

Transaction
Analytical

And here is the *Data Architecture diagram *

On the left we have the client Apps connecting to Microservices through an API Management platform.
The Microservices are hosted in Azure Kubernetes Service. The apps will be hosted within VMs appropriate for running regular applications.
For database we are going with SQL Server Container: The SQL Server Container we will host in a different Nodepool with VMs optimized for Database usage.
Then for Data Warehousing we will use Snowflake. Snowflake is a SaaS-based Cloud Warehouse. They are optimized for cloud and do not have an on-premise offering.
ETL: To move data from SQL Server to Snowflake we will be using Apache Spark Jobs hosted on Azure Databricks.

This article is going to focus on item #3. The SQL Server Container. The others will be visited in future articles. So now we will walk-through on how to

Deploy SQL Server DB Instance into an AKS Cluster.

We will be using Kubernetes Static Persistent Volume storage for DB as opposed to Kubernetes Dynamic Persistent Volume Storage for the DB. This will give us control when we have to restore an existing Database(s) into the SQL Server Container. Using this approach, we can take regular snapshots of our Azure Disk and restore from the snapshot when disaster strikes. Using Dynamic Provisioning we would not have the control to specify an existing Azure Disk for storage/restoration.
We will look at how we are gaining Performance, High Availability, and have a plan in for Disaster Recovery. (restoration from a Snapshot or restore DB from a backup file).

Databases are stateful. That means it needs storage that can be persisted. So we will first look at storage.

AKS Storage

Applications hosted on Azure Kubernetes Service (AKS) may need to store and retrieve data. The data storage requirements of many types:

Fast local data storage and that need not be persisted after the pod is deleted.
Data storage that needs to be persisted even after the pod is deleted or relocated to some other node in the cluster.
Storage may need to be shared between multiple pods.
Also, their Access Modes required by the applications (like read/write) will be different.
For some application workloads, this data storage can use local, fast storage on the node that is no longer needed when the pods are deleted.
Some storage may be used to inject configuration or sensitive data into pods.

Below we will address four concepts that provide storage to applications in AKS

Volumes
Persistent volumes
Storage classes
Persistent volume claims

Volumes

This is the storage and in Azure it comes in two forms.

Azure Disks (there are many flavors of this. Starting with HDD. SDD, Ultra SDD)
Azure Files For our SQL Server container we will be creating an Azure Disk for the data storage requirements. ###Persistent Volume### A persistent volume is a storage resource that is managed by the Kubernetes Master API that can exist beyond the lifetime of Pod. It can be statically created by the Kubernetes cluster or dynamically provisioned. We will be looking at static provisioning.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: azure-disk-pv
  namespace: db
spec:
  capacity:
    storage: 80Gi
  storageClassName: ""
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  azureDisk:
    kind: Managed
    diskName: Kube_static_disk
    diskURI:  /subscriptions/15cxx96af-xxxxx-xxx-a760-1f58cxxxxxfe/resourceGroups/MC_maltax_southeastasia/providers/Microsoft.Compute/disks/Kube_static_disk

Storage Classes (SC)

A storage class defines the tier (Premium/Standard), Access Modes. Reclaim policy.

Persistent Volume Claim (PVC)

When an application requires some persistent storage from AKS it has an issue a claim or Persistent Volume Claim. This has to define the Storage Class, Access Mode, and Size.
If the annotation like below is set in the PVC; then Kubernetes will try to dynamically create the resource. Assuming a matching storageclass called my-storage-class is found.

volume.beta.kubernetes.io/storage-class: m-azure-disk

But in our case, we are going for Static provisioning. We already created an Azure Disk earlier 80GB size. We also created the Persistent Volume (PV).

Now let’s create a Persistent Volume Claim (PVC).

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mssql-data-pvc
  namespace: db
spec:
  storageClassName: ""
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 80Gi

As you can we don't have the annotation for Dynamic provisioning. Instead Kubernetes will map this PVC with earlier created PV.

The above screenshot confirms the PVC is mapped to PV. Or the claim is mapped to real storage(disk).

Now we got storage sorted. Let's deploy the SQL Server Container onto Kubernetes. Below is the file that has the Kubernetes SQL Server Deployment and Service details. Check the section under Volumes.

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: mssql-deployment
  namespace: db
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: mssql
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: mssql
        image: mcr.microsoft.com/mssql/server:2017-latest
        ports:
        - containerPort: 1433
        env:
        - name: MSSQL_PID
          value: "Developer"
        - name: ACCEPT_EULA
          value: "Y"
        - name: SA_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mssql
              key: SA_PASSWORD 
        volumeMounts:
        - name: mssqldb
          mountPath: /var/opt/mssql
      volumes:
      - name: mssqldb
        persistentVolumeClaim:
          claimName: mssql-data-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: mssql-deployment
  namespace: db
spec:
  selector:
    app: mssql
  ports:
    - protocol: TCP
      port: 1433
      targetPort: 1433
  type: LoadBalancer

We will deploy this. See the below screenshot.

We can see after around 40 seconds the SQL Server Instance is up and running. We are able to connect using sqlcmd command.

We can also see that the pod is running on the VM and below screenshot shows that the VM has mounted the storage disk that we provisioned earlier.

Key Parameters

Now SQL Server is set up and running; let's look at some key parameters.

Performance
High Availability
Disaster Recovery

Performance

In AKS a feature called Azure Accelerated Networking is turned by default. Since the Applications and our SQL Server Container Instance are deployed in the same AKS instance we are automatically able to gain these benefits of this feature.

Significantly improved network performance.
Network throughput of up to 30Gbps.
Reduced latency / higher packets per second (pps).
Reduced jitters.
Decreased CPU Utilization: Less CPU Utilization for processing network traffic.

Another way to gain performance is to use Ultra SSD which scale
performance up to 160,000* IOPS and 2 GB/s per disk with zero downtime.

High Availability

Container Level:
Kubernetes regularly check whether the SQL Server Containers Instances are running healthy. If for some reason the instance crashes or stop being responsive. Kubernetes restarts it. When I tried to delete a SQL Server Pod; Kubernetes instantly detected and spun a new Pod within 4 seconds. See the below screenshot.

Disaster Recovery

Our storage for the container is Azure Disks which provides an SLA of 99.999% and Microsoft had no reported outage till now. But still, we should be prepared for a Disaster. One of the options we have to take regular incremental snapshots of Azure Disks. If a disaster strikes, we can restore the most snapshot back to a Disk. This Disk can be then mounted onto a new SQL Server Instance. But since we have the data (maybe old by a few mins or hours based on our Recovery Point Objective (RPO) of Disaster Recovery strategy).

Another way to get back up and running when a disaster strike is to use SqlPackage command. We can regular backups (automated). Disaster strikes we can spin up a new SQL Server Instance and Azure Disk Storage. Then we will restore the bacpac file back on to the Azure Disk for use by SQL Server.