DEV Community: Shakir

🚀 Sample RAG app with Strands, Reflex and S3

Shakir — Mon, 29 Sep 2025 16:16:13 +0000

Hi 👋, these are some setup instructions for the app that lets store some notes for subjects 📚 in S3 as vectordb and query it with Strands. You may checkout the code here.

First goto S3 >> Vector buckets, and create a vector bucket there and give it some name such as studynotes.

Create an index there with dimension 1024, this is going to be the size of the embedding vector.

Let's now clone the app and switch the directory.

git clone https://github.com/networkandcode/studynotes.git
cd studynotes

So we now have the following for our .env file, add it here.

AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_DEFAULT_REGION="us-west-2"
AWS_S3_VECTOR_BUCKET_NAME=studynotes
AWS_S3_VECTOR_INDEX_NAME=books

Note that its using nova micro in the app for the LLM calls, so make sure it's allowed in Bedrock.

⚡ Run the app with uv/reflex.

uv run reflex run

Creating virtual environment at: .venv

App running at: http://localhost:3000/
Backend running at: http://0.0.0.0:8000

You can now access the app at http://localhost:3000/.

A few screenshots below.

Home page

Subjects

Upload notes

Chat

Ok, so that's it for the post. This was done mainly to see how we can use S3 vector with python for RAG. The code related to S3 vector operations is in this file. The app may not be perfect yet, needs some improvement.

New Relic Template for Strands

Shakir — Sun, 14 Sep 2025 10:05:17 +0000

Hi 👋, we’ll see about observability with New Relic / OTEL, for Strands Agents that shows some quick insights such as tokens used, cost for the tokens(based on a static cost set as variable), request duration, errors etc.

Before you begin

You can check this video for explanantion of the poc app used here, basically we would be using kubernetes mcp to interact with a k3s cluster and opentelemetry will be enabled to generate observability, the observability part for strands was discussed in this post and this video. Optionally, if you'd like to know how to collect data on New Relic through otel collector, please check this post.

Ok, so what's new 🤔, we have to just set the OTLP variables and modify the code a bit so that telemetry is sent to NewRelic endpoint. And a newrelic template was built to visualize the telemetry data.

Code

The code for the lab and the dashboard configuration is present here. You can clone and checkout as follows.

git clone https://github.com/networkandcode/networkandcode.github.io/tree/1014fbcffeda3b61d331421d2cc67d11ca98c597

cd networkandcode.github.io/strands-examples/strands-newrelic-demo/

git switch 1014fbc --detach

$ ls -a
.   .env.example     README.md         k8s_mcp_app.py  pyproject.toml    strands-agent-dashboard.json
..  .python-version  k8s_mcp_agent.py  main.py         set_telemetry.py  uv.lock

ENV

You have to first get the API key of type Ingest License and set it as OTEL_EXPORTER_OTLP_HEADERS=api-key=<the-copied-value>. So the env file would now be as follows.

$ cat .env
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_DEFAULT_REGION=us-west-2

OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp.nr-data.net"
OTEL_EXPORTER_OTLP_HEADERS=api-key=
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

The set_telemetry file is modified.

$ cat set_telemetry.py

from dotenv import load_dotenv
from strands.telemetry import StrandsTelemetry

def set_telemetry():
    load_dotenv()
    strands_telemetry = StrandsTelemetry()
    strands_telemetry.setup_otlp_exporter()

Run

We can now run the app, send some prompts.

$ uv run streamlit run k8s_mcp_app.py

Newrelic

Telemetry data should now flow to newrelic, we can visualize data 📊 and here is a screenshot from the dashboard.

It's a single dashboard with 3 pages 🗂️ to show App level, Request(Trace) level and Operation(Span) level data. The dashboard is available as json here:

$ ls strands-agent-dashboard.json 
strands-agent-dashboard.json

And this can be imported to newrelic as new dashboard. You may have a look and inform if they are any errors in the dashboard. Ok so that's it for the post, thank you 🤝 for reading.

OTEL Observability with Langfuse for Strands Agents

Shakir — Sat, 30 Aug 2025 06:44:51 +0000

Hi 👋, in this post we shall see how to use Open Telemetry and Langfuse cloud to observe LLM calls made via Strands Agents.

UV

Let's initialize our project with uv.

uv init strands-langfuse-demo

Copy the code shown in this video to the project, so that the content looks like below. The final code for this lab is anyway here.

$ cd strands-langfuse-demo/

$ ls -a
.  ..  .env  .python-version  README.md  k8s_mcp_agent.py  k8s_mcp_app.py  main.py  pyproject.toml

Let's activate the virtual environment.

uv venv

source .venv/bin/activate

We can now install these dependencies: streamlit, strands-agents

uv add streamlit==1.49.1
uv add strands-agents[otel]==1.6.0

We are adding the extra otel here, as we need to enable telemetry.

K3S

I am using K3S for launching a single node kubernetes cluster.

curl -sfL https://get.k3s.io | sh -

Let's copy the kubeconfig as we will be referring to this in our agent configuration.

sudo cp /etc/rancher/k3s/k3s.yaml .

Langfuse

Go to Langfuse cloud and create a new organization(for ex. my-org), inside that you can create a new project(for ex. my-llm-org). And generate api keys for the project.

We need to add these variables in our env file. So it looks like below.

$ cat .env
AWS_ACCESS_KEY_ID=""
AWS_SECRET_ACCESS_KEY=""
AWS_DEFAULT_REGION=us-west-2
LANGFUSE_PUBLIC_KEY="pk-lf-.."
LANGFUSE_SECRET_KEY="sk-lf-.."
OTEL_EXPORTER_OTLP_ENDPOINT = "https://cloud.langfuse.com/api/public/otel"

Let's use this code to load the env vars and set the telemetry with otlp headers.

$ cat set_telemetry.py

import os
import base64

from dotenv import load_dotenv
from strands.telemetry import StrandsTelemetry


def set_telemetry():
    load_dotenv()

    # Build Basic Auth header.
    LANGFUSE_AUTH = base64.b64encode(
        f"{os.environ.get('LANGFUSE_PUBLIC_KEY')}:{os.environ.get('LANGFUSE_SECRET_KEY')}".encode()
    ).decode()

    # Configure OpenTelemetry endpoint & headers
    os.environ["OTEL_EXPORTER_OTLP_HEADERS"] = f"Authorization=Basic {LANGFUSE_AUTH}"

    strands_telemetry = StrandsTelemetry()
    strands_telemetry.setup_otlp_exporter()

Since we are loading env above, we can remove it from the agent code. These lines can be removed:

$ cat k8s_mcp_agent.py | grep dotenv
from dotenv import load_dotenv
        load_dotenv()

And replace it with the new file we added.

$ cat k8s_mcp_agent.py | grep telemetry
from set_telemetry import set_telemetry
        set_telemetry()

Streamlit

Let's now run our app.

streamlit run k8s_mcp_app.py

I tried a prompt to get the list of namespaces.

We can see the trace for this in langfuse.

We can add some custom attributes to traces, let's add one by the name session.id with the trace_attributes argument.

import uuid

self.agent = Agent(
                callback_handler=None,
                model="us.amazon.nova-micro-v1:0",
                tools=tools,
                trace_attributes={
                    "session.id": str(uuid.uuid4()),
                },
            )

Which we can now see in the trace in the attributes section.

The final agent code will be:

$ cat k8s_mcp_agent.py

import re

from mcp import stdio_client, StdioServerParameters
from strands import Agent
from strands.tools.mcp import MCPClient
import streamlit as st

from set_telemetry import set_telemetry
import uuid

def remove_html_tags(text_with_html):
    text_with_out_html = re.sub(r"<[^>]+>", "", text_with_html)
    return text_with_out_html

async def stream_result(stream):
    result = ""
    placeholder = st.empty()
    async for chunk in stream:
        if 'data' in chunk:
            result += chunk['data']
            result = remove_html_tags(result)
            placeholder.write(result)


class KubernetesMCPAgent:
    def __init__(self):
        set_telemetry()
        server_params = {
            "command": "npx",
            "args": [
                "-y",
                "kubernetes-mcp-server@latest"
            ],
            "env": {
                "KUBECONFIG": "k3s.yaml"
            }
        }

        self.stdio_mcp_client = MCPClient(lambda: stdio_client(
            StdioServerParameters(
                **server_params
            )
        ))

        with self.stdio_mcp_client:
            # Get the tools from the MCP server
            tools = self.stdio_mcp_client.list_tools_sync()

            # Create an agent with these tools
            self.agent = Agent(
                callback_handler=None,
                model="us.amazon.nova-micro-v1:0",
                tools=tools,
                trace_attributes={
                    "session.id": str(uuid.uuid4()),
                },
            )


    async def send_prompt(self, prompt):
        with self.stdio_mcp_client:
            stream = self.agent.stream_async(prompt)
            await stream_result(stream)


kubernetes_mcp_agent = KubernetesMCPAgent()

That's it for the post, hope we got some insights on LLM observability. Thank you for reading!

A simple screening agent with CrewAI

Shakir — Sat, 28 Jun 2025 17:26:14 +0000

Let's try a simple exercise with CrewAI with a single agent and task, that parses information it has from AWS docs, and prepares a list of questions that could be used for basic screening during interviews. Of course we can do this with chatgpt, gemini etc. with their UI, but having some sort of a code can also help, for repeatable tasks, and for further refining the logic and automation. Also, we shall look into some basic concepts of CrewAI such as agents, tasks, crew as we do this. Let's get started! 🚀

Initialize a project with uv and switch directory.

$ uv init screening_agent
$ cd screening_agent/ 

$ ls -a
.               ..              .git            .gitignore      .python-version main.py         pyproject.toml  README.md

Setup a virtual environment and activate it.

$ uv venv
$ source .venv/bin/activate

Add crewai as dependency and install it.

$ cat pyproject.toml | grep dependencies
dependencies = ["crewai==0.134.0"]

$ uv lock

$ uv sync

LLM

I am using gemini/gemma-3n-e2b-it for this exercise, the API key for which was obtained from Google AI studio. We need to setup .env file to hold the key.

GOOGLE_API_KEY=<Paste-your-key>

We can add the llm setup in a separate python file llm.py.

import os
from crewai import LLM
from dotenv import load_dotenv

load_dotenv()

llm = LLM(
    model='gemini/gemma-3n-e2b-it',
    api_key=os.getenv('GOOGLE_API_KEY'),
    temperature=0.0
)

So we have read the api key from .env file and passed it to the LLM.

Agent

We can then define our agent in agents.py. We are using only one agent in this exercise to keep it simple.

from crewai import Agent
from llm import llm

screening_agent = Agent(
    role="Screening Agent for AWS interviews",
    goal="Get content from AWS official documentation about the topics: {topics}",
    backstory="You will explore relevant links from AWS official documentation https://docs.aws.amazon.com/ about the topics: {topics}"
              "Try as much as possible to gather links that have information about all the topics on the same webpage"
              "You will then prepare a total of 10 questions and their answers"
              "The number of questions to be evenly split among the topics as much as possible"
              "The length of each question should be 1 line"
              "The length of the answer should also be 1 line"
              "Display numbering for the questions from 1 to 10"
              "You also need to capture the link from which that question and answer was prepared",
    verbose=True,
    llm=llm,
)

All in plain English, we are setting proper context with the role, goal and backstory arguments. We are setting verbose to see somewhat detailed output while we run the code, and then map the llm we defined in the previous step. This means each agent can be mapped to a separate llm if required. Also, note that there is one parameter enclosed in curly braces {topics}, we will be passing this later as input when we initialize the crew.

Task

Let's define our task in tasks.py, that will call the agent.

from crewai import Task
from agents import screening_agent

screening_task = Task(
    description=(
        "Get key information about the topics: {topics}\n"
        "Make the content understandable for non technical audience as well"
        "As this task could be used by HR recruiters as well, for screening purposes"
    ),
    expected_output=(
        "There should be 10 sections in the output"
        "Each section should have Question:, Answer: and Reference Link:"
    ),
    agent=screening_agent
)

Again in plain English, give appropriate context with the description and expected_output arguments. Note that {topics} is used here as well just like the agent.

Crew

We finally have the starting point of our code in main.py where our crew will be defined. We need to map it with both the agents and tools. Note that we only have a single tool in this case, if there are more tools, those would be executed sequentially, by default.

from crewai import Crew
from agents import screening_agent
from tasks import screening_task

def main():
    crew = Crew(
        agents=[screening_agent],
        tasks=[screening_task],
    )
    topics_list = ["Redshift, Glue, S3"]
    topics = ', '.join(topics_list)

    result = crew.kickoff(
        inputs={"topics": topics}
    )

    with open('output.md', 'w') as f:
        f.write(result.raw)

if __name__ == "__main__":
    main()

Note that in the crew kickoff method, we are passing topics as input which we saw was referred to in both agent and tool configuration. In the code above we are trying to pass the topics as Redshift, Glue, S3 which would be passed to the task and then to the agent, to retreive the questions and answers, we need for screening, the output is finally saved in a markdown file mentioned. Let's try runnig it with uv run main.py.

Here is a preview of the markdown generated.

So that's the end of this post, hope it was beginner friendly, and we just navigated some fundamentals of CrewAI, we could explore further on this such as passing the topics via user input, build multi tool based agents, add tools, mcp, rag and see options for integrating it with UI based tools for day to day use. Thank you for reading!!!

GitHub MCP with Amazon Q CLI

Shakir — Tue, 06 May 2025 06:31:02 +0000

Hi 👋, let's explore some MCP stuff with the amazon q developer cli. We shall try onboarding a new nextjs app with boilerplate code on GitHub in this exercise. We then just clone the code to our machine and run it locally. Let's get started 🚀

Install the q cli.

brew install amazon-q

Once installed open Amazon Q from the launchpad and follow the instructions there to integrate Q with shell, and then login with a builder ID.

Let's create a directory where we can keep our mcp code.

mkdir app-onboarding-with-q
cd app-onboarding-with-q

GitHub MCP server

We can now add the github mcp config.

Create a personal access token on github developer settings with access to All repositories and read and write permissions for Administration and Content.

Add the mcp configuration in the directory on the file .amazonq/mcp.json.

{
    "mcpServers": {
      "github": {
        "command": "docker",
        "args": [
          "run",
          "-i",
          "--rm",
          "-e",
          "GITHUB_PERSONAL_ACCESS_TOKEN",
          "ghcr.io/github/github-mcp-server"
        ],
        "env": {
          "GITHUB_PERSONAL_ACCESS_TOKEN": "<your-github-pat>"
        }
      }
    }
}

Start the docker engine on your machine as the github mcp server defined above needs docker.

Launch q from the cli.

$ q

You should see that the mcp server is launched.

✓ github loaded in 0.33 s
✓ 1 of 1 mcp servers initialized

This should have started a docker container for us.

$ docker ps | grep github
76ff6beb314c   ghcr.io/github/github-mcp-server              "./github-mcp-server…"   26 seconds ago   Up 26 seconds                                                                                                   stupefied_heyrovsky

On the q-prompt if you enter /tools you should see the available tools including the MCP tools. Here is a glimpse.

github (MCP):
- github___search_issues                      * not trusted
- github___get_pull_request_reviews           * not trusted
- github___get_code_scanning_alert            * not trusted
- github___create_pull_request_review         * not trusted

There are certain built in tools as well, other than the mcp tools we installed.

Built-in:
- execute_bash    * trust read-only commands
- fs_write        * not trusted
- use_aws         * trust read-only commands
- fs_read         * trusted
- report_issue    * trusted

Let's create a nextjs boilerplate app.

> create a new repo in github with name nextjs-new-app and put nextjs bolierplate code in it, with the github mcp server

For brevity, Iam not going to copy the complete response, but we will see some key details from it. First, it will try to create a repo and ask us for confirmation.

🛠️  Using tool: create_repository from mcp server github
Allow this action? Use 't' to trust (always allow) this tool for the session. [y/n/t]:

We can press y to confirm. The repo is now created 🔥.

Note that we could also trust the tool by pressing t, however it's better to confirm actions for now(human in the loop), so that we are sure what the action would do.

Let's continue...

Amazon Q would then try to push multiple files to our repo.

🛠️  Using tool: push_files from mcp server github

Press y again after the which the files are all there.

Run the app locally

Let's once try to run the app locally and see if it's all working, for which we have to clone it.

> clone the repo you just created and run the app
🛠️  Using tool: execute_bash

It will then try to do a few actions with the execute_bash tool.

 ● I will run the following shell command: 
cd /Users/networkandcode/app-onboarding-with-q && git clone https://github.com/networkandcode/nextjs-new-app.git

 ● I will run the following shell command: 
cd /Users/networkandcode/app-onboarding-with-q/nextjs-new-app && npm install

 ● I will run the following shell command: 
cd /Users/networkandcode/app-onboarding-with-q/nextjs-new-app && npm run dev

The app should have now started.

> y

> nextjs-new-app@0.1.0 dev
> next dev

 ⚠ Port 3000 is in use, trying 3001 instead.
  ▲ Next.js 14.2.28
  - Local:        http://localhost:3001

 ✓ Starting...
 ✓ Ready in 2.7s

We can try accessing http://localhost:3001 as shown above.

We can press ctrl c when done, so that we are back to the q prompt.
We can type /quit or /exit on the q-prompt anytime to exit.

Let's modify our code

> modify the code in the nextjs repo networkandcode/nextjs-new-app to show an analog clock on the home page with utc +5.30 time, note that it is written in typescript and has the src folder

It first tries to read the existing contents 🛠️ Using tool: get_file_contents from mcp server github. Note that I resumed this exercise a bit later so had to give some extra context.

And then it would try to update files 🛠️ Using tool: create_or_update_file from mcp server github

If we try running it again we should see the clock. Note that I have given a few more prompts to modify code.

As next steps we could try deploying this app on platforms such as AWS Amplify(I think there is no MCP server as of this writing for Amplify deployments). Also, usually we write code first on the local system(we could use the filesystem mcp for that) and then push it to the git repo, however in this blog we tried the other way round.

OTEL Auto Instrumentation Demo for SpringBoot

Shakir — Fri, 11 Oct 2024 17:36:59 +0000

Hello 👋, this post is about auto instrumenting a spring boot app with the otel agent running on EKS, forward the telemetry signals(metrics, logs, traces) to backends such as prometheus, loki, tempo via the open telemetry collector, and visualize those in Grafana. The setup is as in the diagram below:

Note that the datasources are added on Grafana, which means Grafana sends them API requests to fetch the data.

Alright, let's get started!

Prerequisites

Ensure you have installed tools such as the aws cli, eksctl, git, docker, kubectl, helm and that you have authenticated to aws from cli.

Setup the demo app

Let's clone a simple spring boot app from github, and switch to complete directory, where the application code is present.

git clone https://github.com/spring-guides/gs-rest-service.git

cd gs-rest-service/complete

We can add a logger to the application in the controller class.

cat > src/main/java/com/example/restservice/GreetingController.java <<EOF 
package com.example.restservice;

import java.util.concurrent.atomic.AtomicLong;

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

@RestController
public class GreetingController {

        private static final String template = "Hello, %s!";
        private final AtomicLong counter = new AtomicLong();

        private final Logger logger = LoggerFactory.getLogger(GreetingController.class);

        @GetMapping("/greeting")
        public Greeting greeting(@RequestParam(value = "name", defaultValue = "World") String name) {
                logger.info("Received GET greeting request for {}", name);
                return new Greeting(counter.incrementAndGet(), String.format(template, name));
        }
} 
EOF

Build the Docker image

Add a dockerfile to containerize this application, also include the otel agent in it with the curl command. Note that I'm using maven for the build, however the app has both maven and gradle specs, so you can build it with gradle too just by changing the line in dockerfile from ./mvnw clean package to ./gradlew clean build.

cat > Dockerfile <<EOF
# Build
FROM eclipse-temurin:17-jdk-jammy AS build
WORKDIR /usr/app
COPY . .
RUN ./mvnw clean package

# Run
FROM eclipse-temurin:17-jre-jammy
WORKDIR /usr/app
COPY --from=build /usr/app/target/*.jar ./app.jar
RUN curl -O -L https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar
RUN useradd appuser && chown -R appuser /usr/app
USER appuser
HEALTHCHECK --interval=30s --timeout=5s --start-period=5s --retries=3   CMD curl -f http://localhost:8080/ || exit 1 
ENTRYPOINT ["java", "-javaagent:/usr/app/opentelemetry-javaagent.jar", "-jar", "/usr/app/app.jar"]
EOF

We have added the healthcheck command and a nonroot user in the dockerfile above, so that it passes security checks. We can lint/scan it with trivy as follows.

docker run --rm -v ./Dockerfile:/tmp/Dockerfile aquasec/trivy config /tmp/Dockerfile

Add the dockerignore file so that we are not copying unwanted files to the image.

cat > .dockerignore <<EOF
Dockerfile
manifest.yml
build
target
EOF

We can build the image now.

export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

export REGION=us-east-1

docker build -t $ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com/otel-auto-java-demo .

Push the image to ECR

The image we built can be pushed to ECR, for which we have to first create a repo there.

aws ecr create-repository --repository-name otel-auto-java-demo --region $REGION

Then, get the ecr password and login with it from docker.

aws ecr get-login-password --region $REGION | docker login --username AWS --password-stdin $ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com

You should get Login Succeeded.

The image can now be pushed to ECR with the docker cli.

docker push $ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com/otel-auto-java-demo

Create EKS cluster

I'm going to create an EKS cluster with the eksctl cli.

eksctl create cluster --name otel-auto-java-demo --zones=us-east-1a,us-east-1b

When this is done, your kubeconfig should have got updated and you should have scoped to the new cluster. However you can keep this command handy to switch to the correct context aws eks update-kubeconfig --region $REGION --name otel-auto-java-demo.

Setup helmfile

We can deploy our tool stack with helmfile. First check the architecture, for ex. with a command like uname -m and then install helmfile with the correct package. In my case the architecture was x86_64 and hence, I am installing the linux_amd64 package.

uname -m
wget https://github.com/helmfile/helmfile/releases/download/v1.0.0-rc.5/helmfile_1.0.0-rc.5_linux_amd64.tar.gz

tar -xvf helmfile_1.0.0-rc.5_linux_amd64.tar.gz 

rm LICENSE README.md README-zh_CN.md 

sudo mv helmfile /usr/bin/helmfile

Note that you would also need helm as helmfile is a wrapper around helm.

Our helmfile should look as follows.

cat > helmfile.yaml <<EOF
repositories:
- name: open-telemetry
  url: https://open-telemetry.github.io/opentelemetry-helm-charts
- name: prometheus-community
  url: https://prometheus-community.github.io/helm-charts
- name: grafana
  url: https://grafana.github.io/helm-charts

releases:
- name: otel-collector
  chart: open-telemetry/opentelemetry-collector
  namespace: otel-auto-java-demo
  values:
  - otel-collector-values.yaml
- name: prometheus
  chart: prometheus-community/prometheus
  namespace: otel-auto-java-demo
  values:
  - prometheus-values.yaml
- name: loki
  chart: grafana/loki
  namespace: otel-auto-java-demo
  values:
  - loki-values.yaml
- name: tempo
  chart: grafana/tempo-distributed
  namespace: otel-auto-java-demo
  values:
  - tempo-values.yaml
- name: grafana
  chart: grafana/grafana
  namespace: otel-auto-java-demo
  values:
  - grafana-values.yaml
EOF

Setup Helm values

This section shows the different values files, that we are using in the helmfile.

OTEL collector

Let's begin with the otel collector values. Some of the sections such as receivers are omitted, as anyway they will come from the default values file.

cat > otel-collector-values.yaml <<EOF
image:
  repository: "otel/opentelemetry-collector-contrib"
mode: deployment
resources:
  limits:
    cpu: 250m
    memory: 512Mi
config:
  processors:
    batch:
      send_batch_max_size: 500
      send_batch_size: 50
      timeout: 5s
  exporters:
    otlphttp/prometheus:
      endpoint: "http://prometheus-server/api/v1/otlp"
    loki:
      endpoint: "http://loki-write:3100/loki/api/v1/push"
    otlp:
      endpoint: "http://tempo-distributor:4317"
      tls:
        insecure: true
  service:
    pipelines:
      metrics:
        receivers: [otlp]
        exporters: [otlphttp/prometheus]
      logs:
        exporters: [loki]
      traces:
        receivers: [otlp]
        exporters: [otlp]
EOF

Prometheus

We are disabling components that we do not need for this exercise, from the prometheus chart, and we are enabling otlp write receiver, as we are sending metric from the otel collector to prometheus via otlp/http.

cat > prometheus-values.yaml <<EOF
alertmanager:
  enabled: false
prometheus-pushgateway:
  enabled: false
prometheus-node-exporter:
  enabled: false
kube-state-metrics:
  enabled: false
server:
  extraFlags:
  - "enable-feature=exemplar-storage"
  - "enable-feature=otlp-write-receiver"
EOF

Loki

We are installing loki in the simple scalable mode. And have disabled some of the components such as chunks cache, results cache, gateway to keep it simple.

cat > loki-values.yaml <<EOF
# https://grafana.com/docs/loki/latest/operations/caching/
chunksCache:
  enabled: false
resultsCache:
  enabled: false
gateway:
  enabled: false

limits_config:
  allow_structured_metadata: true

loki:
  auth_enabled: false
  schemaConfig:
    configs:
      - from: 2024-04-01
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h
  ingester:
    chunk_encoding: snappy
  tracing:
    enabled: true
  querier:
    # Default is 4, if you have enough memory and CPU you can increase, reduce if OOMing
    max_concurrent: 4

deploymentMode: SimpleScalable

backend:
  replicas: 3
read:
  replicas: 3
write:
  replicas: 3

# Enable minio for storage
minio:
  enabled: true

# Zero out replica counts of other deployment modes
singleBinary:
  replicas: 0

ingester:
  replicas: 0
querier:
  replicas: 0
queryFrontend:
  replicas: 0
queryScheduler:
  replicas: 0
distributor:
  replicas: 0
compactor:
  replicas: 0
indexGateway:
  replicas: 0
bloomCompactor:
  replicas: 0
bloomGateway:
  replicas: 0
EOF

Tempo

This is our tempo values. Note that we are using the tempo distributed chart, i.e. the microservices mode.

cat > tempo-values.yaml <<EOF 
gateway:
  enabled: true
traces:
  otlp:
    grpc:
      enabled: true
EOF

Grafana

And finally grafana, with datasources provisioned.

cat > grafana-values.yaml <<EOF
service:
  type: LoadBalancer
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      url: http://prometheus-server
    - name: Loki
      type: loki
      url: http://loki-read:3100
    - name: Tempo
      type: tempo
      url: http://tempo-gateway
EOF

Setup prerequisites for provisioning volumes

We need to setup a few things on AWS so that we are able to provision volumes that are needed for some of our tools. First we have to install the CSI driver on the cluster.

eksctl create addon --name aws-ebs-csi-driver --cluster otel-auto-java-demo --region $REGION

We have to then the modify the nodegroup's IAM role with appropriate permissions, for which we have to first find the role. Note that the role suffix could be different in your case.

$ aws iam list-roles | jq -r '.Roles[].RoleName' | grep  otel-auto-java | grep node
eksctl-otel-auto-java-demo-nodegr-NodeInstanceRole-tUmFVlpwtUfc

We can then attach the appropriate policy to the role.

$ aws iam attach-role-policy   --role-name eksctl-otel-auto-java-demo-nodegr-NodeInstanceRole-tUmFVlpwtUfc   --policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy

And then we can set the gp2 storage class as default. This is because the volumes we install would look for the default storage class.

$ kubectl patch storageclass gp2 -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
storageclass.storage.k8s.io/gp2 patched

The word default should now appear in the storage class.

$ kubectl get storageclass
NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  20h

Deploy the tool stack

Alright, done some heavy lifting, we can install our tool stack now.

helmfile sync

Once the installation is complete you can check the release and pods status with commands such as helm ls -A, kubectl get po -A.

Deploy the demo app

We should now deploy the demo app with the following manifest, which has both deployment and service configuration.

cat > demoapp.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-auto-java-demo
  labels:
    app: otel-auto-java-demo
  namespace: otel-auto-java-demo
spec:
  selector:
    matchLabels:
      app: otel-auto-java-demo
  template:
    metadata:
      labels:
        app: otel-auto-java-demo
    spec:
      containers:
        - name: otel-auto-java-demo
          image: $ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com/otel-auto-java-demo:latest
          # https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/
          env:
          - name: OTEL_EXPORTER_OTLP_ENDPOINT
            value: "http://otel-collector-opentelemetry-collector:4317"
          - name: OTEL_EXPORTER_OTLP_PROTOCOL
            value: grpc
          imagePullPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: otel-auto-java-demo
  namespace: otel-auto-java-demo
spec:
  type: ClusterIP
  selector:
    app: otel-auto-java-demo
  ports:
    - port: 80
      targetPort: 8080
EOF

Note that the above file has env vars for account id and region, hence we can use envsubst before applying the manifest.

envsubst < demoapp.yaml | kubectl apply -f -

Make some api calls to the app

We can expose the service, so that we can send some calls on the localhost.

kubectl port-forward svc/otel-java-demo -n otel-java-demo 8080:80

Make a few api calls with curl, from a different terminal.

$ curl localhost:8080/greeting; echo
{"id":8,"content":"Hello, World!"}

$ curl localhost:8080/greeting?name=Earth; echo
{"id":13,"content":"Hello, Earth!"}

$ curl localhost:8080/greeting?name=Universe; echo
{"id":14,"content":"Hello, Universe!"}

When done, you may stop the port fowarding with Ctrl C.

Monitor the app

We can now monitor our app from Grafana with the LoadBalancer URL of Grafana.

kubectl get svc grafana -n otel-auto-java-demo
NAME      TYPE           CLUSTER-IP     EXTERNAL-IP                                                               PORT(S)        AGE
grafana   LoadBalancer   10.100.69.33   acdbcf3ef71674a0b9a8be554d62db7c-2016095508.us-east-1.elb.amazonaws.com   80:32283/TCP   73m

Just access the URL above with http:// on the browser, the URL would be different in your case. The username is admin, and the password can be obtained from the secret as shown below. The password would be different in your case.

$ kubectl get secret grafana -n otel-auto-java-demo -o jsonpath={.data.admin-password} | base64 -d ; echo
3cOFbY8Q5H5Qz4CHHJvJZ2a2UDvSeQnmRrFIQlsv

Go to explore tab on let's try one query each for metrics, logs and traces.

Metrics preview:

Logs preview:

Traces preview:

Note that we only added two env vars in our demo app, we haven't added any extra vars such as resource attributes, service name etc. Just to show that it would pick it from the artifact id if not provided. As shown in the image previews above, for metrics(prometheus) the it would become the job label, and for logs(loki) it would become the job and service_name label and for traces(tempo) it would be the resource.service.name tag.

$ cat pom.xml
--TRUNCATED--
<groupId>com.example</groupId>
    <artifactId>rest-service-complete</artifactId>
--TRUNCATED--

So, we were able to get the telemetry data successfully from our SpringBoot Java App all the way down to Grafana. With this as base, we could now build custom dashboards as we like, do some correlation by modifying datasource config, etc.

Cleanup checklist

Delete the helm releases: helmfile destroy
Delete the PVCs: kubectl delete pvc --all -n otel-auto-java-demo
Delete the demo app (optional): kubectl delete -f demoapp.yaml
Delete the cluster: eksctl delete cluster --name otel-auto-java-demo --region $REGION
Delete the ECR repo: aws ecr delete-repository --repository-name otel-auto-java-demo --region $REGION --force

Thank you for reading :)

AWS ALB(Ingress) access logs on Grafana

Shakir — Tue, 01 Oct 2024 10:05:48 +0000

Hi 👋, we shall explore the following use case in this post: Let's say you have an EKS cluster, and you have deployed an application there and exposed it via the AWS ALB controller as ingress(instead of other options like nginx ingress, istio virtual service etc.). We'll see how to check the access logs for this scenario, by forwarding the access logs from the application load balancer to a bucket on S3. And then use an event trigger when new logs are added to S3, to invoke a Lambda function that would write those logs to a Cloud watch log group and stream. We could then use Cloud watch as a datasource on Grafana to visualize/monitor the logs.

So the setup is going to look like this:

Note that though the arrow on the diagram is one sided, it's only referring to who initiates the traffic(request) first, in most cases there is going to be some response back.

EKS cluster

We need to first create a cluster and associate oidc provider with it.

eksctl create cluster --name ingress-logs-demo --zones=us-east-1a,us-east-1b

eksctl utils associate-iam-oidc-provider --region=us-east-1 --cluster=ingress-logs-demo --approve

Ingress

First create a service account with the built in load balancer controller role.

eksctl create iamserviceaccount \
  --cluster=ingress-logs-demo \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --role-name AmazonEKSLoadBalancerControllerRole \
  --attach-policy-arn=arn:aws:iam::$AWS_ACCOUNT_ID:policy/AWSLoadBalancerControllerIAMPolicy \
  --approve

Replace $AWS_ACCOUNT_ID with your account id.

We can now install the AWS Loadbalancer Controller, and map it with the service account we just created.

helm repo add eks https://aws.github.io/eks-charts

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \ 
-n kube-system \ 
--set clusterName=$CLUSTER \ 
--set serviceAccount.create=false \ 
--set serviceAccount.name=aws-load-balancer-controller

Demo app

We can install a demo app - 2048 game that comes with ingress as well.

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.7.2/docs/examples/2048/2048_full.yaml

This should have created an ingress resource for us.

kubectl get ing -n game-2048
NAME           CLASS   HOSTS   ADDRESS                                                                  PORTS   AGE
ingress-2048   alb     *       k8s-game2048-ingress2-cfcea01af4-806164108.us-east-1.elb.amazonaws.com   80      3d3h

We could access the URL with the URL above(it would change in your case).

Access logs

Let's say we want to monitor our app, like how many requests are coming in, how many are errors, what's the response time etc. In that case we can catch the access logs that are going to our app. For this we need to go the application loadbalancer that was created when we installed the aws load balancer controller via helm. And update it to forward the acces logs to S3. Which means we first need to create the S3 bucket.

S3

I have created an S3 bucket with the name ingress-logs-demo in the us-east-2 region with the following policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::127311923021:root"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::ingress-logs-demo/AWSLogs/<AWS_ACCOUNT_ID>/*"
        }
    ]
}

Note that 127311923021 refers to the AWS account for loadbalancers in the us-east-2 region. Check this link for more info. Also, replace <AWS_ACCOUNT_ID> with your account id. The policy above means any loadblancer created in the us-east-2 region should be able to put objects in this bucket in the defined path.

We can now modify the application loadbalancer to forward logs to S3.

If we start using the app ui now via browser and do a few hits, we should be able to see the access logs for those on S3 in compressed format.

Cloudwatch Logs

We can create a log watch group(/aws/lambda/ingress-logs-demo) and stream(access-logs) where we can forward the access logs from S3. Note that I am going to use the same log group for both the access logs(alb to s3 to cloud watch) and for any logs coming from the lambda function itself. That is why I have named the logwatch group with that format, as any logs from a function with name (ingress-logs-demo) would be stored in (/aws/lambda/ingress-logs-demo)

Lambda

We should need a way to retreive these logs and forward those to Cloudwatch, for which I would be using a Lambda function in Python with the same name ingress-logs-demo(for simplicity) with the following code.

import boto3
import gzip
import io
from datetime import datetime

s3_client = boto3.client('s3')
logs_client = boto3.client('logs')

log_group_name = '/aws/lambda/ingress-logs-demo'
log_stream_name = 'access-logs'

def lambda_handler(event, context):
    # Get S3 bucket and log file details from the event
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = event['Records'][0]['s3']['object']['key']

    # Download the log file from S3
    response = s3_client.get_object(Bucket=bucket, Key=key)

    # Read the gzipped file
    with gzip.GzipFile(fileobj=io.BytesIO(response['Body'].read())) as file:
        logs = file.read().decode('utf-8').splitlines()

    # Send each log line as an event to CloudWatch
    log_events = []
    for line in logs:
        timestamp = int(datetime.now().timestamp() * 1000)


        log_events.append({
            'timestamp': timestamp,
            'message': line
        })

    if log_events:
        logs_client.put_log_events(
            logGroupName=log_group_name,
            logStreamName=log_stream_name,
            logEvents=log_events
        )

    return {
        'statusCode': 200,
        'body': f'Successfully processed {len(log_events)} log events from {key}'
    }

Note the we are passing the logs as is but we could use some logic in our code, to parse the content and restructure it as desired. Please check this link for the log format.

Also we need a trigger to invoke this function when any objects are added to S3.

Role

We need an IAM role that lets the lambda function read from S3 and write to Cloudwatch Logs. We can create a new one with name ingress-logs-demo, trusted entity type as AWS service, Service or use case as Lambda and in the permission policies, select AWSLambdaBasicExecutionRole and AmazonS3ReadOnlyAccess.

All set

All set now, we can now hit the 2048 game url on browser a couple of times, and try playing the game. This should run our workflow and store logs in cloud watch.

Grafana

We can now go to Grafana, add cloudwatch as a datasource and play with the data and do some visualization, display the logs etc.
Let's try a couple of exercises with panels.

Just filter for logs that has "200" typically meaning we wan't to see logs that have response code 200.

Let's check 404 this time. And create a stat panel with it.

I have used this query:

fields @timestamp, @message
| filter @logStream = "access-logs"
| filter @message like /"404"/
| sort @timestamp desc

and the calculation was set to count for the field @message

Alright, so that's end the post, we were finally able to get our access logs on cloudwatch and see those on Grafana. This should help us monitoring any urls thats accessed through the ingress.

Cleanup

Checklist:

Delete the function
Delete the IAM role
Empty the S3 bucket and delete it
Delete the cloudwatch log group
Delete the load balancer controller and remove the helm repo

helm uninstall aws-load-balancer-controller -n kube-system
helm repo remove eks

Delete the eks cluster: eksctl delete cluster --name ingress-logs-demo

Thank you for reading :)

OTEL Demo with EKS and New Relic

Shakir — Thu, 18 Jul 2024 04:09:25 +0000

Hi👋, In this post we shall deploy the OTEL demo app on EKS, and send the metrics, logs and traces to New Relic.

Prerequisites

It's assumed you know how to install and use tools such as eksctl, kubectl, helm, helmfile and know some observability basics.

Cluster

Create a cluster on EKS with this command.

eksctl create cluster --name newrelic-otel-demo --zones=us-east-1a,us-east-1b

Once the cluster is created, the kubeconfig's current context will be automatically changed to it.

$ kubectl config current-context
nc@newrelic-otel-demo.us-east-1.eksctl.io

License

We need a license key🔑 first from New Relic, log in to the New Relic website and headerover to Administration > API keys and copy the license key and set it as a variable. Note that the free account is sufficient for this demo.

export NEW_RELIC_LICENSE_KEY='<NEW_RELIC_LICENSE_KEY>'

We can create a namespace for our demo and keep a secret in it for the license.

$ kubectl create ns otel-demo-newrelic
namespace/newrelic-otel-demo created

$ kubectl create secret generic newrelic-license-key --from-literal=license-key="$NEW_RELIC_LICENSE_KEY" -n otel-demo-newrelic
secret/newrelic-license-key created

Helm

We would be using the following helm values. We have disabled extra components that come along with the helm chart which we do not need for this demo.

$ cat otel-demo-newrelic-values.yaml
prometheus:
  enabled: false
opensearch:
  enabled: false
grafana:
  enabled: false
jaeger:
  enabled: false

opentelemetry-collector:
  config:
    exporters:
      otlphttp/newrelic:
        endpoint: https://otlp.nr-data.net:4317
        headers:
          api-key: ${NEW_RELIC_LICENSE_KEY}
        tls:
          insecure: false
    service:
      pipelines:
        logs:
          exporters: [otlphttp/newrelic, debug]
        metrics:
          exporters: [otlphttp/newrelic, debug]
          receivers: [httpcheck/frontendproxy, redis, otlp]
        traces:
          exporters: [otlphttp/newrelic, debug]
  extraEnvs:
    - name: NEW_RELIC_LICENSE_KEY
      valueFrom:
        secretKeyRef:
          key: license-key
          name: newrelic-license-key
    - name: OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE
      value: delta

And the helmfile would refer to the public helm chart and the helm values file we just defined above.

$ cat otel-demo-newrelic-helmfile.yaml
repositories:
- name: open-telemetry
  url: https://open-telemetry.github.io/opentelemetry-helm-charts

releases:
- name: otel-demo-newrelic
  chart: open-telemetry/opentelemetry-demo
  namespace: otel-demo-newrelic
  values:
  - otel-demo-newrelic-values.yaml

It's time to deploy the helm release.

helmfile sync -f otel-demo-newrelic-helmfile.yaml

We can check the helm release.

$ helm ls -n otel-demo-newrelic
NAME                NAMESPACE           REVISION    UPDATED                                 STATUS      CHART                       APP VERSION
otel-demo-newrelic  otel-demo-newrelic  1           2024-07-17 22:02:13.440621 +0530 IST    deployed    opentelemetry-demo-0.32.1   1.11.0

Pods

In some time, all the pods should be running. I had seen image pull back off for sometime, however it fixed itself in a while.

$ kubectl get po -n otel-demo-newrelic
NAME                                                        READY   STATUS    RESTARTS   AGE
otel-demo-newrelic-accountingservice-76fc85f7f9-vxps4       1/1     Running   0          25m
otel-demo-newrelic-adservice-7f8964f98-z2nqd                1/1     Running   0          25m
otel-demo-newrelic-cartservice-bbdfdc59c-w4cqg              1/1     Running   0          25m
otel-demo-newrelic-checkoutservice-f6fd788db-khgq6          1/1     Running   0          25m
otel-demo-newrelic-currencyservice-f9f77f79d-8n946          1/1     Running   0          25m
otel-demo-newrelic-emailservice-66c84d6b64-frs8b            1/1     Running   0          25m
otel-demo-newrelic-flagd-7444ccd98d-cknxl                   1/1     Running   0          25m
otel-demo-newrelic-frauddetectionservice-7c85f7c6bf-mpbw9   1/1     Running   0          25m
otel-demo-newrelic-frontend-c995cfcdd-dsgvc                 1/1     Running   0          25m
otel-demo-newrelic-frontendproxy-76bb8f75cf-kzmn6           1/1     Running   0          25m
otel-demo-newrelic-imageprovider-66db4b77cf-7rg97           1/1     Running   0          25m
otel-demo-newrelic-kafka-74fbd4c749-dspqm                   1/1     Running   0          25m
otel-demo-newrelic-loadgenerator-864475d886-wtnxs           1/1     Running   0          25m
otel-demo-newrelic-otelcol-6b89d8dd84-l5mx4                 1/1     Running   0          25m
otel-demo-newrelic-paymentservice-59677d8c8f-bds4f          1/1     Running   0          25m
otel-demo-newrelic-productcatalogservice-845d79f865-6hs7c   1/1     Running   0          25m
otel-demo-newrelic-quoteservice-76b5f85685-5zz86            1/1     Running   0          25m
otel-demo-newrelic-recommendationservice-8d6bb74c-6zfm4     1/1     Running   0          25m
otel-demo-newrelic-shippingservice-6d44b76489-n4tlw         1/1     Running   0          25m
otel-demo-newrelic-valkey-7f5ffc95ff-wz9zg                  1/1     Running   0          25m

New Relic

We can now go to New Relic and see if the data is flowing. Check All Entities > Services-OpenTelemetry, we should start seeing the data for all our services.

We could either click on one of the service names on the pic above, or choose a service from the APM & Services section to see quite a lot of information about a particular service including built-in dashboards📈.

We could also check the traces page for a particular service, as we are sending traces too, based on our otel collector configuration.

Already a lot of information, forget about creating custom panels or alerts 😃, understading and managing what we have already given by New Relic is in itself a great administration task😉.

Yet we will try to run a simple query using the query builder.

We could make custom dashboards with such queries. Alright so that's the end of this post. We just saw how we can use the open telemetry collector to send observability data to newrelic and just explored some of the UI functionalities on New Relic. See below for the housekeeping check.

Cleanup

Note that free account comes with a data ingestion limit of 100GB per month. I think it ingested around 17GB approximately for about 10 hours(I left my cluster ON and more than 85% of data was coming from traces). So it's a good idea to stop sending traffic once your demo is done, so you can leverage the free limit later when required.

Delete the namespace to delete all the objects we created in the kubernetes cluster for this demo. This would stop sending data to newrelic.

kubectl delete ns otel-demo-newrelic
namespace "otel-demo-newrelic" deleted

Also you may unset the license key variable.

unset NEW_RELIC_LICENSE_KEY

Finally delete the cluster.

eksctl delete cluster --name newrelic-otel-demo --region=us-east-1

Thank you for reading !!!.

Grafana OTEL App Observability Demo

Shakir — Fri, 12 Jul 2024 17:20:00 +0000

Introduction

We are going to explore the built in app observability solution on Grafana cloud, which generates some built in dashboards for us. We would be using OTEL traces for this particular demo. You may check the video version here:

We shall install a sample demo app that simulates a microservices environment. It would generate otel compatible traces which are sent to a local alloy instance. From the local alloy, the traces are sent to Grafana's OTLP gateway(alloy) on the cloud.

The setup would typically look as below.

Note that certain metrics will be generated from the traces. So traces will be sent to tempo and the generated metrics from the traces will be sent to Mimir. Both Mimir and Tempo on the cloud are available as datasources in Grafana cloud.

OTLP endpoint

Go to grafana.com, sign in and access your account, the url of which would be like https://grafana.com/orgs/<org-name>. Click on the Stack at the top of the page. In my case, I am using a free account and the organization and stack name are both same. Note that I can only have one organization in the free account.

In the manage stack page, there should be an OTLP section, click on it and copy the gateway URL and instance id from there. Note that the instance id would be our username as well. On this page, you may generate a token with the predefined scope set:alloy-data-write. This has the sub scopes metrics:write, logs:write, traces:write, profiles:write, fleet-management:read. I have generated a new API token with the predefined options and given it a name app-observability-demo. Ensure you copy the token.

Alloy config

My alloy config would look as below. We need to put the username and password from what we collected in the previous step.

$ cat config.alloy 

otelcol.receiver.otlp "otlp_receiver" {
  grpc {
    endpoint = "0.0.0.0:4317"
  }

  output {
    traces = [otelcol.exporter.otlphttp.grafana_cloud.input,]
  }
}

otelcol.exporter.otlphttp "grafana_cloud" {
    client {
        endpoint = "https://otlp-gateway-prod-us-east-0.grafana.net/otlp"
        auth     = otelcol.auth.basic.grafana_cloud.handler
    }
}

otelcol.auth.basic "grafana_cloud" {
  username = "<username>"
  password = "<api-token>"
}

Create a namespace and then a configmap with this config.

kubectl create ns app-o11y-demo

kubectl create configmap \
    alloy-config \
    --namespace app-o11y-demo \
    "--from-file=config.alloy=config.alloy"

Alloy chart

We can define the helm values for alloy by referring to the configmap as follows.

$ cat alloy-values.yaml 
alloy:
  configMap:
    create: false
    name: alloy-config
    key: config.alloy
  extraPorts:
  - name: "grpc"
    port: 4317
    targetPort: 4317
    protocol: "TCP"

And call this values file in the helmfile.

$ cat helmfile.yaml
repositories:
- name: grafana
  url: https://grafana.github.io/helm-charts

releases:
- name: alloy
  chart: grafana/alloy
  namespace: app-o11y-demo
  values:
  - alloy-values.yaml
  version: 0.5.0

We can now install the helm chart with helmfile.

$ helmfile sync

The alloy pod should be running.

$ kubectl get po -n app-o11y-demo
NAME                                  READY   STATUS    RESTARTS   AGE
alloy-cd4zp                           2/2     Running   0          6m10s

And the service, endpoints can be validated.

$ kubectl get svc -n app-o11y-demo
NAME    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)              AGE
alloy   ClusterIP   10.100.121.96   <none>        12345/TCP,4317/TCP   6m55s

$ kubectl get ep -n app-o11y-demo
NAME    ENDPOINTS                          AGE
alloy   10.1.4.159:12345,10.1.4.159:4317   7m11s

$ kubectl get po -n app-o11y-demo -o wide
NAME                                  READY   STATUS    RESTARTS   AGE     IP           NODE             NOMINATED NODE   READINESS GATES
alloy-cd4zp                           2/2     Running   0          7m27s   10.1.4.159   docker-desktop   <none>           <none>

Demo app

We can now deploy a demo app that would generate traces and send it alloy via grpc. The manifest for which is as follows.

$ cat xk6-client-tracing.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: xk6-client-tracing
  namespace: app-o11y-demo
spec:
  minReadySeconds: 10
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: xk6-client-tracing
      name: xk6-client-tracing
  template:
    metadata:
      labels:
        app: xk6-client-tracing
        name: xk6-client-tracing
    spec:
      containers:
      - env:
        - name: ENDPOINT
          value: alloy.app-o11y-demo.svc.cluster.local:4317
        image: ghcr.io/grafana/xk6-client-tracing:v0.0.2
        imagePullPolicy: IfNotPresent
        name: xk6-client-tracing

Note the ENDPOINT variable above is pointing to the alloy service that we deployed earlier. We can now deploy it with kubectl.

$ kubectl create -f xk6-client-tracing.yaml
deployment.apps/xk6-client-tracing created

The pod status can be checked.

$ kubectl get po -n app-o11y-demo | grep k6
xk6-client-tracing-6c49cdfbdb-cm25h   1/1     Running   0          5m44s

If everything works as expected, the demo app should have sent traces to alloy and then to tempo backend in Grafana cloud. Likewise it should sent the trace metrics to alloy and then to Mimir on cloud.

Grafana cloud

We can now headover to Grafana cloud. The url of which would look like https://<stack-name>.grafana.net/. You may go to Home > Application and click on Enable Metrics and activate it. After that add a new a connection and select OTLP. Just scroll down and enter the service name as shop-backend as that's the root service of our demo app. Test the connection now, it should be successful.

Once we click on Go to Application observability we should be able to see the pre built dashboards based on the traces and the trace metrics, predominantly based on the RED(Rate, Error, Duration method.

Note that in our case we are only sending traces from our application and the trace metrics were generated at Grafana cloud. We could also have sent direct metrics for ex. if it's a go application we could have sent go metrics, likewise we could also have send logs from our application, all of this to the same gateway(alloy on cloud). With the just the traces we have a handful of data to debug web app issues.

Also, though shop-backend is our root service, we also have other simulated microservices in this application.

There are a lot of other options in these built in dashboards to explore. Though we have these built in dashboards, we could also built any custom dashboards. For ex. let's explore what we get in the explore pane keeping tempo as the datasource and build a complete service map for all the services that are part of our demo app. Note that the service map from this is shown in the cover image of the blog.

Let's try one more for the trace metrics with mimir as the datasource.

Summary

Alright so we have come to the end of this post, we saw how one of Grafana cloud's solutions around app observability could help us get built in dashboards quite quickly once the data starts flowing to the cloud backend. However note that instrumentation is still necessary on the app side, we were able to get a demo app that already had the OTEL tracing instrumentation for us. As a cleanup check you may run the following commands:

helm uninstall alloy -n app-o11y-demo
kubectl delete -f xk6-client-tracing.yaml

# if you also want to delete the ns
kubectl delete ns app-o11y-demo

Thanks for reading!!!

Synthetic Monitoring with Grafana Cloud and Lambda

Shakir — Sat, 18 May 2024 08:48:03 +0000

🕐Time for some synthetic monitoring fun with Grafana...📊 We'd deploy a simple function on AWS Lambda that generates some data. We then expose this function with a public URL which would then be probed with a synthetic test from Grafana cloud servers.

AWS Lambda

Create a new AWS lambda function with runtime as nodejs and a name such as synthetic-monitoring-demo. Put the following code in index.js, which returns data similar to what's returned by this Grafana provided endpoint. We just create a lambda function here so that we could test as required and delete it when done.

const randomInt = (min, max) => Math.floor(Math.random() * (max - min + 1)) + min
const randomChoice = (arr) => arr[randomInt(0, arr.length - 1)]

const generateCrocodileData = () => {
    const names = [
        "Smiley", "Snappy", "Bitey", "Chompy", "Jaws", "Scaly", "Cruncher", 
        "Munch", "Gator", "Nibbles", "Swampy", "Toothless", "Grin", "Sly", 
        "Snapper", "Croc", "Dagger", "Ripjaw", "Spike", "Thrasher"
    ]
    const species = [
        "Saltwater Crocodile", "Nile Crocodile", "Mugger Crocodile", 
        "American Crocodile", "Freshwater Crocodile", "Morelet's Crocodile", 
        "Cuban Crocodile", "Siamese Crocodile", "Philippine Crocodile", 
        "Orinoco Crocodile", "Dwarf Crocodile", "Slender-snouted Crocodile"
    ]

    return {
        name: randomChoice(names),
        species: randomChoice(species),
        age: randomInt(1, 70),
        sex: randomChoice(["M", "F"]),
    }
}

export const handler = async () => {
    const numRecords = 8
    const crocodiles = Array.from({ length: numRecords }, generateCrocodileData)
    return {
        statusCode: 200,
        body: JSON.stringify(crocodiles),
    }
}

The crocodile species names were taken via generative AI, so not quite sure if it's correct😉.

Deploy the function, and then create a function URL in the configuration section with auth type as none, so that it could be accessed publicly🌍.

Try accessing the function URL.

K6

The synthetic monitoring in Grafana is based on k6. So we shall try some k6 basics on the local machine first.

Install k6.

brew install k6

The command k6 new test.js should create a file with name test.js with some references in it. However, for our case we can add a file test.js from scratch. Let's first import the functions/modules we need.

import { check } from 'k6'
import http from 'k6/http'

We just need to create a function that's exported by default in this code inside which we can try calling the api with http modules's get() function. Let's use the function URL we generated as endpoint for this purpose and try printing the status, response body as an array and length of the returned response array.

export default function main() {
  const res = http.get('https://ieaivrjwpcdbuxh4tws5autdoa0sflqi.lambda-url.us-east-1.on.aws/')
  console.log('The response status is', res.status)
  console.log('The response array is', res.json())
  console.log('The length of the response array is', res.json().length)
}

We could now this run this code with k6 and see the output as well as some test results.

$ k6 run test.js

          /\      |‾‾| /‾‾/   /‾‾/   
     /\  /  \     |  |/  /   /  /    
    /  \/    \    |     (   /   ‾‾\  
   /          \   |  |\  \ |  (‾)  | 
  / __________ \  |__| \__\ \_____/ .io

     execution: local
        script: test.js
        output: -

     scenarios: (100.00%) 1 scenario, 1 max VUs, 10m30s max duration (incl. graceful stop):
              * default: 1 iterations for each of 1 VUs (maxDuration: 10m0s, gracefulStop: 30s)

INFO[0001] The response status is 200                    source=console
INFO[0001] The response array is [{"name":"Chompy","species":"Mugger Crocodile","age":41,"sex":"M"},{"name":"Snappy","species":"Orinoco Crocodile","age":24,"sex":"M"},{"name":"Spike","species":"Mugger Crocodile","age":64,"sex":"M"},{"species":"Siamese Crocodile","age":40,"sex":"M","name":"Gator"},{"species":"Freshwater Crocodile","age":44,"sex":"F","name":"Snapper"},{"sex":"F","name":"Sly","species":"Slender-snouted Crocodile","age":11},{"age":24,"sex":"M","name":"Ripjaw","species":"Mugger Crocodile"},{"name":"Thrasher","species":"Orinoco Crocodile","age":67,"sex":"M"}]  source=console
INFO[0001] The length of the response array is 8         source=console

     data_received..................: 6.5 kB 7.3 kB/s
     data_sent......................: 542 B  606 B/s
     http_req_blocked...............: avg=533.74ms min=533.74ms med=533.74ms max=533.74ms p(90)=533.74ms p(95)=533.74ms
     http_req_connecting............: avg=246.67ms min=246.67ms med=246.67ms max=246.67ms p(90)=246.67ms p(95)=246.67ms
     http_req_duration..............: avg=356.86ms min=356.86ms med=356.86ms max=356.86ms p(90)=356.86ms p(95)=356.86ms
       { expected_response:true }...: avg=356.86ms min=356.86ms med=356.86ms max=356.86ms p(90)=356.86ms p(95)=356.86ms
     http_req_failed................: 0.00%  ✓ 0        ✗ 1
     http_req_receiving.............: avg=357µs    min=357µs    med=357µs    max=357µs    p(90)=357µs    p(95)=357µs   
     http_req_sending...............: avg=297µs    min=297µs    med=297µs    max=297µs    p(90)=297µs    p(95)=297µs   
     http_req_tls_handshaking.......: avg=285.27ms min=285.27ms med=285.27ms max=285.27ms p(90)=285.27ms p(95)=285.27ms
     http_req_waiting...............: avg=356.21ms min=356.21ms med=356.21ms max=356.21ms p(90)=356.21ms p(95)=356.21ms
     http_reqs......................: 1      1.117563/s
     iteration_duration.............: avg=894.17ms min=894.17ms med=894.17ms max=894.17ms p(90)=894.17ms p(95)=894.17ms
     iterations.....................: 1      1.117563/s


running (00m00.9s), 0/1 VUs, 1 complete and 0 interrupted iterations
default ✓ [======================================] 1 VUs  00m00.9s/10m0s  1/1 iters, 1 per VU

So based on the output above, the api is returning 8 crocodiles with it's name, gender and other details. The test results give us some metrics such as data recd., sent and other http metrics. Note that we haven't added any checks so far, though we have imported the check function.

For this usecase let's do a few checks such as:

Check the response status to be 200
Is it sending back 8 crocodile details
Is it giving back the details of 4 male and 4 crocodiles, meaning is the response distrubuted equally based on gender
There should be atleast one crocodile under 15 years of age

The code for these checks should look like below:

  check(res, {
    'is status 200': (r) => r.status === 200,
    'is response length 8': (r) => r.json().length === 8,
    'is response distributed equally based on gender': (r) => {
      let maleCount = 0
      let femaleCount = 0
      r.json().forEach(crocodile => {
        if (crocodile.sex === 'M') {
          maleCount++
        } else if (crocodile.sex === 'F') {
          femaleCount++
        }
      })
      return (maleCount === femaleCount)
    },
    'is any crocodile under 15 years of age': (r) => {
      const isAnyUnder15 = r.json().some(crocodile => crocodile.age < 15)
      return isAnyUnder15
    }
  })

Please keep the checks block inside the main function after which we can run the test again.

% k6 run test.js
---TRUNCATED---
     ✓ is status 200
     ✓ is response length 8
     ✗ is response distributed equally based on gender
      ↳  0% — ✓ 0 / ✗ 1
     ✗ is any crocodile under 15 years of age
      ↳  0% — ✓ 0 / ✗ 1
---TRUNCATED---

So, couple of our checks are failing.

All good👍, so we got some grip on how to run k6 and add some checks. We can try doing the same stuff on grafana.

Grafana

Go to Grafana Cloud > Home > Testing & synthetics > Add a new check > Scripted check. I have defined the following options in the form.

Job name: test-job
Instance: test-instance
Probe locations: Seoul(I have chosen only one)

In the script from, just put the same script we used before, and test/save it.

Note that by default the logs and metrics related to our test would go to the loki and mimir endpoints of Grafana cloud.

The out of box dashboard for our check should appear in sometime. The assertions panel which gives a stat and timeseries graph(expanded view) for the four checks.

As a bonus let's go to Home > Dashbooards > New Dashboard and try creating a custom dashboard panel with mimir(prometheus type datasource).

Here is the panel json for this.

{
  "datasource": {
    "uid": "grafanacloud-prom",
    "type": "prometheus"
  },
  "type": "stat",
  "title": "Current probe checks status",
  "gridPos": {
    "x": 0,
    "y": 8,
    "w": 12,
    "h": 8
  },
  "id": 1,
  "targets": [
    {
      "datasource": {
        "type": "prometheus",
        "uid": "grafanacloud-prom"
      },
      "refId": "A",
      "expr": "sum(probe_checks_total{job=\"test-job\"}) by(result)",
      "range": false,
      "instant": true,
      "editorMode": "code",
      "legendFormat": "__auto",
      "useBackend": false,
      "disableTextWrap": false,
      "fullMetaSearch": false,
      "includeNullMetadata": true,
      "format": "time_series",
      "exemplar": false
    }
  ],
  "options": {
    "reduceOptions": {
      "values": true,
      "calcs": [
        "lastNotNull"
      ],
      "fields": ""
    },
    "orientation": "auto",
    "textMode": "auto",
    "wideLayout": true,
    "colorMode": "value",
    "graphMode": "area",
    "justifyMode": "auto",
    "showPercentChange": false
  },
  "transformations": [
    {
      "id": "calculateField",
      "options": {
        "mode": "binary",
        "reduce": {
          "reducer": "sum"
        },
        "binary": {
          "left": "fail",
          "right": "pass"
        },
        "alias": "total"
      }
    },
    {
      "id": "calculateField",
      "options": {
        "mode": "binary",
        "binary": {
          "left": "pass",
          "operator": "/",
          "right": "total"
        },
        "alias": "success ratio"
      }
    }
  ],
  "fieldConfig": {
    "defaults": {
      "mappings": [],
      "thresholds": {
        "mode": "absolute",
        "steps": [
          {
            "value": null,
            "color": "green"
          },
          {
            "value": 80,
            "color": "red"
          }
        ]
      },
      "color": {
        "mode": "thresholds"
      }
    },
    "overrides": [
      {
        "matcher": {
          "id": "byName",
          "options": "fail"
        },
        "properties": [
          {
            "id": "thresholds",
            "value": {
              "mode": "absolute",
              "steps": [
                {
                  "color": "green",
                  "value": null
                },
                {
                  "color": "red",
                  "value": 1
                }
              ]
            }
          }
        ]
      },
      {
        "matcher": {
          "id": "byName",
          "options": "success ratio"
        },
        "properties": [
          {
            "id": "unit",
            "value": "percentunit"
          },
          {
            "id": "thresholds",
            "value": {
              "mode": "absolute",
              "steps": [
                {
                  "color": "red",
                  "value": null
                },
                {
                  "color": "yellow",
                  "value": 0.8
                },
                {
                  "value": 0.9,
                  "color": "green"
                }
              ]
            }
          }
        ]
      },
      {
        "matcher": {
          "id": "byName",
          "options": "total"
        },
        "properties": [
          {
            "id": "color",
            "value": {
              "mode": "shades",
              "fixedColor": "purple"
            }
          }
        ]
      }
    ]
  },
  "pluginVersion": "11.1.0-70724"
}

And one with loki for logs.

Cool, we came to the end of this post👏, here we tried synthetic testing first with k6 and used the same script with Grafana cloud and visualized the data with couple of panels. As a housekeeping reminder, you may delete the check that's created in Grafana, so that it stops polling the endpoint, and then may delete the function in AWS Lambda. Thanks for reading!!!

Logging demo with OTEL Collector, CloudWatch and Grafana

Shakir — Sat, 11 May 2024 14:18:27 +0000

Hello 👋, in this post, we would be using the OTEL Demo App with OpenTelemetry Collector and export the logs from the microservices in the app to AWS CloudWatch.

Let's get started!!! We can create a separate namespace in the kubernetes cluster with kubectl create ns otelcol-logs-demo.

And then create a secret that holds the AWS credentials.

kubectl create secret generic aws-credentials \
    --from-literal=AWS_ACCESS_KEY_ID=<access-key-id> \
    --from-literal=AWS_SECRET_ACCESS_KEY=<aws-secret-access-key> \
    -n otelcol-logs-demo

We would be deploying open telemetry demo app, otel collector as well as Grafana with a single helm chart. The chart comes with other components too, we could disable those that we do not need for this lab. Our helm values would like below.

$ cat values.yaml 
opentelemetry-collector:
  config:
    exporters:
      awscloudwatchlogs:
        log_group_name: otel-logs-demo
        log_stream_name: otel-demo-app
        region: ap-south-2
    service:
      pipelines:
        logs:
          exporters:
            - awscloudwatchlogs
        metrics: {}
        traces: {}
  extraEnvsFrom:
    - secretRef:
        name: aws-credentials
prometheus:
  enabled: false
jaeger:
  enabled: false
opensearch:
  enabled: false

In the values above, we have the cloudwatchlogs exporter section where we specify the region, log group and stream names. We are only setting up logs in the pipeline as we are only dealing with that for this demo. We are then setting the AWS credentials via env vars so that the collector could authenticate and send logs to cloud watch. Then, finally we are disabling certain applications which we do not need for this exercise.

Let's install the chart.

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts

helm install my-otel-demo open-telemetry/opentelemetry-demo -n otelcol-logs-demo -f values.yaml

Head over to CloudWatch logs in the mentioned region and you should be able to see the log group. And, inside the log group we should have the stream.

We should be able to view the logs, for ex. from cartservice as shown in the screenshot below.

We can now go to Grafana cloud and add the cloud watch datasource with access credentials.

And should now be able to see the logs in Grafana for our log group, on the explore tab.

As a bonus let's try adding a panel that shows the no. of messages per severity.

Add a panel with query as shown in the screenshot below.

Go to the transform data tab and apply a transformation to parse the severity_text from the json message.

And then do a group by transformation to calculate the no. of messages per severity.

Can choose a visualization type such as pie chart like below.

The options I have set in the panel are as follows:

Title: Logs by severity
Value options > Show > All values
Legend > Legend values: Value, Percent

Ok that's it for this post, we saw how to get logs from the otel demo app via open collector to AWS CloudWatch Logs and explored it on Grafana with a sample visualization. We just need to delete the namespace with kubectl delete ns otelcol-logs-demo so that logs are not sent to AWS cloudwatch. Thanks for reading !!!

OTEL Tracing demo with ADOT, AWS X-Ray and Grafana

Shakir — Fri, 26 Apr 2024 10:01:17 +0000

Introduction

Hello 👋, In this post we'll see about sending traces from a demo app to AWS X-Ray via the ADOT(AWS Distro for OpenTelemetry) collector. We would then visualize this on Grafana. Note that we'd deploy the workloads on a kubernetes cluster.

Here is a picture of what we are trying to accomplish:

Alright, let's get started!!!

Namespace

We shall deploy the workloads on a separate namespace. Let's create one.

kubectl create ns adot-traces-demo

Credentials

Store the AWS credentials as a kubernetes secret.

kubectl create secret generic aws-credentials \
    --from-literal=AWS_ACCESS_KEY_ID=<access-key-id> \
    --from-literal=AWS_SECRET_ACCESS_KEY=<aws-secret-access-key> \
    -n adot-traces-demo

ADOT Config

Set the ADOT config in a file.

$ cat adot-config.yaml
exporters:
  awsxray:
    region: ap-south-2
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
service:
  pipelines:
    traces:
      exporters:
        - awsxray
      receivers:
        - otlp

And create a config map with this file.

kubectl create configmap adot-config --from-file=adot-config.yaml -n adot-traces-demo

ADOT Deployment

Setup deployment spec in a file, that injects the secret we created earlier as environment variables and the configmap as a volume.

$ cat adot-deploy.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: adot-collector
  name: adot-collector
spec:
  replicas: 1
  selector:
    matchLabels:
      app: adot-collector
  template:
    metadata:
      labels:
        app: adot-collector
    spec:
      containers:
        - args:
            - '--config=/etc/adot-config.yaml'
          envFrom:
            - secretRef:
                name: aws-credentials
          image: public.ecr.aws/aws-observability/aws-otel-collector:latest
          name: adot-collector
          volumeMounts:
            - mountPath: /etc/adot-config.yaml
              name: config-volume
              subPath: adot-config.yaml
      volumes:
        - configMap:
            name: adot-config
          name: config-volume

Create the deployment.

kubectl create -f adot-deploy.yaml -n adot-traces-demo

The pod in the deployment should be running.

$ kubectl get po -n adot-traces-demo
NAME                              READY   STATUS    RESTARTS   AGE
adot-collector-7cbf849b89-b4bkl   1/1     Running   0          3m26s

ADOT Service

We can expose the ADOT deployment with a service spec that exposes the grpc port 4317 as follows.

$ cat adot-svc.yaml 
apiVersion: v1
kind: Service
metadata:
  name: adot-collector-service
spec:
  selector:
    app: adot-collector
  ports:
    - protocol: TCP
      port: 4317
      targetPort: 4317

We can now create the service.

kubectl create -f adot-svc.yaml -n adot-traces-demo

The endpoint IP should match with the pod IP.

$ kubectl get ep -n adot-traces-demo
NAME                     ENDPOINTS         AGE
adot-collector-service   10.1.3.187:4317   22s

$ kubectl get po -n adot-traces-demo -o wide
NAME                              READY   STATUS    RESTARTS   AGE     IP           NODE             NOMINATED NODE   READINESS GATES
adot-collector-7cbf849b89-b4bkl   1/1     Running   0          7m11s   10.1.3.187   docker-desktop   <none>           <none>

Demo app

We can now deploy the sample demo app which can send traces to ADOT collector, with the following manifest.

$ cat k6-tracing-deploy.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: xk6-tracing
spec:
  replicas: 1
  selector:
    matchLabels:
      app: xk6-tracing
  template:
    metadata:
      labels:
        app: xk6-tracing
    spec:
      containers:
        - env:
            - name: ENDPOINT
              value: adot-collector-service:4317
          image: ghcr.io/grafana/xk6-client-tracing:v0.0.2
          name: xk6-tracing

Let's create the deployment.

kubectl create -f k6-tracing-deploy.yaml -n adot-traces-demo

Both the ADOT collector and k6-tracing pods should now be running.

$ kubectl get po -n adot-traces-demo   
NAME                              READY   STATUS    RESTARTS   AGE
adot-collector-7cbf849b89-b4bkl   1/1     Running   0          14m
xk6-tracing-69b48fcfd9-bjzbd      1/1     Running   0          24s

X-Ray

We can now headover to AWS X-Ray, in ap-south-2 region that we mentioned in the adot-config.

The nodes(services) shown in the screenshot belong to our demo application. We could filter for traces that passes through a particular service name for ex. article service, like below.

If we click on a single trace we should be able to see a complete service map for that trace, that shows all the services that trace traverses.

If we go a further down on this we should be able to see the details for segments/spans with in this trace.

Grafana

So we far we were able to see the traces in AWS X-Ray, we can do a similar exercise on Grafana. I am using a Grafana Cloud Free subscription for this lab.

Go to Connections, Add a new connection and search for X-Ray and install it.

You can then go to datasources, add a new X-Ray datasource with the access key id, secret access key, and default region(I have chosen ap-south-2 which matches with adot config).

All good, we can try adding a new panel, go to dashboards > new dashboard and a new visualization with table as panel type and a sample query for ex. service(id(name: "article-service" ))

We can click on one of the traces we should take us to the explore view where we can see the node graph(service map)

We should also see the trace explorer that shows the individual spans.

Okay so we reached this far, that was some fun exploring traces on AWS and Grafana with Open Telemetry. Thank you for reading !!!

CleanUp

Just delete the namespace with kubectl delete ns adot-traces-demo and it should remove the workloads we deployed from kubernetes and stop sending any new data to the cloud.