DEV Community: Austin Cunningham

Building an AI-Powered Log Analyser with RAG

Austin Cunningham — Fri, 14 Nov 2025 11:16:21 +0000

I was looking to build something useful with AI development. I frequently use AI to analyze individual log files, so I thought it would be valuable to create a log analyzer that can process entire directories of log files to identify issues and provide an overall picture of system problems.

Traditional approaches fail:

grep/awk: Too rigid, misses context
Manual reading: Impossible with large files or many individual files
Basic search: No understanding of log semantics

I will build a Retrieval-Augmented Generation (RAG) system that transforms your log files into an intelligent, queryable knowledge base.

Prerequisites

# Install Ollama (for local LLM)
curl -fsSL https://ollama.ai/install.sh | sh

# Pull the Mistral model
ollama pull mistral

Project Setup

# Create project directory
mkdir rag_log_analyser
cd rag_log_analyser

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install langchain langchain-community langchain-ollama langchain-huggingface chromadb sentence-transformers

Core Architecture

📁 rag_log_analyser/
├── main.py              # Main application
├── utils/
│   └── loaders.py       # Custom log file loader
└── requirements.txt     # Dependencies

Custom Log Loader (`utils/loaders.py`)

The idea here is treating each log line as an individual document. Let's break down exactly what each part does:

Set the Type of Files to Find

def load_sop_files(directory: str):
    allowed_exts = ('.log', '.txt', '.out', '.err', '.access', '.csv', '.json', '.yaml', '.yml', '.md', '.asciidoc')

    docs = []
    file_count = 0
    total_lines = 0

Defines supported file extensions (covers most log formats)
Initializes tracking variables for progress reporting
Creates empty list to store document objects

Processing Files Line by Line

for root, _, files in os.walk(directory):
    for file in files:
        file_lower = file.lower()
        if file_lower.endswith(allowed_exts):
            path = os.path.join(root, file)
            file_count += 1
            print(f" Processing file {file_count}: {file}")

            try:
                with open(path, 'r', encoding='utf-8') as f:
                    line_count = 0
                    for i, line in enumerate(f):
                        if line.strip():  # Skip empty lines
                            docs.append({
                                "page_content": line.strip(),
                                "metadata": {"source": path, "line_number": i + 1}
                            })
                            line_count += 1
                            total_lines += 1

                            # Show progress for large files
                            if line_count % 10000 == 0:
                                print(f"   Processed {line_count:,} lines...")

                    print(f"   Completed: {line_count:,} lines processed")
            except Exception as e:
                print(f" Error loading {path}: {e}")

print(f" Summary: Processed {file_count} files, {total_lines:,} total log entries")
return [Document(**d) for d in docs]

This function does several important things:

Finds all log files: The os.walk(directory) function recursively searches through directories, while file_lower.endswith(allowed_exts) filters files by extension using case-insensitive matching
Processes each line separately: The for i, line in enumerate(f) loop treats every non-empty line as its own document using if line.strip()
Tracks metadata: Each line gets stored with {"source": path, "line_number": i + 1} metadata, making it easy to trace back to the original file location
Shows progress: The if line_count % 10000 == 0 condition displays progress every 10,000 lines with print(f"📊 Processed {line_count:,} lines...") so you know it's working on large files
Handles errors gracefully: The try/except block catches file reading errors with except Exception as e and logs them while continuing with other files
Returns LangChain documents: The final return [Document(**d) for d in docs] converts our dictionary format into LangChain Document objects needed for embedding and vector storage

Main Application (`main.py`)

Let's break down the main application into logical sections:

Imports & Dependencies

The imports

import warnings
warnings.filterwarnings("ignore", category=DeprecationWarning)

from utils.loaders import load_sop_files
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.vectorstores import Chroma
from langchain.chains import RetrievalQA
from langchain_ollama import OllamaLLM

Directory Input & Validation

try:
    LOG_DIRECTORY = input("-> Please enter the full path to the log directory: ")
    if not os.path.isdir(LOG_DIRECTORY):
        print(f" Error: Directory not found at '{LOG_DIRECTORY}'")
        sys.exit(1)
except KeyboardInterrupt:
    print("\n\n Exiting...")
    sys.exit(0)

Prompts user for log directory path
Validates directory exists before processing
Keyboard interrupt handling: Graceful exit with Ctrl+C
User-friendly error messages

Document Loading & Processing

try:
    print(f" Loading log files from: {LOG_DIRECTORY}...")
    docs = load_sop_files(LOG_DIRECTORY)
    print(f" Loaded {len(docs)} log entries from files")
except KeyboardInterrupt:
    print("\n\n File loading interrupted by user. Exiting ...")
    sys.exit(0)

Calls our custom loader to process all log files
Shows progress during file processing
Reports total number of log entries found

Vector Database Creation (The Heavy Lifting)

print(f" Creating vector database with {len(chunks)} log entries...")
print(" This may take several minutes for large files...")

# Check if database already exists
db_path = "./chroma_db"
db_exists = os.path.exists(db_path)
if db_exists:
    rebuild_choice = input(" Vector database already exists. Rebuild? (y/n): ")

# Optimized embedding settings
embeddings = HuggingFaceEmbeddings(
    model_name="sentence-transformers/all-MiniLM-L6-v2",
    model_kwargs={'device': 'cpu'},
    encode_kwargs={'normalize_embeddings': True, 'batch_size': 32}
)

if db_exists and rebuild_choice != 'y':
    print(" Found existing vector database, loading...")
    db = Chroma(persist_directory=db_path, embedding_function=embeddings)
    print(" Vector database loaded successfully")
else:
    print(" Creating new vector database...")
    db = Chroma.from_documents(
        chunks, embeddings,
        persist_directory=db_path,
        collection_metadata={"hnsw:space": "cosine"}
    )
    print(" Vector database created and saved")

Persistent Storage: Database saved to ./chroma_db/ for reuse
Optimized Embeddings: Fast model with batch processing (32 items/batch)
Smart Loading: Reuses existing database unless rebuild requested
This is the bottleneck: Large files take time on first run only

RAG Chain Setup

retriever = db.as_retriever()
llm = OllamaLLM(model="mistral")
qa = RetrievalQA.from_chain_type(llm=llm, retriever=retriever, return_source_documents=True)

Retriever: Finds most relevant log entries for each query
LLM: Mistral model for generating intelligent responses
RAG Chain: Combines retrieval + generation for contextual answers
Source Documents: Returns original log entries for verification

Interactive Query Loop & Prompt Engineering

Loop for handling queries and prompt engineering

try:
    while True:
        query = input("\n You (e.g., 'What errors occurred in the last hour?'): ")
        if query.lower() in ("exit", "quit"):
            print(" Bye! Take care.")
            break

        # LLM prompt engineering This is what give the LLM it's context to formulate it's response
        analysis_query = (
          "You are an expert log analyser. Review the provided log entries and answer the user's question. "
          "Provide a concise summary, highlight any potential issues, and mention the relevant source log files. "
          f"Question: {query}"
        )

        try:
            result = qa.invoke({"query": analysis_query})
            # ... display results ...
        except KeyboardInterrupt:
            print("\n Query processing interrupted. You can ask another question.")
            continue
except KeyboardInterrupt:
    print("\n\n Goodbye! Exiting...")
    sys.exit(0)

Response Display & Source Attribution

print("\n Assistant:\n", result["result"])

print("\n Sources:")
for doc in result["source_documents"]:
    source = doc.metadata.get('source')
    line_number = doc.metadata.get('line_number')
    print(f" - {source}{f' (Line {line_number})' if line_number else ''}")

Displays AI-generated analysis
Shows source files and line numbers for verification
Enables traceability back to original log entries

How It Works

You start Ollama

ollama run mistral

In a separate window start the log analyser and

python main.py
Starting script...
_______________________________________________________________________________
 _ __ __ _  __ _    | | ___   __ _      __ _ _ __   __ _| |_   _ ___  ___ _ __
| '__/ _` |/ _` |___| |/ _ \ / _` |___ / _` | '_ \ / _` | | | | / __|/ _ | '__|
| | | (_| | (_| |___| | (_) | (_| |___| (_| | | | | (_| | | |_| \__ |  __| |
|_|  \__,_|\__, |   |_|\___/ \__, |    \__,_|_| |_|\__,_|_|\__, |___/\___|_|
           |___/              |___/                         |___/
_______________________________________________________________________________



-> Please enter the full path to the log directory (e.g., /home/user/support/):

Pass in the full directory to the log files you wish to query

Step 1: Document Processing

The log file will get processed output will look similar to below

Processing file 1: application.log
   Processed 10,000 lines...
   Processed 20,000 lines...
   Completed: 45,123 lines processed
Summary: Processed 1 files, 45,123 total log entries

What happens:

Each log line becomes a separate document
Metadata tracks source file and line number
Progress tracking for large files

Step 2: Vector Database Creation

Next the database is created.

Creating vector database with 45,123 log entries...
This may take several minutes for large files...
Vector database already exists. Rebuild? (y/n): y
Using optimized embedding model for faster processing...
Generating embeddings...
Vector database created successfully

What happens:

Each log line gets converted into a unique numerical "fingerprint" (384 numbers)
Vectors stored in ChromaDB for fast retrieval
This can take a while

NOTE: if you have built the vector DB already you have an opt out here so you can use an existing DB rather than rebuilding each time.

Once the vector database is successfully built, you can query the LLM to get answers based on the loaded log files.

Vector database loaded successfully
Log Analyser Assistant ready. Type your question below. Type 'exit' to quit.

You (e.g., 'What errors occurred in the last hour?'):

Sample response

Log Analyser Assistant ready. Type your question below.

You: What errors occurred in the last hour?

Assistant:
Based on the log entries, I found several errors in the last hour:

1. **Database Connection Error** (2 occurrences)
   - Time: 14:23:15, 14:45:22
   - Error: "Connection timeout to database server"
   - Severity: CRITICAL

2. **Authentication Failure** (5 occurrences)
   - Time: 14:12:33, 14:18:45, 14:25:12, 14:31:56, 14:42:18
   - Error: "Invalid credentials for user admin"
   - Severity: WARNING

Sources:
 - /var/log/application.log (Line 1247)
 - /var/log/application.log (Line 1253)
 - /var/log/application.log (Line 1289)

Code lives here https://github.com/austincunningham/rag_log_analyser

Build an Ollama LLM software engineering language bot

Austin Cunningham — Wed, 24 Sep 2025 14:55:45 +0000

What

Another step in AI learning I decided to see if I could build a AI software engineering language bot with an SRE context. We are going to use Ollama, a locally hosted LLM, all thought you could use any LLM chat endpoint by adjusting the backend.

Prerequisites

ollama
python3
pip

Install and run ollama

curl -fsSL https://ollama.com/install.sh | sh
ollama run llama3

Scaffold

Create the following layout

sre-language-bot/
├── static/
│   ├── css/
│   │   └── style.css
│   └── js/
│       └── main.js
├── templates/
│   └── index.html
└── app.py

The ccs and js can remain empty for now

Backend

We setup a basic flask app in app.py

import os
import requests
import json
from flask import Flask, request, jsonify, render_template

# Initialize Flask
app = Flask(__name__)

# Home route to serve the HTML UI from the templates folder
@app.route('/')
def index():
    # load the html on the default route
    return render_template('index.html')


if __name__ == '__main__':
    # Flask will run on localhost:5000 by default.
    # The debug=True flag automatically reloads the server on code changes.
    app.run(debug=True)

And add something to index.html

<!DOCTYPE html>
<html lang="en">
<head>
</head>
<body>
    <h1>Chat bot goes here</h1>
</body>
</html>

Run the python app.py to see if it works

python app.py
 * Serving Flask app 'app'
 * Debug mode: on
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:5000
Press CTRL+C to quit
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 661-293-891
127.0.0.1 - - [08/Aug/2025 12:42:06] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [08/Aug/2025 12:42:06] "GET /favicon.ico HTTP/1.1" 404 -

Add the chat bot post request

In app.py we add the post request, basically create a /chat endpoint and create a prompt (the important bit for giving the llm context on which to base your answers) for the llm which favours software engineering answers in payload.messages.content

@app.route('/chat', methods=['POST'])
def chat():
    # get the chat history
    data = request.json
    chat_history = data.get('messages', [])

    if not chat_history:
        return jsonify({'error': 'No message history provided'}), 400

    try:
        ollama_model = "llama3"
        ollama_api_url = "http://localhost:11434/api/chat"

        # The payload for the Ollama API request, including a specialized system prompt
        payload = {
            "model": ollama_model,
            "stream": False,  # We want a single, complete response
            "messages": [
                            {
                                "role": "system",
                                "content": "You are a helpful and knowledgeable assistant specializing in **Golang, Python,Bash and JavaScript** programming and **Site Reliability Engineering (SRE)** with a focus on **Kubernetes**. Provide detailed, accurate, and professional answers to technical questions. Always respond in markdown format."
                            }
                        ] + chat_history  # Append the user's messages after the system prompt
        }

        # Send the request to the Ollama API
        response = requests.post(ollama_api_url, json=payload, timeout=120)
        response.raise_for_status()  # Raise an error for bad status codes

        # Parse the JSON response from Ollama
        ollama_data = response.json()
        # Extract the content from the model's message
        model_response = ollama_data['message']['content']

        return jsonify({'response': model_response})
    # Add some error handling
    except requests.exceptions.RequestException as e:
        print(f"Error calling Ollama API: {e}")
        return jsonify(
            {'error': 'Failed to connect to Ollama server. Please ensure it is running and accessible.'}), 500
    except Exception as e:
        print(f"An unexpected error occurred: {e}")
        return jsonify({'error': 'An internal server error occurred.'}), 500

Now test its working with a curl to the /chat endpoint

curl -X POST -H "Content-Type: application/json" -d '{"messages": [{"role": "user", "content": "how are you."}]}' http://127.0.0.1:5000/chat

{
  "response": "**I'm doing well, thank you for asking!** I'm a knowledgeable assistant specializing in Golang, Python, Bash, and JavaScript programming, as well as Site Reliability Engineering (SRE) with a focus on Kubernetes. I'm here to help answer any technical questions you may have, so feel free to ask me anything! **Let's get started!**"
}

Ok that's pretty much the backend created

Frontend

Using semantic ui for a bit of style to create a chatbox, don't think I need the static/ccs/style.ccs at this point as
semantic ui will handle the style.

<div id="messages" class="ui segment inverted chat-container">
   <div class="ui center aligned basic inverted segment">
      <div class="sub header">Start the conversation! What would you like to know?</div>
   </div>
</div>

<div class="ui action fluid input inverted">
   <input type="text" id="user-input" placeholder="Ask a question..." autocomplete="off">
   <button class="ui button inverted green" id="send-btn">
      <i class="paper plane outline icon"></i>
   </button>
</div>

And this gives us this basic ui with a chat box and send button

And call the script that handles the call to the llm and formats and returns the response.

    <script src="{{ url_for('static', filename='js/main.js') }}"></script>

To control the user interface and handle the interaction with our backend, we need to add some JavaScript.
The following code goes into static/js/main.js.

First, we set up our event listeners and variables, ensuring the script only runs after the entire HTML document has been loaded.

// Waiting for a send message and button click in the ui
document.addEventListener('DOMContentLoaded', (event) => {
    // Get references to the HTML elements we will interact with
    const chatBox = document.getElementById('messages');
    const inputField = document.getElementById('user-input');
    const sendBtn = document.getElementById('send-btn');
    // An array to store the conversation history for context
    let chatHistory = [];

This initial block of code finds and stores references to our key UI elements (messages, user-input, and send-btn) and initializes an empty array, chatHistory, which will for maintaining conversation context.

Next, we define a helper function to format and display messages in the chat interface.

const appendMessage = (message, sender) => {
    // Create a new div element for the message
    const messageDiv = document.createElement('div');
    messageDiv.classList.add('ui', 'message', 'segment');

    if (sender === 'user') {
        // Style the message differently if it's from the user
        messageDiv.classList.add('user-message', 'right', 'aligned');
        messageDiv.innerHTML = `<p>${message}</p>`;
    } else {
        // Style the message for the model's response
        messageDiv.classList.add('model-message', 'left', 'aligned');
        // Use marked.js to convert markdown from the model's response to HTML
        messageDiv.innerHTML = marked.parse(message);
    }

    // Add the message to the chat container
    chatBox.appendChild(messageDiv);
    // Automatically scroll to the bottom to show the new message
    chatBox.scrollTop = chatBox.scrollHeight;

    // After the model's message is added, highlight any code blocks
    if (sender === 'model') {
        hljs.highlightAll();
    }
};

The appendMessage function dynamically creates a new HTML div element, assigns appropriate CSS classes based on the sender (user or model), and inserts the message content. For the model's response, it uses the marked.js library to parse Markdown into rich HTML, which allows for formatted text, lists, and code blocks.
Finally, hljs.highlightAll() is called to apply syntax highlighting to any code blocks present in the message.

Now, we define the main logic for sending a message, which includes calling the backend and handling the response.

const sendMessage = async () => {
    const message = inputField.value.trim();
    if (message === '') return; // Don't send empty messages

    // Clear the initial "Start the conversation" message
    if (chatBox.querySelector('p')) {
        chatBox.innerHTML = '';
    }

    // Display the user's message in the chat
    appendMessage(message, 'user');
    inputField.value = ''; // Clear the input field
    sendBtn.disabled = true;
    inputField.disabled = true;

    // Add the user message to our conversation history
    chatHistory.push({ role: 'user', content: message });

    // Add a "Typing..." loading indicator
    const loadingIndicator = document.createElement('div');
    loadingIndicator.classList.add('ui', 'message', 'segment', 'loading', 'left', 'aligned');
    loadingIndicator.innerHTML = `<p>Typing...</p>`;
    chatBox.appendChild(loadingIndicator);
    chatBox.scrollTop = chatBox.scrollHeight;

    try {
        // Make an asynchronous POST request to our Flask backend
        const response = await fetch('/chat', {
            method: 'POST',
            headers: { 'Content-Type': 'application/json' },
            body: JSON.stringify({ messages: chatHistory }) // Send the full chat history
        });

        if (response.ok) {
            const data = await response.json();
            loadingIndicator.remove(); // Remove the loading indicator
            appendMessage(data.response, 'model'); // Display the model's response
            // Add the model's response to the conversation history
            chatHistory.push({ role: 'assistant', content: data.response });
        } else {
            loadingIndicator.remove();
            appendMessage('Sorry, something went wrong. Please try again.', 'model');
            console.error('Server error:', response.statusText);
        }
    } catch (error) {
        // Handle any network or connection errors
        loadingIndicator.remove();
        appendMessage('Failed to connect to the server. Please check your connection.', 'model');
        console.error('Fetch error:', error);
    } finally {
        // Re-enable the input field and button, regardless of the outcome
        sendBtn.disabled = false;
        inputField.disabled = false;
        inputField.focus();
    }
};

The sendMessage function is the core of our frontend logic. It first retrieves the user's input, validates it, and disables the input and button to prevent multiple submissions. It then adds the user's message to chatHistory and displays it.

A loadingIndicator is added to provide user feedback while the request is being processed. The fetch call then sends the entire chatHistory array to our /chat endpoint. Once a response is received, the loading indicator is removed, the model's response is displayed using appendMessage, and it's also added to chatHistory.

The try...catch...finally block is for robust error handling and to ensure that the input field and button are re-enabled, regardless of whether the request succeeded or failed.

Finally, we attach the sendMessage function to the UI elements that trigger it.

// Add event listeners to trigger the sendMessage function
sendBtn.addEventListener('click', sendMessage);
inputField.addEventListener('keypress', (e) => {
    if (e.key === 'Enter') {
        sendMessage();
    }
});

This final block adds two event listeners. The first one listens for a click event on the sendBtn (the paper plane icon), and the second listens for a keypress event on the inputField. If the key pressed is "Enter," it also calls the sendMessage function.

The message is formatted and appended: The appendMessage function formats the Markdown response into HTML and adds it to the chat container. highlight.js then applies syntax highlighting to any code blocks.

To break down and add it all together

A message comes in: A user types a message and sends it by either clicking the sendBtn or pressing "Enter" in the inputField. This triggers the sendMessage function via an addEventListener.
It's sent to the API endpoint: Inside the sendMessage function, a fetch call is made to the /chat endpoint. The body of the request is JSON.stringify({ messages: chatHistory }), which includes the new message and the entire conversation history.
It waits: The code uses the await keyword with the fetch call, which pauses the execution of the sendMessage function until a response is received from the server.
It responds: The server sends back a message, and the response object is received by the fetch call. The code then
uses await response.json() to parse the data.
The message is formatted and appended to a div: The appendMessage function is called with the server's response.
Inside this function:
1. document.createElement('div') creates a new div.
2. marked.parse(message) formats the message's Markdown into HTML.
3. chatBox.appendChild(messageDiv) appends the newly created and formatted div to the main chat container.
4. hljs.highlightAll() applies syntax highlighting to any code blocks within the new message.

Start the app to test

Start the app with python app.py

python app.py
 * Serving Flask app 'app'
 * Debug mode: on
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:5000
Press CTRL+C to quit
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 661-293-891
127.0.0.1 - - [11/Aug/2025 10:48:18] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [11/Aug/2025 10:48:18] "GET /static/css/style.css HTTP/1.1" 404 -
127.0.0.1 - - [11/Aug/2025 10:48:18] "GET /static/js/main.js HTTP/1.1" 304 -
127.0.0.1 - - [11/Aug/2025 10:49:28] "POST /chat HTTP/1.1" 200 -

And in the browser on localhost:5000 we can a query the llm

Repo is here https://github.com/austincunningham/sre-language-bot

Deploying Ollama on Openshift with the Ollama Operator

Austin Cunningham — Thu, 31 Jul 2025 14:11:58 +0000

Why Run Ollama on OpenShift?

Running Ollama on OpenShift gives you a scalable, centralized service that's ready for team collaboration and production workloads.

The Ollama Operator makes it incredibly easy to deploy and manage Ollama instances on a cluster. This guide will walk you through the process, from installing the operator to testing your deployed Ollama service.

What is the Ollama Operator?

The Ollama Operator is designed to streamline the deployment and management of Ollama on Kubernetes and OpenShift clusters. It automates tasks like creating necessary resources, managing deployments, and ensuring your Ollama inference server is up and running.

You can find the ollama operator's repository on github.

Install the Operator

First, let's install the Ollama Operator itself. This will create a new project/namespace (llama-stack-k8s-operator-system) for Ollama and then apply the operator's manifest.

oc apply -f https://raw.githubusercontent.com/llamastack/llama-stack-k8s-operator/main/release/operator.yaml

You should see output similar to this, indicating the creation of various Kubernetes resources:

namespace/llama-stack-k8s-operator-system created
customresourcedefinition.apiextensions.k8s.io/llamastackdistributions.llamastack.io created
serviceaccount/llama-stack-k8s-operator-controller-manager created
role.rbac.authorization.k8s.io/llama-stack-k8s-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/llama-stack-k8s-operator-manager-role created
clusterrole.rbac.authorization.k8s.io/llama-stack-k8s-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/llama-stack-k8s-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/llama-stack-k8s-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/llama-stack-k8s-operator-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/llama-stack-k8s-operator-proxy-rolebinding created
configmap/llama-stack-k8s-operator-manager-config created
service/llama-stack-k8s-operator-controller-manager-metrics-service created
deployment.apps/llama-stack-k8s-operator-controller-manager created

This confirms that the operator has been successfully deployed to the llama-stack-k8s-operator-system namespace.

Run the Quickstart Script

Now that the operator is running, we can use a convenient quickstart script provided in the operator's repository to deploy an Ollama instance.

Clone the Repository

git clone https://github.com/opendatahub-io/llama-stack-k8s-operator.git

Navigate to the Repository Directory

cd llama-stack-k8s-operator

Execute the Quickstart Script ./hack/deploy-quickstart.sh
The script will create a new namespace (ollama-dist) and deploy Ollama as a provider. It will also handle necessary OpenShift security context constraints (SCCs) if you're on OpenShift.

./hack/deploy-quickstart.sh
Checking if namespace ollama-dist exists...
Creating namespace ollama-dist...
namespace/ollama-dist created
Start deploying ollama as provider with configuration:
  ServingRuntime Image: ollama/ollama:latest
  Inference Server: ollama
  Model: llama3.2:1b
Checking if ServiceAccount ollama-sa exists...
Creating ServiceAccount ollama-sa...
serviceaccount/ollama-sa created
Checking if OpenShift prerequisites exist in namespace: ollama-dist for service account: ollama-sa...
Creating ollama-scc, ollama-scc-role and ollama-scc-rolebinding for ollama-sa
securitycontextconstraints.security.openshift.io/ollama-scc created
role.rbac.authorization.k8s.io/ollama-scc-role created
rolebinding.rbac.authorization.k8s.io/ollama-scc-rolebinding created
Annotating ServiceAccount to clarify that it uses ollama-scc...
serviceaccount/ollama-sa annotate
Creating ollama-server deployment and service with image: ollama/ollama:latest...
deployment.apps/ollama-server created
service/ollama-server-service created
This may take up to 5 minutes for the image to be pulled and container to start...
Waiting for deployment "ollama-server" rollout to finish: 0 of 1 updated replicas are available...
deployment "ollama-server" successfully rolled out
Deployment is ready!
ollama inference server is now running!
  Namespace: ollama-dist
  Service: ollama-server-service
  Port: 11434
Access at:
  http://ollama-server-service.ollama-dist.svc.cluster.local:11434

The script will wait for the ollama-server deployment to be ready, which may take a few minutes as the Ollama image (ollama/ollama:latest) and the llama3.2:1b model are pulled.

Test the Ollama Service

Once the deployment is ready, you can test if Ollama is working correctly.

Port Forward the Ollama Service

To access the Ollama service from your local machine, port-forward it:

oc port-forward -n ollama-dist svc/ollama-server-service 11434:11434

This command will forward local port 11434 to the Ollama service inside your cluster. You'll see output similar to:

Forwarding from [::1]:11434 -> 11434
Handling connection for 11434
Handling connection for 11434

Keep this command running in a separate terminal.

Test the /api/tags Endpoint

Now, open another terminal and test the /api/tags endpoint to see the models available:

curl http://localhost:11434/api/tags

You should receive a JSON response showing the llama3.2:1b model that was deployed:

{"models":[{"name":"llama3.2:1b","model":"llama3.2:1b","modified_at":"2025-07-31T09:31:39.9132431Z","size":1321098329,"digest":"baf6a787fdffd633537aa2eb51cfd54cb93ff08e28040095462bb63daf552878","details":{"parent_model":"","format":"gguf","family":"llama","families":["llama"],"parameter_size":"1.2B","quantization_level":"Q8_0"}}}]}

Test the /api/generate Endpoint

Finally, let's test the /api/generate endpoint to interact with the model:

curl -X POST http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.2:1b",
    "prompt": "Hello, how are you?",
    "stream": false
  }'

You should receive a JSON response showing the api chat response

{"model":"llama3.2:1b","created_at":"2025-07-31T09:42:46.926734668Z","response":"I'm doing well, thank you for asking. Is there anything I can help you with or would you like to talk about something in particular?","done":true,"done_reason":"stop","context":[128006,9125,128007,271,38766,1303,33025,2696,25,6790,220,2366,18,271,128009,128006,882,128007,271,9906,11,1268,527,499,30,128009,128006,78191,128007,271,40,2846,3815,1664,11,9901,499,369,10371,13,2209,1070,4205,358,649,1520,499,449,477,1053,499,1093,311,3137,922,2555,304,4040,30],"total_duration":5444261976,"load_duration":2251218959,"prompt_eval_count":31,"prompt_eval_duration":734841277,"eval_count":30,"eval_duration":2457145861}

Conclusion and Next Steps

You have successfully deployed a fully functional and scalable Ollama inference server on your OpenShift cluster. By using the Ollama Operator, you've bypassed complex manual setup and now have a robust service ready to be integrated into your projects.

So, what's next? Here are a few ideas to explore:

Try Different Models: Explore the Ollama model library for other models. The quickstart deployed llama3.2:1b, but you can easily try others with the --provider and --model flags. e.g.

./hack/deploy-quickstart.sh --provider ollama --model mistral:latest

NOTE: llama3.2:1b is a small model other models may take longer to deploy

Expose Your Service: Instead of using port-forward, create an OpenShift Route to expose your Ollama service with a public URL. This makes it accessible to external applications and users. You can do this with a simple command:

oc expose svc/ollama-server-service -n ollama-dist

Integrate with an Application: Now that you have a stable API endpoint, connect it to your own applications. Whether it's a RAG (Retrieval-Augmented Generation) pipeline, a custom chatbot, or an automation script, your Ollama service is ready to power it.
Explore the Operator: Dive deeper into the Ollama Operator's documentation to understand its features, such as managing multiple models and custom configurations.

AI-Powered Kubernetes Debugging with Python and Ollama

Austin Cunningham — Mon, 28 Jul 2025 12:43:06 +0000

More playing around with AI development. Decided to see if I could build something useful. As I work with Kubernetes I thought I would build some sort of pod troubleshooter.

What It Does

This script does the heavy lifting in Kubernetes pod debugging:

Scans a namespace for failing pods
Collects diagnostics: describe, logs, and events
Sends the info to Ollama running llama3
Displays a full AI-generated root cause analysis

Requirements

Python 3.9+
rich
requests
kubectl
Ollama with llama3 model

Install dependencies

pip install rich requests

Install and Start Ollama

curl -fsSL https://ollama.com/install.sh | sh

ollama pull llama3
ollama run llama3

How It Works

First we pull in the imports and create a helper function to handle shell commands in string format

import argparse
import subprocess
import json
import requests
from rich.console import Console
from rich.panel import Panel

# Initialize rich  
console = Console()

def run_command(command):
    """
    Runs a shell command. If the command succeeds, it returns stdout.
    If it fails with a CalledProcessError (like a normal kubectl error),
    it returns stderr so the AI can analyze the error message.
    For other exceptions, it prints an error and returns an error string.
    """
    try:
        # Using shell=True for simplicity, but be cautious with untrusted input. Not sanitizing command line input here.  
        result = subprocess.run(
            command,
            shell=True,
            check=True,
            capture_output=True,
            text=True,
            timeout=30
        )
        return result.stdout
    except subprocess.CalledProcessError as e:
        # This is an expected failure (e.g., `logs --previous` on a new pod).
        # We return the error message for the AI to analyze.
        return e.stderr
    except Exception as e:
        # For any other kind of error, log it and return a failure message.
        # This prevents the whole script from crashing.
        error_message = f"An unexpected error occurred running command: {command}\nError: {e}"
        console.print(f"[bold red]{error_message}[/bold red]")
        return error_message

Finding Failing Pods

Next we create a function for Finding Failing Pods

def get_failing_pods(namespace):
    """Finds all pods in a namespace that are not in a 'Running' or 'Succeeded' state."""
    console.print(f"\n[bold cyan]🔍 Searching for failing pods in namespace '{namespace}'...[/bold cyan]")
    command = f"kubectl get pods -n {namespace} -o json"
    pods_json_str = run_command(command)

    if not pods_json_str:
        console.print(f"[bold red]Could not fetch pods in namespace '{namespace}'. No output received.[/bold red]")
        return []

    try:
        pods_data = json.loads(pods_json_str)
    except json.JSONDecodeError:
        console.print(f"[bold red]Failed to parse JSON output from kubectl for namespace '{namespace}'.[/bold red]")
        # The output might be an error message from kubectl
        console.print(f"Received: {pods_json_str}")
        return []

    failing_pods = []
    for pod in pods_data.get("items", []):
        pod_name = pod.get("metadata", {}).get("name")
        phase = pod.get("status", {}).get("phase")

        # Pod phase is not Running, Succeeded, or Completed
        if phase not in ["Running", "Succeeded"]:
            failing_pods.append(pod_name)
            continue

        # Pod phase is Running, but its containers are not ready or have crashed.
        if phase == "Running":
            container_statuses = pod.get("status", {}).get("containerStatuses", [])
            if not container_statuses:
                # A running pod should have container statuses. If not, it's an issue.
                failing_pods.append(pod_name)
                continue

            for container in container_statuses:
                # If any container is not ready, the pod is considered failing.
                if not container.get("ready", False):
                    failing_pods.append(pod_name)
                    break  # Found a failing container, no need to check others in this pod.

                # Explicitly check for terminated state with non-zero exit code
                terminated_state = container.get("state", {}).get("terminated")
                if terminated_state and terminated_state.get("exitCode", 0) != 0:
                    failing_pods.append(pod_name)
                    break

    # Remove duplicates that might occur if a pod is both 'not ready' and has other issues.
    return sorted(list(set(failing_pods)))

The script runs:

kubectl get pods -n <namespace> -o json

It filters out pods not in "Running" or "Succeeded" phase, and checks:

Missing or unhealthy container statuses
Containers not ready
Terminated containers with non-zero exit codes

Collect Diagnostics

Next we Collect Diagnostics

def get_pod_diagnostics(pod_name, namespace="default"):
    """Gathers describe, logs, and events for a given pod."""
    console.print(f"[bold cyan]🔍 Gathering diagnostics for pod '{pod_name}' in namespace '{namespace}'...[/bold cyan]")

    diagnostics = {}
    # Get pod  description
    diagnostics["describe"] = run_command(f"kubectl describe pod {pod_name} -n {namespace}")

    # Get pod logs (including previous container if it crashed)
    diagnostics["logs"] = run_command(f"kubectl logs {pod_name} -n {namespace} --all-containers=true")
    diagnostics["previous_logs"] = run_command(
        f"kubectl logs {pod_name} -n {namespace} --all-containers=true --previous")

    # Get relevant events in the namespace
    uid_command = f"kubectl get pod {pod_name} -n {namespace} -o jsonpath='{{.metadata.uid}}'"
    pod_uid = run_command(uid_command).strip()
    if pod_uid and 'error' not in pod_uid.lower():
        diagnostics["events"] = run_command(
            f"kubectl get events -n {namespace} --field-selector involvedObject.uid={pod_uid}")
    else:
        diagnostics["events"] = "Could not retrieve pod UID to filter events."

    return diagnostics

For each failing pod, it gathers:

kubectl describe pod <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> --all-containers
kubectl logs <pod-name> -n <namespace> --all-containers --previous

And also filters relevant events:

kubectl get events -n <namespace> --field-selector involvedObject.uid=<uid>

Analyze with Ollama

Next we create a AI prompt with the diagnostic data and pass it to Ollama that is running locally

def analyze_with_ollama(diagnostics, pod_name):
    """Sends diagnostics to Ollama for analysis and streams the response."""
    console.print(f"[bold cyan]🧠 Analyzing diagnostics for '{pod_name}' with Ollama...[/bold cyan]")

    prompt = f"""
You are an expert Kubernetes Site Reliability Engineer (SRE). Your task is to diagnose a failing pod based on the following `kubectl` outputs.

Analyze the provided data and perform the following steps:
1.  **Identify the Root Cause:** State the most likely root cause of the problem in a single, concise sentence.
2.  **Provide a Detailed Explanation:** Explain your reasoning. Reference specific lines from the logs, events, or pod description to support your conclusion.
3.  **Suggest a Next Step:** Recommend a single, concrete `kubectl` command or action for the user to take next to fix or further investigate the issue.

Here is the diagnostic information:

---
### 1. KUBECTL DESCRIBE POD:
{diagnostics.get('describe', 'Not available')}
---
### 2. KUBECTL LOGS (Current Container):
{diagnostics.get('logs', 'Not available')}
---
### 3. KUBECTL LOGS (Previous Container):
{diagnostics.get('previous_logs', 'Not available')}
---
### 4. KUBECTL GET EVENTS:
{diagnostics.get('events', 'Not available')}
---

Provide your analysis in a clear, easy-to-read format.
"""

    # Use a Panel to visually separate the analysis for each pod
    panel = Panel(f"[italic]Waiting for AI response for [bold]{pod_name}[/bold]...[/italic]",
                  title=f"[bold green]AI Diagnosis for {pod_name}[/bold green]", border_style="green")
    console.print(panel)

    full_response = ""
    try:
        with requests.post(
                "http://localhost:11434/api/generate",
                json={
                    "model": "llama3",
                    "prompt": prompt,
                    "stream": True
                },
                stream=True,
                timeout=300
        ) as response:
            response.raise_for_status()

            console.print(f"\n[bold green]AI Analysis for {pod_name}:[/bold green]")
            for line in response.iter_lines():
                if line:
                    chunk = json.loads(line)
                    content = chunk.get("response", "")
                    full_response += content
                    console.print(content, end="", style="white")
                    if chunk.get("done"):
                        console.print()
                        break
        return full_response

    except requests.exceptions.RequestException as e:
        console.print(f"\n[bold red]Error connecting to Ollama API: {e}[/bold red]")
        console.print("Please ensure Ollama is running and the 'llama3' model is available (`ollama pull llama3`).")
        return None

It streams this prompt to Ollama's local API running the llama3 model and prints the response.

Bring it all together

In the main function we

setup the prompt
handle the namespace arguments.
We then call the get_failing_pods(namespace) function to find the failing pods
We then call get_pod_diagnostics(pod_name, namespace) to get the kubectl diagnostic information
Finally we call analyze_with_ollama(pod_diagnostics, pod_name) to pass in the diagnostic data and build the AI prompt and query the AI(ollama) api endpoint for answers.

if __name__ == "__main__":
    console.print(f"[bold cyan]888      .d8888b.                     888                888 [/bold cyan] ")
    console.print(f"[bold cyan]888     d88P  Y88b                    888                888 [/bold cyan]")
    console.print(f"[bold cyan]888     Y88b. d88P                    888                888 [/bold cyan]")
    console.print(f"[bold cyan]888  888 \"Y88888\" .d8888b         .d88888 .d88b.  .d8888b888888 .d88b. 888d888\" [/bold cyan]")
    console.print(f"[bold cyan]888 .88P.d8P\"\"Y8b.88K            d88\" 888d88\"\"88bd88P\"   888   d88\"\"88b888P\" [/bold cyan]")
    console.print(f"[bold cyan]888888K 888    888\"Y8888b.888888 888  888888  888888     888   888  888888 [/bold cyan]")
    console.print(f"[bold cyan]888 \"88bY88b  d88P     X88       Y88b 888Y88..88PY88b.   Y88b. Y88..88P888 [/bold cyan]")
    console.print(f"[bold cyan]888  888 \"Y8888P\"  88888P\'        \"Y88888 \"Y88P\"  \"Y8888P \"Y888 \"Y88P\" 888    [/bold cyan] ")
    console.print(" ")
    parser = argparse.ArgumentParser(
        description="AI-powered Kubernetes Pod Troubleshooter.",
        formatter_class=argparse.RawTextHelpFormatter
    )
    parser.add_argument("-n", "--namespace", help="The namespace to inspect.")

    args = parser.parse_args()

    namespace = args.namespace
    if not namespace:
        namespace = console.input("[bold yellow]Please enter the Kubernetes namespace: [/bold yellow]")
        if not namespace:
            console.print("[bold red]Namespace cannot be empty. Exiting.[/bold red]")
            exit(1)

    # Automatically find all failing pods
    failing_pods = get_failing_pods(namespace)

    if not failing_pods:
        console.print(f"[bold green]✅ No failing pods found in namespace '{namespace}'.[/bold green]")
        exit(0)

    console.print(f"[bold yellow]Found {len(failing_pods)} failing pod(s): {', '.join(failing_pods)}[/bold yellow]")

    # Loop through each failing pod and diagnose it
    for pod_name in failing_pods:
        pod_diagnostics = get_pod_diagnostics(pod_name, namespace)
        if pod_diagnostics:
            analyze_with_ollama(pod_diagnostics, pod_name)
            console.print("-" * 80)  # Separator for clarity

Running the script

python k8s-doctor.py -n my-namespace

If you omit -n, the script will prompt you to enter a namespace interactively.

Code lives here

View the full Python script on GitHub

To summarize I am using this script to automate log/event gathering for failed pods, creating a prompt for an AI LLM and returning a response from AI. Is this a wrapper for AI? Is this automation? a bit of both I think.

Build a Local AI RAG App with Ollama and Python

Austin Cunningham — Thu, 10 Jul 2025 14:18:00 +0000

I was looking to do a little development around AI and decided to see what I could build on my PC.

We tried Ollama for a local development environment. Ollama is an open-source tool that makes it easy to download, run, and manage large language models (LLMs) on your own computer.

curl -fsSL https://ollama.com/install.sh | sh

Once downloaded and installed you can run as follows

~ ollama -v         
ollama version is 0.6.5
~ ollama run mistral
>>> tell me about ollama run mistral
 "Olama Run Mistral is a high-performance, open-source machine learning platform developed by OLMA.AI, a company based in Paris, France. The platform is designed to simplify the development and deployment of large-scale machine 
learning models with a focus on deep learning.
Mistral stands out for its scalability, flexibility, and ease of use. It's built around a distributed architecture that allows users to train and deploy large-scale machine learning models efficiently across multiple GPUs and 
clusters, reducing the time required for training and inference significantly.
Some key features of Olama" # ====> you get the idea a chatbot in your terminal

There is a bunch of models you can run https://ollama.com/search we will stick to mistral for the time being. And we need to keep it running to use it.

So what are we going to try to do ? We are going to create an chat app that can query my local files directory and return answers based off the contents.

First create requirements.txt and add the following, basically some libraries for interacting with text and ollama

langchain
langchain-community
langchain-ollama
langchain-huggingface
chromadb
sentence-transformers

Setup a virtual environment and install the requirements

sudo dnf install gcc-c++ python3-devel
pip install langchain chromadb sentence-transformers ollama
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

I create two files a main.py and utils/loaders.py

utils/loaders.py

from langchain_community.document_loaders import DirectoryLoader, TextLoader

import os


def load_sop_files(directory: str):
    allowed_exts = ('.md', '.asciidoc', '.txt')

    docs = []
    for root, _, files in os.walk(directory):
        for file in files:
            if file.lower().endswith(allowed_exts):
                path = os.path.join(root, file)
                try:
                    loader = TextLoader(path, encoding='utf-8')
                    docs.extend(loader.load())
                except Exception as e:
                    print(f"❌ Error loading {path}: {e}")
    return docs

utils/loaders.py function takes in a directory path and goes through all the files and finds type md,asciidoc and txt and breaks them up into an array utf-8 formatted langchain docs.

main.py

First thing I do in main.py is add the libraries and then call the function in utils/loaders.py to load my docs into the application.

from utils.loaders import load_sop_files 
from langchain.text_splitter import 
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.vectorstores import Chroma
from langchain.chains import RetrievalQA
from langchain_ollama import OllamaLLM
from langchain_huggingface import HuggingFaceEmbeddings




# Load and prepare documents
print("📂 Loading SOP documents...")
# pointing to local directory that is the same level as this project
docs = load_sop_files("../help/sops/")

LLM's need data split up into smaller sizes

splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=100)
chunks = splitter.split_documents(docs)

We then need to convert the document data into a numeric format that can be handled by the LLM


print("🧠 Creating vector database...")
embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
db = Chroma.from_documents(chunks, embeddings)

We set up Ollama mistral to take the numeric db data and use to
to formulate its answers, We are not training a modal here but
using the RAG(Retrieval-Augmented Generation) pattern

retriever = db.as_retriever()
llm = OllamaLLM(model="mistral")
qa = RetrievalQA.from_chain_type(llm=llm, retriever=retriever, return_source_documents=True)

NOTE : We are using the OllamaLLM and RetrievalQA to connect to our local LLM API but we could use the Ollama API here with the /api/generate end point, but this would be more verbose.

We then have a loop here to answer queries using the qa we created earlier as a source with the invoke function


print("🤖 SOP Assistant ready. Type your question below. Type 'exit' to quit.")



while True:
   query = input("\n📝 You: ")
   if query.lower() in ("exit", "quit"):
       print("👋 Bye! Take care.")
       break


   result = qa.invoke({"query": query})


   print("\n🤖 Assistant:\n", result["result"])


   print("\n📎 Sources:")
   for doc in result["source_documents"]:
       print(f" - {doc.metadata.get('source')}")

Repo https://github.com/austincunningham/sop_assistant

It's returning answers based off the ../help/sops directory we passed in earlier

Credit to https://github.com/valerymo for a lot of the investigation on getting this operational.

Monitor internet uptime on your local PC

Austin Cunningham — Fri, 28 Feb 2025 15:53:00 +0000

I wanted to monitor my Internet uptime and graph it over time on my PC which is on Fedora 39. I gave some taught to running ping scripts but these didn't meet my needs. After a bit of research I found that Prometheus has all the tools I need. First Installed Prometheus first I tried dnf but that failed

sudo dnf install -y prometheus
No match for argument: prometheus
Error: Unable to find a match: prometheus

So I installed from the binary when it's downloaded you get two binarys Prometheus,Promtool and a config file prometheus.yml after I extracted these I moved the binary to /usr/local/bin and the config file to /etc/prometheus/

sudo cp prometheus /usr/local/bin
sudo cp promtool /usr/local/bin
sudo cp prometheus.yml /etc/prometheus

I simplified my config prometheus.yml to two job one for prometheus and one for node_exporter(we will get to this in a while)

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node_exporter"
    static_configs:
      - targets: ["localhost:9100"]

Make sure a prometheus user exists and has permissions to run the prometheus service

# check the user exists
id prometheus
uid=977(prometheus) gid=971(prometheus) groups=971(prometheus)
# add permissions to prometheus files
sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /usr/local/bin/prometheus

I then created a service file which points at the binary and the config file and the storage destination

sudo vi /etc/systemd/system/prometheus.service

with the following contents

[Unit]
Description=Prometheus
After=network.target

[Service]
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus/

[Install]
WantedBy=multi-user.target

I then started the prometheus service

sudo systemctl enable --now prometheus
Created symlink /etc/systemd/system/multi-user.target.wants/prometheus.service → /etc/systemd/system/prometheus.service.

I needed to also install node_exporter to expose machine metrics from my local PC, fortunately dnf worked for this so no messing around with moving binarys and setting up config files and services

sudo yum install -y node_exporter

I could start the service out of the box

sudo systemctl enable --now node_exporter
Created symlink /etc/systemd/system/multi-user.target.wants/prometheus-node-exporter.service → /usr/lib/systemd/system/prometheus-node-exporter.service.

I now had the prometheus ui up and running with metrics exposed from the PC on http://localhost:9090 .

I needed a way to get my internet up time. Prometheus itself doesn't have a way to ping a remote service but it does have a package that can do this called blackbox_exporter. I tried installing with dnf but it failed

sudo dnf install blackbox_exporter
No match for argument: blackbox_exporter
Error: Unable to find a match: blackbox_exporter

The binary was available for download at the prometheus download site. So after I downloaded and extracted it I had a binary blackbox_exporter and a config file blackbox.yml. Similar process to before I copied the binary to /usr/local/bin and copied the config to /etc. Again I created a service

sudo vi /etc/systemd/system/blackbox_exporter.service

With the following contents

[Unit]
Description=Blackbox Exporter
After=network.target

[Service]
ExecStart=/usr/local/bin/blackbox_exporter --config.file=/etc/blackbox.yml
Restart=always
User=blackbox
Group=blackbox

[Install]
WantedBy=multi-user.target

But when I attempted to start the service it failed as I hadn't setup the User and group as the status tells me

sudo systemctl enable --now blackbox_exporter        
sudo systemctl status blackbox_exporter        
× blackbox_exporter.service - Blackbox Exporter
     Loaded: loaded (/etc/systemd/system/blackbox_exporter.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Fri 2025-02-28 08:28:12 GMT; 7s ago
   Duration: 1ms
    Process: 323975 ExecStart=/usr/local/bin/blackbox_exporter --config.file=/etc/blackbox.yml (code=exited, status=216/GROUP)
   Main PID: 323975 (code=exited, status=216/GROUP)
        CPU: 907us

Feb 28 08:28:12 fedora systemd[1]: blackbox_exporter.service: Scheduled restart job, restart counter is at 5.
Feb 28 08:28:12 fedora systemd[1]: blackbox_exporter.service: Start request repeated too quickly.
Feb 28 08:28:12 fedora systemd[1]: blackbox_exporter.service: Failed with result 'exit-code'.
Feb 28 08:28:12 fedora systemd[1]: Failed to start blackbox_exporter.service - Blackbox Exporter.

So first I checked the blackbox user exists and created them and gave them permissions to the binary and config file

# check the user
id blackbox
id: ‘blackbox’: no such user
# create the user
sudo useradd -r -s /bin/false blackbox
# confirm they were create 
id blackbox                           
uid=976(blackbox) gid=970(blackbox) groups=970(blackbox)
# give ownership to blackbox user and group the blackbox binary and config file
sudo chown blackbox:blackbox /usr/local/bin/blackbox_exporter
sudo chown blackbox:blackbox /etc/blackbox.yml         
# restart daemon and service
sudo systemctl daemon-reload
sudo systemctl restart blackbox_exporter
# check the status
sudo systemctl status blackbox_exporter
# status is good
● blackbox_exporter.service - Blackbox Exporter
     Loaded: loaded (/etc/systemd/system/blackbox_exporter.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (running) since Fri 2025-02-28 08:31:09 GMT; 7s ago
   Main PID: 324468 (blackbox_export)
      Tasks: 10 (limit: 76747)
     Memory: 21.3M
        CPU: 10ms
     CGroup: /system.slice/blackbox_exporter.service
             └─324468 /usr/local/bin/blackbox_exporter --config.file=/etc/blackbox.yml

Feb 28 08:31:09 fedora systemd[1]: Started blackbox_exporter.service - Blackbox Exporter.
Feb 28 08:31:09 fedora blackbox_exporter[324468]: time=2025-02-28T08:31:09.098Z level=INFO source=main.go:86 msg="Starting blackbox_exporter" version=">
Feb 28 08:31:09 fedora blackbox_exporter[324468]: time=2025-02-28T08:31:09.098Z level=INFO source=main.go:87 msg="(go=go1.23.6, platform=linux/amd64, u>
Feb 28 08:31:09 fedora blackbox_exporter[324468]: time=2025-02-28T08:31:09.099Z level=INFO source=main.go:99 msg="Loaded config file"
Feb 28 08:31:09 fedora blackbox_exporter[324468]: time=2025-02-28T08:31:09.099Z level=INFO source=tls_config.go:347 msg="Listening on" address=[::]:9115
Feb 28 08:31:09 fedora blackbox_exporter[324468]: time=2025-02-28T08:31:09.099Z level=INFO source=tls_config.go:350 msg="TLS is disabled." http2=false >

Now I can confirm that the blackbox_exporter is working with the following curl command and it should return metrics

curl -s "http://localhost:9115/probe?module=icmp&target=8.8.8.8" 
# HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in seconds
# TYPE probe_dns_lookup_time_seconds gauge
probe_dns_lookup_time_seconds 9.23e-06
# HELP probe_duration_seconds Returns how long the probe took to complete in seconds
# TYPE probe_duration_seconds gauge
probe_duration_seconds 0.010457657
# HELP probe_icmp_duration_seconds Duration of icmp request by phase
# TYPE probe_icmp_duration_seconds gauge
probe_icmp_duration_seconds{phase="resolve"} 9.23e-06
probe_icmp_duration_seconds{phase="rtt"} 0.010111051
probe_icmp_duration_seconds{phase="setup"} 0.000212389
# HELP probe_icmp_reply_hop_limit Replied packet hop limit (TTL for ipv4)
# TYPE probe_icmp_reply_hop_limit gauge
probe_icmp_reply_hop_limit 57
# HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to detect if the IP address changes.
# TYPE probe_ip_addr_hash gauge
probe_ip_addr_hash 2.350491669e+09
# HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6
# TYPE probe_ip_protocol gauge
probe_ip_protocol 4
# HELP probe_success Displays whether or not the probe was a success
# TYPE probe_success gauge
probe_success 1

Ok I am seeing metrics returned. We next need to add these metrics to prometheus. For this we need to add a job to the /etc/prometheus/prometheus.yml config file so we edit the config scrape_config so the file looks like

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "blackbox"
    metrics_path: /probe
    params:
      module: [icmp]  # this module is defined in the /etc/blackbox.yml file
    static_configs:
      - targets:
          - "8.8.8.8"  # Google DNS (change to another IP to monitor)
          - "8.8.4.4" 
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115

  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node_exporter"
    static_configs:
      - targets: ["localhost:9100"]

NOTE: this would be good to monitor any web page or services ip address by updating the target list for now we will stick to the google dns. See Prometheus docs for more info

Once this is updated restart the prometheus service

sudo systemctl restart prometheus

The result on localhost:9090 we can check the up metric to see if our internet is up over time 1 is good 0 is bad.

I installed grafana to get a prettier graph, dnf worked

sudo dnf install grafana

The Grafana services was running on http://localhost:3000/ logged in with the default user admin and the default password admin, was asked to update the password by the ui. I had an issue adding the Prometheus data source , Was seeing the follow error

Error reading Prometheus: Get "http://127.0.0.1:9090/api/v1/query?query=1%2B1&time=1740751852.139": dial tcp 127.0.0.1:9090: connect: permission denied

After a lot of messing around found it was a selinux restriction as I could connect to the datasource when selinux was disabled

# works when this is disabled
sudo setenforce 0
# doesn't when enabled
sudo setenforce 1
# checked the journalctl log
sudo journalctl -t setroubleshoot | tail -20
Feb 28 14:25:33 fedora setroubleshoot[359567]: SELinux is preventing grafana-server from name_connect access on the tcp_socket port 9090.

                                               *****  Plugin catchall_boolean (89.3 confidence) suggests   ******************

                                               If you want to allow grafana to can tcp connect prometheus port
                                               Then you must tell SELinux about this by enabling the 'grafana_can_tcp_connect_prometheus_port' boolean.

                                               Do
                                               setsebool -P grafana_can_tcp_connect_prometheus_port 1

                                               *****  Plugin catchall (11.6 confidence) suggests   **************************

                                               If you believe that grafana-server should be allowed name_connect access on the port 9090 tcp_socket by default.
                                               Then you should report this as a bug.
                                               You can generate a local policy module to allow this access.
                                               Do
                                               allow this access for now by executing:
                                               # ausearch -c 'grafana-server' --raw | audit2allow -M my-grafanaserver
                                               # semodule -X 300 -i my-grafanaserver.pp

# I set the selinux bool as the first section of the journal log suggested
sudo setsebool -P grafana_can_tcp_connect_prometheus_port 1

And that was it was able to add the datasource in grafana.

Sometimes you really get into the weeds. Anyway here is grafana dashboard

Load testing 3scale

Austin Cunningham — Fri, 26 Apr 2024 15:01:50 +0000

I was interested in doing some load testing against 3scale. After a bit of digging around I found that there was an open source load testing toolkit in https://github.com/3scale-labs/perftest-toolkit.

At first look I was unsure what this gave me or what it required. So after playing around with it a while I eventually figured it out along with a few pain points.

What do we need?

Prerequisites:

An Openshift Kubernetes cluster
An instance of 3scale on this cluster
An ec2 instance that can connect a route on your cluster. It also can connect to your local pc via ssh. As your load testing this ec2 instance needs to be resourced to handle the load e.g.c5.xlarge.
Ansible 2.9.14 (this is used by the repo to deploy resources)
An endpoint like the echo-api endpoint, the following commands below to deploy an echo-api in an echo-api namespace

oc new-project echo-api
cat << EOF | oc create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo-api
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
  template:
    metadata:
      labels:
        app: echo-api
    spec:
      containers:
        - name: echo-api
          image: quay.io/3scale/echoapi:stable
          livenessProbe:
            tcpSocket:
              port: 9292
            initialDelaySeconds: 10
            timeoutSeconds: 1
          readinessProbe:
            httpGet:
              path: /test/200
              port: 9292
            initialDelaySeconds: 15
            timeoutSeconds: 1
          ports:
            - containerPort: 9292
              protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: echo-api
spec:
  ports:
  - name: echo-api-port
    port: 9292
    protocol: TCP
    targetPort: 9292
  selector:
    app: echo-api
  type: ClusterIP
---
kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: echo-api
  namespace: echo-api
spec:
  to:
    kind: Service
    name: echo-api
  port:
    targetPort: echo-api-port
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Allow
  wildcardPolicy: None
---
EOF

What does it give us?

Well the perftest-toolkit creates 3scale resources like backends and products.

It installs Hyperfoil(cli load testing tool)locally and on the remote ec2 instance. This can be accessed via the browser on the ec2 instance public address via port 8090 or directly on the ec2 instances

It uploads a benchmark yaml file like the following

name: 3scale-benchmark
agents:
  ec2-agent: 
    host: ec2-54-93-91-242.eu-central-1.compute.amazonaws.com
    port: 22
http:
- host: https://perf-test-dwxpoent-3scale-apicast-production.apps.aucunnin-ccs.lpi0.s1.devshift.org:443
  sharedConnections: 50
usersPerSec: 228
duration: 3600s
maxDuration: 3600s
scenario:
- testSequence:
  - randomCsvRow:
     file: /tmp/3scale.csv
     skipComments: 'True'
     removeQuotes: 'True'
     columns:
       0: target-host
       1: uri
  - template:
      pattern: ${target-host}:443
      toVar: target-authority
  - httpRequest:
      authority:
        fromVar: target-authority
      GET:
        fromVar: uri
      headers:
        HOST:
          fromVar: target-host

threads: 4

It also uploads a csv with endpoint data to the /tmp directory on the ec2 instance which is consumed by the benchmark above.

cat 3scale.csv 
"perf-test-dwxpoent-3scale-apicast-production.apps.aucunnin-ccs.lpi0.s1.devshift.org","/pets?user_key=<some_user_key_from_3scale>"

It will run the test automatically and returns a html report when done e.g. 0001.html

Pain Points

Firstly a couple of pain points. I needed to install Docker on the ec2 instance as it was a Redhat RHEL 8 AIM I used the following commands on the ec2 instance

sudo yum-config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo &&
sudo yum install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin &&
sudo systemctl start docker &&
sudo usermod -aG docker $USER &&
newgrp docker

The version of Hyperfoil I was using requires id_rsa key pair on the ec2 instance, so I generated them and added the public key to the authorized_keys

cd ~/.ssh/
ssh-keygen -t rsa
cat id_rsa.pub >> authorized_keys

There are few Ansible playbooks to run as part of the perftest-toolkit but I hit a few blockers when trying to run them. In the end I ended up using ansible-galaxy to install the missing collections

# You may not have to do this ansible-galaxy collection list will show you if the are installed or not
ansible-galaxy collection install ansible.posix
ansible-galaxy collection install community.docker
ansible-galaxy collection install community.general

In deployment/ansible.cfg I needed to point the ssh key to allow ansible to access the ec2 instance via ssh so added the [ssh_connection] section. Updating this was in the readme but was missing the key

# project specific configuration for Ansible 

[defaults]
inventory_plugins=inventory
command_warnings=False

[inventory]
enable_plugins=3scale_inventory_plugin,ini

[ssh_connection]
ssh_args = -o ServerAliveInterval=30 -i "~/.ssh/pem_key_name.pem"
pipelining = True

In deployment/roles/platform-setup/vars/RedHat.yml need to set the java version

---
java_package:
  - java-11-openjdk
unzip_package:
  - unzip

One final pain point found that running Hyperfoil for longer runs 4 to 6 hours ended hitting up against some Garbage collection issue

Error [
"ec2-agent: Jitter watchdog was not invoked for 109 ms; check GC settings.",
"ec2-agent: Agent unexpectedly left the cluster."
]

Got around this by increasing the Java memory settings with extras: -XX:+UseShenandoahGC -Xms1G -Xmx4G in the hyperfoil benchmark to the max my ec2 instance could handle in this case 4GB. I updated deployment/benchmark/3scale-benchemark.yaml.j2 as follows

Getting started

Run the requirements playbook to install Hyperfoil locally

git clone https://github.com/3scale-labs/perftest-toolkit
cd deployment
ansible-galaxy install -r requirements.yaml

Next we edit a couple of files. First deployment/hosts we add in the ec2 instances public address

upstream ansible_host=ec2-54-93-91-242.eu-central-1.compute.amazonaws.com ansible_user=ec2-user

[hyperfoil_controller]
ec2 ansible_host=ec2-54-93-91-242.eu-central-1.compute.amazonaws.com ansible_user=ec2-user

[hyperfoil_agent]
ec2-agent ansible_host=ec2-54-93-91-242.eu-central-1.compute.amazonaws.com ansible_user=ec2-user

In the deployment/run.yml we set

shared_connections (Hyperfoil setting number of concurrent connections to your load testing endpoint. A pain point if set to low Hyperfoil will fail the load test and if set to high can overload the test, you have to play around with it. Found for a 5 million request per day shared_connection of 30 worked where as 100 we see an increase in the 4xx error rate)
users_per_sec (rate per second e.g. 5 million request per day 5000,000÷60÷60÷24 = 57.87 so round to 58 rps)
duration_sec (length of load test run in seconds)

---
- hosts: hyperfoil_controller
  roles:
    - hyperfoil.hyperfoil_test
  vars:
    test_name: 3scale-benchmark
    test_files:
      - "{{ csv_dest_file_path }}"
    shared_connections: 50
    users_per_sec: 58
    duration_sec: 3600
- hosts: hyperfoil_controller
  become: yes
  roles:
    - hyperfoil_generate_report
- hosts: hyperfoil_controller
  tasks:
    - name: Retrieve the report
      fetch:
        src: "{{ reports_path }}/{{ test_runid }}.html"
        dest: "{{ playbook_dir }}/{{ toolkit_csv_file_path }}/"
        flat: yes

Next we need to populate the main.yml found in deployments/roles/profiled-traffic-generator/defaults/main.yml
You need to get the 3scale-admin route, 3scale admin access token and the echo-api route

oc get routes -n <3scale-namespace-here> | grep 3scale-admin | awk '{print $2}'
3scale-admin.apps.aucunnin-ccs.lpi0.s1.devshift.org                                          
oc get secret system-seed -n <3scale-namespace-here> -o jsonpath="{.data.ADMIN_ACCESS_TOKEN}"| base64 --decode
<Some_token>
oc get routes -n echo-api | awk '{print $2}'
echo-api-echo-api.apps.aucunnin-ccs.lpi0.s1.devshift.org

And add them to main.yml as follows

---
# defaults file for profiled-traffic-generator

# URI that includes your password and portal endpoint in the following format: <schema>://<password>@<admin-portal-domain>.
# The <password> can be either the provider key or an access token for the 3scale Account Management API.
# <admin-portal-domain> is the URL used to log into the admin portal.
# Example: https://access-token@account-admin.3scale.net
threescale_portal_endpoint: https://<Some-token>@3scale-admin.apps.aucunnin-ccs.lpi0.s1.devshift.org/

# Used traffic for performance testing is not real traffic.
# It is synthetically generated traffic based on traffic models.
# Information about available traffic profiles (or test plans) can be found here:
# https://github.com/3scale/perftest-toolkit/blob/master/buddhi/README.md#profiles
# Currently available profiles: [ simple | backend | medium | standard ]
traffic_profile: simple

# Private Base URL
# Make sure your private application behaves like an echo api service
# example: https://echo-api.3scale.net:443
private_base_url: https://echo-api-echo-api.apps.aucunnin-ccs.lpi0.s1.devshift.org

# Public Base URL
# Public address of your API gateway in the production environment.
# Optional. When it is left empty, public base url will be the hosted gateway url
# example: https://gw.example.com:443
public_base_url:

Note we have selected the simple profile here as it's quickest to setup and is a single product and backend.Other profiles are backend, medium, standard with standard producing the most endpoints and being a slower setup.

So once we have the hosts, run and main set we can run the following playbook which is basically setup and deployment jobs

ansible-playbook -i hosts profiled-injector.yml

Once completed you can run your test with this playbook

ansible-playbook -i hosts -i benchmarks/3scale.csv run.yml

So how do you know it's working?

You will see a polling in the Ansible output

The Ansible output above is showing that this is run/0001 we can hit that endpoint using the ec2 public address and port 8090 and can check the status, below is an example of a complete status

You can ssh to the ec2 instance and access via the Hyperfoil cli in /tmp/hyperfoil/hyperfoile-0.15/bin and run the cli with ./cli.sh

As stated above you should get a run html file each time you run the load test in the deployments/benchmarks directory which can be opened in the browser.

We can then gather things from the report like percentile response times, number of successful/failed request etc.

Automate 3scale monitoring stack deployment

Austin Cunningham — Fri, 01 Mar 2024 11:37:22 +0000

I wanted to automate the deployment the a monitoring stack for 3scale which is documented here as there were a lot of manual steps.

First I clone the 3scale-operator repo as I need access to files there. I already have 3scale-operator setup in the 3scale-test project so I change to that project.

#!/bin/bash
git clone git@github.com:3scale/3scale-operator.git
cd 3scale-operator/doc/monitoring-stack-deployment
oc project 3scale-test

There were a couple of steps that involved using the Openshift UI to install operators. I needed a way to deploy the Prometheus and Grafana operators from the command line. As this was an Openshift 4 Kubernetes cluster it had OLM(operator lifecycle manager) installed. So it was just a matter of creating subscriptions which is the resource that OLM is watching for to deploy operators. And I got the prometheus and grafana operators installed

# Create a subscription for prometheus operator
oc apply -f - <<EOF
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: rhods-prometheus-operator
  namespace: 3scale-test
spec:
  channel: beta
  installPlanApproval: Automatic
  name: rhods-prometheus-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  startingCSV: rhods-prometheus-operator.4.10.0
EOF
# Create Grafana Subscription
oc apply -f - <<EOF
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: grafana-operator
  namespace: 3scale-test
spec:
  channel: v4
  installPlanApproval: Automatic
  name: grafana-operator
  source: community-operators
  sourceNamespace: openshift-marketplace
  startingCSV: grafana-operator.v4.10.1
EOF
# added a sleep to let the installs finish
sleep 60

You can see them installed in the Openshift UI

You have to enable monitoring in the APIManger custom resource the following command does this.

# patch apimanager called apimanger-sample CR monitoring enabled true
oc patch apimanager apimanager-sample --type='json' -p='[{"op": "add", "path": "/spec/monitoring", "value": {"enabled": true}}]'

This basically gets the 3scale-operator to deploy grafana and promentheus resources for monitoring 3scale i.e. grafana dashboards, prometheus alerts.

Next came setting up the scrapeconfig that allows scraping monitoring data from the clusters monitoring stack. This requires getting a token from the Openshift monitoring stack and patching that token into the scrapconfig file and create a secret from that file. Steps outline below

# Get the SECRET name that contains the THANOS_QUERIER_BEARER_TOKEN
SECRET=`oc get secret -n openshift-user-workload-monitoring | grep  prometheus-user-workload-token | head -n 1 | awk '{print $1 }'`
# Get the THANOS_QUERIER_BEARER_TOKEN using the SECRET name
THANOS_QUERIER_BEARER_TOKEN=$(oc get secret $SECRET -n openshift-user-workload-monitoring -o jsonpath="{.data.token}" | base64 -d)
# patch the THANOS_QUERIER_BEARER_TOKEN in the 3scale-scrape-configs.yaml
sed -i "s|<THANOS_QUERIER_BEARER_TOKEN>|$THANOS_QUERIER_BEARER_TOKEN|g" 3scale-scrape-configs.yaml
# create secret addition-scrape-configs from 3scale-scrape-configs.yaml file
oc create secret generic additional-scrape-configs --from-file=3scale-scrape-configs.yaml=./3scale-scrape-configs.yaml

NOTE: I have kubeadmin privilages, so I haven't investigated the minimum permissions required to get this secret

Next we need to set the route in the prometheus.yaml file and apply the file to create an instance of Prometheus and expose a route to access from the browser

# Prometheus CR
DOMAIN=$(oc get routes console -n openshift-console -o json | jq -r '.status.ingress[0].routerCanonicalHostname' | sed 's/router-default.//')
EXTERNALURL=https://prometheus.3scale-test.$DOMAIN
sed -i "s|externalUrl:.*|externalUrl: $EXTERNALURL|" prometheus.yaml
oc apply -f prometheus.yaml
sleep 5
oc expose service prometheus-operated --hostname prometheus.3scale-test.$DOMAIN

Then we can apply the grafana CR's(Custom Resources) to create an instance of the Grafana

# Grafana CR's
oc apply -f datasource.yaml
oc apply -f grafana.yaml

At this stage we should have a grafana and prometheus setup with all the 3scale monitoring available

And finally clean up the operator that was cloned

# remove 3scale-operator dir
cd ../../../
rm -rf 3scale-operator

code lives here

NOTE: this file also installs the prometheus-exporter to add a grafana dashboard for the backend redis.

Install Keycloak with the Operator

Austin Cunningham — Fri, 15 Sep 2023 10:46:42 +0000

Install the Keycloak via the operator

Create a project

oc new-project keycloak

I am going to install the operator the OperatorHub in Openshift which uses OLM(Operator Lifecyecle Manager). Select Operator/OperatorHub from the side bar and search for keycloak and install it in the keycloak namespace

The install should finish

I will go through the requirements now. This will not be a production setup.

Install an DB

Going to install an Ephemeral PostgreSQL DB as this is just a demo, create a yaml file called example-postgres.yaml with the following content.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgresql-db
spec:
  serviceName: postgresql-db-service
  selector:
    matchLabels:
      app: postgresql-db
  replicas: 1
  template:
    metadata:
      labels:
        app: postgresql-db
    spec:
      containers:
        - name: postgresql-db
          image: postgres:latest
          env:
            - name: POSTGRES_PASSWORD
              value: postgres #<-- change this
            - name: POSTGRES_USER
              value: postgres #<-- change this
            - name: PGDATA
              value: /data/pgdata
            - name: POSTGRES_DB
              value: keycloak
---
apiVersion: v1
kind: Service
metadata:
  name: postgres-db
spec:
  selector:
    app: postgresql-db
  ports:
  - port: 5432
    targetPort: 5432

And then apply the file to the cluster

oc apply -f example-postgres.yaml

The result

statefulset.apps/postgresql-db created
service/postgres-db created

After checking the postgres pod logs I see the following error

mkdir: cannot create directory ‘/data’: Permission denied

Nothing is ever easy. This is because Openshift blocks containers being run as root. To get this to work we need to add a service account with the correct permissions and add that service account to the statefulset. This is a good reference blog for more details. Here are the commands I used

# create the service account
~ oc create sa postgres-sa
serviceaccount/postgres-sa created
# add policy
~ oc adm policy add-scc-to-user anyuid -z postgres-sa
clusterrole.rbac.authorization.k8s.io/system:openshift:scc:anyuid added: "postgres-sa"
# set the policy on the namespace
~ oc adm policy add-scc-to-user anyuid -z postgres-sa -n keycloak
clusterrole.rbac.authorization.k8s.io/system:openshift:scc:anyuid added: "postgres-sa"
# add the service account to the statefulset
~ oc set sa statefulset postgresql-db postgres-sa
statefulset.apps/postgresql-db serviceaccount updated
# delete the failing existing pod
oc delete po postgresql-db-0
# check the pods 
~ oc get po
NAME                                 READY   STATUS    RESTARTS   AGE
keycloak-operator-5659f58f4b-vtrm9   1/1     Running   0          29m
postgresql-db-0                      1/1     Running   0          8m55s

Ok that's working

Hostname

We need a resolvable domain name. You can add any domain to your openshift cluster using a customDomain. As I couldn't be bothered setting up an actual domain name here or a valid cert this a quick hack for creating a resolvable domain name on Openshift.

# create a self signed cert 
openssl req -subj '/CN=apps.austin.me/O=Test Keycloak./C=US' -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 365 -out certificate.pem
# create cert project
oc new-project certs
# create a secret from that cert in the cert namespace
oc create secret tls austin-me-tls --cert=austin.me.crt --key=austin.me.key -n certs
# create a custom domain based on the cert
oc apply -f - <<EOF                                                      
---
apiVersion: managed.openshift.io/v1alpha1
kind: CustomDomain
metadata:
  name: cunningham
spec:
  domain: apps.austin.me
  scope: External
  certificate:
    name: austin-me-tls
    namespace: certs
EOF
# Wait for the custom domain to become ready 
oc get customdomains                                              
NAME         ENDPOINT                                          DOMAIN           STATUS
cunningham   oeffrs.cunningham.aucunnin.lpi0.s1.devshift.org   apps.austin.me   Ready

All pretty standard a this point but the domain apps.austin.me doesn't exist. So here is the hack edit the custom domain oc edit customdomain cunningham and replace the spec.host with the endpoint.

apiVersion: managed.openshift.io/v1alpha1
kind: CustomDomain
metadata: 
  name: cunningham
spec:
  certificate:
    name: austin-me-tls
    namespace: certs
  domain: oeffrs.cunningham.aucunnin.lpi0.s1.devshift.org #<---- was apps.austin.me now points at the endpoint
  loadBalancerType: Classic
  scope: External

So our domain going forward is the endpoint oeffrs.cunningham.aucunnin.lpi0.s1.devshift.org

Create a DB secret

oc project keycloak
# change these to match the secret used in your statefulset
kubectl create secret generic keycloak-db-secret \
  --from-literal=username=postgres \
  --from-literal=password=postgres

Create the Keycloak CR

The CR(Custom Resource) creates a instance of the Keycloak UI. Create a yaml file called example-kc.yaml

apiVersion: k8s.keycloak.org/v2alpha1
kind: Keycloak
metadata:
  name: example-kc
spec:
  instances: 1
  db:
    vendor: postgres
    host: postgres-db
    usernameSecret:
      name: keycloak-db-secret
      key: username
    passwordSecret:
      name: keycloak-db-secret
      key: password
  http:
    tlsSecret: austin-me-tls
  hostname:
    hostname: oeffrs.cunningham.aucunnin.lpi0.s1.devshift.org

And then apply the file

kubectl apply -f example-kc.yaml

We can check the CR status of keycloak, it will look like this when it is type: Ready status: "True"

# using https://mikefarah.gitbook.io/yq/
~ oc get keycloak example-kc -oyaml | yq '.status'
conditions:
  - lastTransitionTime: "2023-09-15T08:08:56.665783288Z"
    message: ""
    observedGeneration: 1
    status: "True"      #<------ This is what we are looking for I hate these condition to difficult to read a first glance
    type: Ready
  - lastTransitionTime: "2023-09-15T08:08:26.147242224Z"
    message: ""
    observedGeneration: 1
    status: "False"
    type: HasErrors
  - lastTransitionTime: "2023-09-15T08:08:26.147242224Z"
    message: ""
    observedGeneration: 1
    status: "False"
    type: RollingUpdate
instances: 1
observedGeneration: 1
selector: app=keycloak,app.kubernetes.io/managed-by=keycloak-operator,app.kubernetes.io/instance=example-kc

That looks to have completed and if we port-forward the service it looks to be working

~ oc port-forward service/example-kc-service 8443:8443
Forwarding from 127.0.0.1:8443 -> 8443
Forwarding from [::1]:8443 -> 8443
Handling connection for 8443
Handling connection for 8443
Handling connection for 8443
Handling connection for 8443
Handling connection for 8443

And if we check the route we can open the keycloak landing page.

oc get routes --namespace keycloak
NAME                       HOST/PORT                                                    PATH   SERVICES             PORT    TERMINATION            WILDCARD
example-kc-ingress-76p8j   oeffrs.cunningham.aucunnin.lpi0.s1.devshift.org ... 1 more          example-kc-service   https   passthrough/Redirect   None

Kicking the tyres of 3scale API Management with the operator

Austin Cunningham — Fri, 17 Feb 2023 15:56:35 +0000

This is a shameless plug as I work on this product. This is a quick startup guide for getting 3scale up and running quickly with the 3scale-operator. This is not a production configuration and doesn't cover all configuration options.

Install with the 3scale-operator

First we create a project

oc new-project 3scale-test

In the Openshift Administrator view we can then use the Operatorhub to install the 3scale operator into the project

Once installed we can create an APIManager CR to create an instance of 3scale. To do this we need a wildCardDomain that is resolvable in Openshift. You can setup a custom domain or you can use the Openshift default application router which I will used. This can be got from your Openshift console route e.g.

console-openshift-console.apps.aucunnin.4mog.s1.devshift.org

You also need an s3 bucket setup. This bucket is used for storing CMS data for the customer web portal which I won't be covering in this blog. So once you have the bucket setup you will need its buck id , region and AWS key and secret. We create a secret in the project namespace

kind: Secret
apiVersion: v1
metadata:
  name: s3-credentials
  namespace: 3scale-test
data:
  AWS_ACCESS_KEY_ID: UkVQTEFDRV9NRQ==
  AWS_BUCKET: UkVQTEFDRV9NRQ==
  AWS_REGION: UkVQTEFDRV9NRQ==
  AWS_SECRET_ACCESS_KEY: UkVQTEFDRV9NRQ==
type: Opaque

the values are base64 encoded e.g.

echo UkVQTEFDRV9NRQ== | base64 -d
REPLACE_ME

Once the secret is created you can create the APIManager CR

apiVersion: apps.3scale.net/v1alpha1
kind: APIManager
metadata: 
  name: apimanager-sample
  namespace: 3scale-test
spec: 
  system: 
    fileStorage: 
      simpleStorageService: 
        configurationSecretRef: 
          name: s3-credentials
  wildcardDomain: apps.aucunnin.4mog.s1.devshift.org

Once the CR is created it should install 3scale. You can confirm the install is finished by checking the APIManager CR status , should look like the following.

# using https://mikefarah.gitbook.io/yq/
oc get apimanager apimanager-sample -oyaml | yq '.status'
conditions:
  - lastTransitionTime: "2023-02-17T14:29:27Z"
    status: "True"
    type: Available
deployments:
  ready:
    - apicast-production
    - apicast-staging
    - backend-cron
    - backend-listener
    - backend-redis
    - backend-worker
    - system-app
    - system-memcache
    - system-mysql
    - system-redis
    - system-sidekiq
    - system-sphinx
    - zync
    - zync-database
    - zync-que

Login to 3scale

Once we reach this point we can play around with 3scale. We can use the 3scale-admin route or the master route to log in the credentials for these are in the system-seed secret. I will use the admin route .

Route

Password

Click on the route and login as admin with the copied password and go through the welcome wizard.

Add your own API

I am going to use a free API to demo this

curl -k https://catfact.ninja/fact  
{"fact":"Both humans and cats have identical regions in the brain responsible for emotion.","length":81}%

Create a Backend

From the dashboard click on Create Backend and fill out the form adding a valid api.

Create a Product

From the dashboard click on Create Product and fill out the form

Add Application Plan

From the Product screen select Application and Application Plan and fill out the form, you can add thing like price plans, trial periods etc.

Add an Application

From the Product screen select Application and Listing and then the Create Application button and fill out the form again

Link your Backend to your Product

From the dashboard click on the product you created earlier, Click on Integration\Configuration and add the backend to your product.

Promote to staging and production

From the products screen click on Integration\Configuration again and click on the button to promote to staging and production

You can check the staging url out with the curl command e.g.

curl "https://catfact-3scale-apicast-staging.apps.aucunnin.4mog.s1.devshift.org:443/?user_key=073a4f36364b635c156811dcc1728d32"
{"fact":"Phoenician cargo ships are thought to have brought the first domesticated cats to Europe in about 900 BC.","length":105}%

That it for kicking the tyres , there is a lot more you can add to your product such as analytics, metrics, policies, API docs\ActiveDocs. There are plenty of other features but I won't go into them as I would just be covering what is already in the docs

Upgrading Postgresql-v10 to v13 on Openshift

Austin Cunningham — Fri, 17 Feb 2023 11:50:24 +0000

On Openshift v4 cluster I am looking to upgrade Postgresql-v10 to Postgersql-V13. For the sake of this blog I will spin up a Posgresql-v10 first via the Openshift templates

NOTE: I am using quay.io registry here, you can use the Red Hat certified images also

Can we just change the image in the deploymentConfig to quay.io/centos7/postgresql-13-centos7 ?

Unfortunately your postgresql pod will crash with the following error

Incompatible data directory. This container image provides
PostgreSQL '13', but data directory is of
version '10'.
This image supports automatic data directory upgrade from
'12', please _carefully_ consult image documentation
about how to use the '$POSTGRESQL_UPGRADE' startup option.

This does give us a hit on how to solve the issue but it looks like we need to upgrade through the versions of Postgresql. After a bit of reading about $POSTGRESQL_UPGRADE we have two options for this env var, copy and hardlink. Using copy seems to be the lowest risk upgrade. I found we can go from

Postgresql 10 - 12
Postgresql 12 - 13

NOTE: $POSTGRESQL_UPGRADE does come with the usually warnings about backing up your data before upgrading your DB. And you should be comfortable with restoring from a back up.

Postgresql 10 to 12

Before we can patch the deployment config image we need to remove the imageChange deployment config trigger as it will revert any image change

# remove all
oc set triggers dc/postgresql --remove-all
# add back in the config change trigger
oc set triggers dc/postgresql --from-config

We can then patch the image in the deployment config to v12 quay.io/centos7/postgresql-12-centos7 with this ugly one-liner patch command

oc patch dc postgresql -n postgresql-test --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"quay.io/centos7/postgresql-12-centos7"}]'

This will cause a new roll out.

NOTE: I use a lot of cli commands as I am looking to eventually automate this in a script. You can also achieved the same by manually editing the image in the deploymentConfig.

The roll out will have a pod in crashloopbackoff due to not having the $POSTGRESQL_UPGRADE env var set. We can set the environment variable in the deploymentConfig with the following command

oc set env dc/postgresql -n postgresql-test POSTGRESQL_UPGRADE=copy

This will cause another roll out. Occasionally I found that this upgrade can fail with the following error

pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2023-02-03 14:14:21.615 UTC [45] FATAL:  lock file "postmaster.pid" already exists

So the long and the short of it is there is a lock file in place in postgresql and we need to shut down the postgresql service correctly to remove it. I ran the following commands before I set the $POSTGRESQL_UPGRADE env var.

# get the non deployment pod name
POD=$(oc get po --all-namespaces | grep "postgresql-" | awk '{print $2}' | grep -wv deploy)
# first stop the postgres service note the paths used in the commands the postgresql database name 'userdata' in this case
oc exec -it $POD -c postgresql -n postgresql-test -- /usr/bin/pg_ctl stop -D /var/lib/pgsql/data/userdata
# then remove the lock file to be sure
oc exec -it $POD -c system-postgresql -n $THREESCALE_NS  -- rm /var/lib/pgsql/data/userdata/postmaster.pid

I found that once the upgrade is finished the $POSTGRESQL_UPGRADE env var being set can cause some problems with future roll outs of the deploymentConfig e.g.

== WARNING!! ==
PostgreSQL server version matches the datadir PG_VERSION.
The $POSTGRESQL_UPGRADE makes no sense and you probably
made some mistake, keeping the variable set you might
risk a data loss in future!
===============

So we must unset the $POSTGRESQL_UPGRADE env var

oc set env dc/postgresql -n postgresql-test POSTGRESQL_UPGRADE-

This will cause another roll out and that will be stable with v12 installed. To confirm the upgrade was successful you can run the following command

# run a postgres -V on the pod for postgresql, command finds the only pod that is not the deploy pod
oc exec -it $(oc get po --all-namespaces | grep "postgresql-" | awk '{print $2}' | grep -wv deploy) -n postgresql-test -- postgres -V
# should give you the version
postgres (PostgreSQL) 12.7

Postgresql 12 to 13

We can apply the same steps again for 12 to 13. First patch the image in the deploymentConfig to quay.io/centos7/postgresql-13-centos7 again with the patch command

oc patch dc postgresql -n postgresql-test --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"quay.io/centos7/postgresql-13-centos7"}]'

Again postgresql pod will crash with the following error

Incompatible data directory. This container image provides
PostgreSQL '13', but data directory is of
version '12'.
This image supports automatic data directory upgrade from
'12', please _carefully_ consult image documentation
about how to use the '$POSTGRESQL_UPGRADE' startup option.

set the environment variable with the following command

oc set env dc/postgresql -n postgresql-test POSTGRESQL_UPGRADE=copy

This will cause another roll out.

Again you need to unset the $POSTGRESQL_UPGRADE env var to avoid issues on future roll outs

oc set env dc/system-postgresql -n postgresql-test POSTGRESQL_UPGRADE-

This will cause another roll out and that will be stable with v13 installed.

# run a postgres -v on the none deployment pod for posgres
oc exec -it $(oc get po --all-namespaces | grep "postgresql-" | awk '{print $2}' | grep -wv deploy) -n postgresql-test -- postgres -V
# should give you the version
postgres (PostgreSQL) 13.3

All in all it took 7 changes to the deploymentConfig to successfully upgrade.

Keycloak 19.0.1 and Setting the id_token_hint

Austin Cunningham — Fri, 16 Sep 2022 20:37:04 +0000

Created a blog some time ago setting up keycloak express and OIDC client and it happily worked fine with keycloak-17.0.0 and still does. On keycloak-19.0.2 same code completely falls over. (Note to self always set prerequisites on dependencies in blogs).

First issue I needed to enable Standard flow for the client in keycloak as I was seeing the following error in the logs



2022-09-16 11:56:37,512 WARN  [org.keycloak.events] (executor-thread-0) type=LOGIN_ERROR, realmId=78ae1441-0616-4745-b021-5ca93fc9f779, clientId=null, userId=null, ipAddress=127.0.0.1, error=invalid_code
2022-09-16 11:58:56,673 ERROR [org.keycloak.services] (executor-thread-3) KC-SERVICES0095: Client is not allowed to initiate browser login with given response_type. Standard flow is disabled for the client.

It's in the Clients\Capability config section

Secondly I was using localhost for my uri's in keycloak and in my code base and keycloak-19 did not play well with it. Changed everything to 127.0.0.1 and most routes started working again. With the exception of logout. I was hitting the following issue.

The keycloak logs had a similar error with more details.



2022-09-16 13:00:28,208 WARN  [org.keycloak.events] (executor-thread-57) type=LOGOUT_ERROR, realmId=1bfc6eb7-ffa8-497f-96cc-69840b6c3f39, clientId=null, userId=null, ipAddress=127.0.0.1, error=invalid_request
2022-09-16 13:11:48,730 WARN  [org.keycloak.protocol.oidc.endpoints.LogoutEndpoint] (executor-thread-75) Either the parameter 'client_id' or the parameter 'id_token_hint' is required when 'post_logout_redirect_uri' is used.

So because I was using post_logout_redirect_uri I need to use either client_id or id_token_hint parameter. So I had three options

stop using post_logout_rediret_uri
add a client_id parameter to post_logout_redirect_uri
add a id_token_hint parameter to post_logout_redirect_uri

Stop using post_logout_redirect_uri

Remove it from the keycloakeIssuer.Client



const keycloakIssuer = await Issuer.discover("http://127.0.0.1:8080/realms/keycloak-express")
// I just comment out the line in the client 
const client = new keycloakIssuer.Client({
    client_id: 'keycloak-express',
    client_secret: 'long_secret-here',
    redirect_uris: ['http://127.0.0.1:3000/auth/callback'],
    //post_logout_redirect_uris: ['http://127.0.0.1:3000/logout/callback'],
    response_types: ['code'],
  });

Looks like this it prompts for a logout and leaves you at a keycloak logged out screen.

Add client_id parameter

We have set the client_id in the keycloakeIssuer.Client so it just a matter of setting it in the logout as a parameter



// start logout request
app.get('/logout', (req, res) => {
    res.redirect(client.endSessionUrl({
        client_id: "keycloak-express"
    }
    ));
});

Looks like this as you can see it ask for another confirmation before redirecting to the app.

Add id_token_hint parameter

What is id_token_hint and how do I populate it?
Me I am terrible at reading/comprehending documentation and much prefer a good example. In this case I didn't find any examples. I found the following documentation referencing id_token_hint

OpenID Connect RP-Initiated Logout which I found too abstract to formulate what to do to generating id_token_hint or find it.
the node-oidc-client gave me a hint

There is a tokenSet object that is created as part of the passportjs strategy login flow. I inspected this object and found that it has a id_token and I made this id_token_hint = id_token and my issue was solved.

So I was using passwordjs and node-openid-client



var TokenSet
passport.use('oidc', new Strategy({client}, (tokenSet, userinfo, done)=>{
        TokenSet = tokenSet;
        return done(null, tokenSet.claims());
    })
)
//And on logout we can set the id_token_hint parameter
app.get('/logout', (req, res) => {
    res.redirect(client.endSessionUrl({
        id_token_hint: TokenSet.id_token
    }
    ));
});

Looks like this and is a much better user experience

Code lives here

DEV Community: Austin Cunningham

Building an AI-Powered Log Analyser with RAG

Prerequisites

Project Setup

Core Architecture

Custom Log Loader (utils/loaders.py)

Set the Type of Files to Find

Processing Files Line by Line

Main Application (main.py)

Imports & Dependencies

Directory Input & Validation

Document Loading & Processing

Vector Database Creation (The Heavy Lifting)

RAG Chain Setup

Interactive Query Loop & Prompt Engineering

Response Display & Source Attribution

How It Works

Step 1: Document Processing

Step 2: Vector Database Creation

Build an Ollama LLM software engineering language bot

What

Prerequisites

Scaffold

Backend

Add the chat bot post request

Frontend

Start the app to test

Deploying Ollama on Openshift with the Ollama Operator

Why Run Ollama on OpenShift?

What is the Ollama Operator?

Install the Operator

Run the Quickstart Script

Test the Ollama Service

Port Forward the Ollama Service

Test the /api/tags Endpoint

Test the /api/generate Endpoint

Conclusion and Next Steps

AI-Powered Kubernetes Debugging with Python and Ollama

What It Does

Requirements

Install dependencies

Install and Start Ollama

How It Works

Finding Failing Pods

Collect Diagnostics

Analyze with Ollama

Bring it all together

Running the script

Code lives here

Build a Local AI RAG App with Ollama and Python

utils/loaders.py

main.py

Monitor internet uptime on your local PC

Load testing 3scale

What do we need?

What does it give us?

Pain Points

Getting started

So how do you know it's working?

Automate 3scale monitoring stack deployment

Install Keycloak with the Operator

Install the Keycloak via the operator

Install an DB

Hostname

Create a DB secret

Create the Keycloak CR

Kicking the tyres of 3scale API Management with the operator

Install with the 3scale-operator

Login to 3scale

Add your own API

Create a Backend

Create a Product

Add Application Plan

Add an Application

Link your Backend to your Product

Promote to staging and production

Upgrading Postgresql-v10 to v13 on Openshift

Postgresql 10 to 12

Postgresql 12 to 13

Keycloak 19.0.1 and Setting the id_token_hint

Custom Log Loader (`utils/loaders.py`)

Main Application (`main.py`)