DEV Community: DevOps VN

Are You a Software Engineer, or Just an AI Content Paster?

DevOps VN — Wed, 09 Jul 2025 14:59:49 +0000

Evolving from AI Paster to True Developer

As an engineer who has been in the industry for a while, I’ve seen my share of game-changing technologies. The current wave of AI coding assistants is perhaps the most transformative yet. They offer a level of productivity that was unimaginable just a few years ago. But for those of you just starting your journey as a developer, this incredible tool presents a hidden trap.

I see a lot of junior engineers fall into it. They use AI as a magic black box, a crutch to generate code without truly understanding the “why” behind it. While it might help you close a ticket today, this approach will stunt your growth and limit your potential tomorrow.

The real craft of software engineering isn’t about producing lines of code; it’s about deep problem-solving, architectural thinking, and building robust, elegant solutions. If you outsource that thinking to an AI too early in your career, you’re not building the mental muscles you need to thrive.

So, how do you use these powerful tools to become a better engineer, not just a faster one? It’s about shifting your mindset from being a passive user to an active, critical collaborator with your AI.

The Pitfall: When Convenience Kills Competence

Let’s be honest: when you’re facing a complex problem and a looming deadline, asking an AI to “just fix it” is incredibly tempting. But this habit of seeking the quick fix can lead to some serious long-term problems:

Surface-Level Understanding: You pick up the basics, like syntax, but miss the deeper meaning. True mastery comes from wrestling with a problem, hitting dead ends, and finally getting that “aha!” moment. Skipping the struggle means skipping real learning.
Overtrusting AI: It’s tempting to think AI’s output is flawless. I’ve seen this cause sneaky bugs, clunky code, or even security risks in projects because the initial result “seemed fine.”
Weakened Problem-Solving: Your greatest strength as a developer is thinking critically and creatively. Leaning on AI for every solution is like having someone else do your workout — you won’t grow stronger.

The Strategy: Use AI to Support, Not Replace

The goal isn’t to avoid AI. It’s to use it strategically. Instead of treating your AI assistant like someone who does all the work for you, see it as a teammate who pushes you to improve.

1. Let AI Be Your Teacher

Don’t just ask for code — ask AI to explain it. This helps you learn better.

“Break down this code like I’m new to programming.”
“How does this method compare to another way of doing it?”
“Why pick this data structure? How does it affect speed?”

Make AI explain things clearly. This turns copying code into a real learning experience.

2. Always Check AI’s Work

Never, ever trust AI-generated code implicitly. Review every line carefully.

Go through each line. Do you get what it does? Can you explain why it’s there?
Question if it’s clear and efficient. Does it fit your team’s coding style? Is there a better way?
Write your tests. Testing AI’s code helps you understand it and find hidden problems

3. Know When to Work Without AI

For learning key concepts, like basic algorithms or new patterns, try solving them yourself first. The struggle helps you grow.

Use AI for tasks that speed things up without skipping learning, like writing repetitive code, creating test data, or tweaking code you already know well.

Stay Curious and Keep Growing

By being intentional about how you use these tools, you can avoid the pitfalls and unlock their true potential. Use AI to spark your curiosity and boost your skills, but don’t let it do all the thinking for you. Your brain is your best tool. Now, build something awesome!

This article is based on concepts from my book “PromptOps: From YAML to AI”, a comprehensive guide to leveraging AI for DevOps workflows. The book covers everything from basic prompt engineering to building team-wide AI-assisted practices, with real-world examples for Kubernetes, CI/CD, cloud infrastructure, and more.

Want to dive deeper? The full book includes:
Advanced prompt patterns for every DevOps domain
Team collaboration strategies for AI-assisted workflows
Security considerations and validation techniques
Case studies from real infrastructure migrations
A complete library of reusable prompt templates

Follow me for more insights on AI-driven DevOps practices, or connect with me to discuss how these techniques can transform your infrastructure workflows.

I Generated Production-Ready Kubernetes Configs in 30 Seconds (Here's How You Can Too)

DevOps VN — Thu, 03 Jul 2025 10:41:37 +0000

The 5-letter framework that turned my AI from a glorified search engine into a senior DevOps engineer

I've seen it hundreds of times. A DevOps engineer opens ChatGPT, types "write a Kubernetes deployment," gets a basic YAML file, and then spends the next hour manually fixing security issues, adding resource limits, and making it production-ready.

Sound familiar?

Here's the thing: The AI isn't the problem. Your prompt is.

After working with AI tools for infrastructure automation for the past two years, I've discovered that the difference between getting generic, unusable output and getting production-ready code comes down to one thing: how you ask for it.

Today, I want to share the exact framework that transformed my DevOps workflow and helped me generate infrastructure code that I trust to deploy.

The Problem with How We Prompt AI

Most technical professionals treat AI like Google Search. We throw in a few keywords and hope for the best:

"Create a Dockerfile for Python"
"Write a backup script"
"Make a CI/CD pipeline"

But here's what we're doing: We're asking a highly sophisticated AI assistant to read our minds. And when it inevitably fails to deliver exactly what we need, we blame the AI.

The reality? AI isn't mind-reading. It's pattern matching. And the patterns it matches are entirely dependent on the information you provide.

Enter the C.R.A.F.T. Framework

After analyzing hundreds of successful AI interactions for DevOps tasks, I developed a simple framework that consistently delivers professional-grade results. I call it C.R.A.F.T.:

Context: Provide the background and current situation
Role: Assign a job title or persona to the AI
Action: What specific thing do you want the AI to do
Format: What should the final output look like
Tone: What style should the AI use in its response

Let me show you how dramatically this changes your results.

The Before and After That Will Blow Your Mind

❌ The Bad Prompt:

"Make a Kubernetes deployment for Nginx."

✅ The Good Prompt (Using C.R.A.F.T.):

(Role) Act as a certified Kubernetes administrator.

(Context) I have a standard Kubernetes cluster on GKE. I need to deploy 
a simple Nginx web server that will serve as a reverse proxy for a 
Node.js application running on port 8080.

(Action) Generate the YAML for a Kubernetes Deployment and a Service.

(Format) The Deployment should use the official nginx:latest image, 
have 3 replicas, and include readiness and liveness probes. The Service 
should be of type LoadBalancer and expose port 80.

(Tone) Add comments to the YAML explaining what each major section does.

The difference in output quality is night and day.

The first prompt gives you a basic deployment that's missing:

Resource limits
Health checks
Security contexts
Proper labeling
Service configuration
Any real-world considerations

The second prompt delivers a complete, production-ready configuration with security best practices, proper resource management, and comprehensive documentation.

Why Context Is Your Secret Weapon

The Context component is where most people fail, but it's also where you can create the biggest impact. Here's what game-changing context looks like:

🎯 Include Your "Why"

Instead of: "Create a firewall rule"
Try: "I need to open port 5432 to allow our new analytics service to connect to the production PostgreSQL database. Security is critical."

🔧 Specify Your Tech Stack

Cloud Provider: AWS
CI/CD System: GitHub Actions  
IaC Tools: Terraform v1.5
Runtime: Python 3.11, Node.js 18

📋 Define Your Constraints

"Must run as non-root user"
"All S3 buckets need encryption enabled"
"Memory-efficient for small container instances"
"Follow PEP 8 style guidelines"

📊 Show Data Structures

If you're working with JSON, YAML, or databases, show the AI exactly what format you're dealing with.

The Role Revolution

Here's something most people don't realize: AI models have been trained on millions of examples of how different professionals write code.

When you tell the AI to "Act as a Senior Site Reliability Engineer," you're not just giving it a title—you're activating an entire knowledge pattern of how SREs think about:

Security
Scalability
Monitoring
Error handling
Best practices

Compare these two Dockerfile requests:

Generic: "Create a Dockerfile for a Python app"
Role-Based: "Act as a Senior Site Reliability Engineer. Create a Dockerfile for a production Python web application."

The second one automatically includes:

Multi-stage builds
Non-root user configuration
Optimized image layers
Security scanning considerations
Production-ready configurations

Action Words That Actually Work

Stop saying "help me with" or "can you." Start using precise action verbs:

Generate (for new code/configs)
Refactor (for improving existing code)
Debug (for troubleshooting)
Explain (for understanding)
Optimize (for performance improvements)
Compare (for evaluating options)

Format: Get Exactly What You Need

The AI can output in virtually any format, but you have to ask:

"Provide as numbered bash commands"
"Output as Terraform HCL"
"Format as a Markdown table"
"Generate both Dockerfile and docker-compose.yml"
"Include comprehensive comments"

Real-World Results

Since implementing C.R.A.F.T., I've:

✅ Reduced my infrastructure code review cycles by 60%
✅ Generated production-ready Terraform modules in minutes instead of hours
✅ Created comprehensive CI/CD pipelines with proper error handling and security scanning
✅ Built monitoring dashboards that actually caught real issues
✅ Automated backup scripts that handle edge cases I didn't even think of

More importantly, I trust the code that comes out of these prompts enough to deploy it (after proper testing, of course).

Your Next Steps

Start with Context: Next time you prompt an AI, spend 30 seconds providing proper context. Include your environment, constraints, and the "why" behind your request.
Assign Roles: Always tell the AI what kind of professional perspective you want. "Act as a DevOps engineer" vs "Act as a security specialist" will give you dramatically different outputs.
Be Specific: Replace vague requests with precise actions and format requirements.
Iterate: Don't settle for the first output. Ask follow-up questions, request modifications, and refine until it's exactly what you need.

The Future Is Conversational Infrastructure

We're moving from "Infrastructure as Code" to what I call "Infrastructure as Conversation." The engineers who master this shift—who learn to direct AI effectively rather than just hoping for good results—will be the ones building the future.

The C.R.A.F.T. framework isn't just about getting better AI outputs. It's about fundamentally changing how you work. It's about spending your time on architecture, strategy, and creative problem-solving, rather than wrestling with YAML syntax and boilerplate code.

This article is based on concepts from my upcoming book "PromptOps: From YAML to AI" - a comprehensive guide to leveraging AI for DevOps workflows. The book covers everything from basic prompt engineering to building team-wide AI-assisted practices, with real-world examples for Kubernetes, CI/CD, cloud infrastructure, and more.

Want to dive deeper? The full book includes:

Advanced prompt patterns for every DevOps domain
Team collaboration strategies for AI-assisted workflows
Security considerations and validation techniques
Case studies from real infrastructure migrations
A complete library of reusable prompt templates

Follow me for more insights on AI-driven DevOps practices, or connect with me to discuss how these techniques can transform your infrastructure workflows.

A step-by-step guide to building a personalized AI assistant from scratch using Python

DevOps VN — Tue, 24 Jun 2025 09:11:31 +0000

Have you ever wished for a personal assistant who just gets you? I’m not talking about a generic chatbot that can tell you the weather. I mean something that knows what projects you’re working on, remembers your goals, and can instantly find that one brilliant note you scribbled down weeks ago and forgot about.

What if you could build that assistant yourself?

That’s exactly what we’re going to do together. In this book, we're not just going to learn about AI; we’re going to roll up our sleeves and create one from scratch. We will build a custom AI assistant that runs on your computer, learns from your data, and becomes a "second brain" that is uniquely yours. Imagine being able to ask your computer, "What were my main takeaways from that article I read last Tuesday?" and getting a real, helpful answer. That’s the goal.

It might sound impossibly complicated, but I promise you, it's not. We can boil this whole project down to three simple, understandable parts:

The Brain: This is the powerful “engine” that does the thinking. We’ll use a ready-made Large Language Model (LLM), so we don’t have to build the engine ourselves, just learn how to use it.
The Memory: This is where your AI will store everything it learns about you. We'll set up a special kind of digital filing cabinet to hold your notes, ideas, and facts, making them easy for the AI to find.
The Connectors: This is our Python code. Think of it as the glue that connects the Brain to the Memory, allowing them to talk to each other and work together to answer your questions.

And here’s the best part: you don't need to be an AI wizard to do this. In fact, you don't need any AI experience at all. If you know the basics of Python, you have everything it takes to get started. We'll skip the heavy math and dense theory and focus on the fun part: building something amazing.

So, let’s get your workshop set up. By the end of this journey, you won't just have a new skill—you'll have your very own AI assistant, built by you, for you.

Part 1: The First Spark of Intelligence

Chapter 1: Your First Conversation with an AI
Chapter 2: Building the Chatbot Shell

Part 2: Giving Your Assistant a Memory

Chapter 3: Giving Your AI Its First Memory
Chapter 4: The Memory Bank - Your First Vector Database
Chapter 5: Retrieval - Finding the Right Memory

Part 3: Connecting the Brain to the Memory

Chapter 6: The RAG Pipeline - Thinking with Your Memories

Part 4: Growing Your Assistant

Chapter 7: The Automatic Learner - Ingesting Data Daily
Chapter 8: A Friendly Face - Building a Simple Web UI

Part 5: The Responsible AI Owner

Chapter 9: Privacy, Security, and Future Steps

Read full series here: Building Your AI Assistant.

How to handle alerts from various tools such as Grafana, Kibana, Sentry, AWS, etc.

DevOps VN — Sat, 05 Apr 2025 17:50:52 +0000

In today’s DevOps world, managing alerts from multiple monitoring tools can quickly become difficult. Tools like Grafana, Kibana, Sentry, and AWS CloudWatch each provide important updates about system health, but their different alert formats and notification methods.

Imagine having to set up separate webhooks for each tool, dealing with inconsistent message layouts, and manually sending alerts to the right team—all while trying to fix a production issue quickly. It’s a tough situation.

Instead of dealing with multiple webhook endpoints and custom scripts, let's use Versus to act as a central hub, turning alerts into a standard format and delivering them where your team already works.

Step 1: Create a Unified Alert Template

Versus uses Go templates to format incoming alerts. This lets us make one template that works for different alert sources based on their data structure. Here’s an example template for AWS, Grafana, Kibana, and Sentry:

{{/* Alert from AWS */}}
{{ if .source }}
  {{ if eq .source "aws.glue" }}
  🚨 Job: {{.detail.jobName}} run failed
  {{ else if eq .source "aws.ec2" }}
  🚨 Instance: {{.detail.instance-id}} terminating
  {{ end }}

{{/* Alert from Grafana */}}
{{ else if .receiver }}
  {{ if eq .receiver "payment-team" }}
  🔥 Transaction {{ .alerts[0].annotations.summary }} failed, please check!
  {{ else if eq .receiver "devops-team" }}
  🚨 Node {{ .alerts[0].annotations.summary }} down
  {{ end }}

{{/* Alert from Kibana */}}
{{ else if .kibanaUrl }}
❌ Kibana Alert: {{.name}}

Message: {{.message}}
Kibana URL: <{{.kibanaUrl}}|View in Kibana>

{{/* Alert from Sentry */}}
{{ else }}
🚨 Sentry Alert: {{.data.issue.title}}

Project: {{.data.issue.project.name}}
Issue URL: {{.data.issue.url}}
{{ end }}

AWS Alerts: Looks for a .source field (e.g., aws.glue or aws.ec2) and pulls out details like job names or instance IDs.
Grafana Alerts: Uses the .receiver field to send alerts to specific teams (e.g., payment-team or devops-team) and grabs data from the alert annotations.
Kibana Alerts: Spots alerts by the .kibanaUrl field and adds a clickable link.
Sentry Alerts: Uses a default case to pull issue details from the .data.issue object.

Save this template as config/slack_message.tmpl in a local config directory.

Step 2: Configure Versus

Next, create a config/config.yaml file to tell Versus how to run and connect to Slack:

name: versus
host: 0.0.0.0
port: 3000

alert:
  slack:
    enable: true
    token: ${SLACK_TOKEN}         # Your Slack Bot OAuth Token
    channel_id: ${SLACK_CHANNEL_ID} # Your Slack channel ID
    template_path: "/app/config/slack_message.tmpl" # Path inside the container

  telegram:
    enable: false

  msteams:
    enable: false

Step 3: Deploy Versus with Docker

Run Versus using Docker, attaching the config directory and adding the Slack details:

docker run -d \
  -p 3000:3000 \
  -v $(pwd)/config:/app/config \
  -e SLACK_ENABLE=true \
  -e SLACK_TOKEN=your_slack_token \
  -e SLACK_CHANNEL_ID=your_channel_id \
  --name versus \
  ghcr.io/versuscontrol/versus-incident

After starting, Versus will listen for webhook POST requests at http://localhost:3000/api/incidents.

Step 4: Expose Versus with ngrok (Optional)

To test with external tools, use ngrok to make your local Versus instance available online:

ngrok http 3000 --url your-versus-https-url.ngrok-free.app

This gives you a public URL (e.g., https://your-versus-https-url.ngrok-free.app). The Versus API will be at https://your-versus-https-url.ngrok-free.app/api/incidents. Save this URL for setting up webhooks.

Step 5: Set Up Webhooks in Your Tools

Now, connect each tool’s webhook to the Versus endpoint. Here’s how:

AWS (SNS)

Create an SNS topic in AWS and add an HTTPS subscription: https://your-versus-https-url.ngrok-free.app/api/incidents.
Send a test message:

  aws sns publish \
    --topic-arn your_sns_topic_arn \
    --message '{"source": "aws.glue", "detail": {"jobName": "ETLJob1"}}'

Grafana

In Grafana, go to Alerting > Notification Channels > New Channel.
Choose Webhook, set the URL to https://your-versus-https-url.ngrok-free.app/api/incidents, and use this JSON:

  {
    "receiver": "devops-team",
    "alerts": [{"annotations": {"summary": "Server1"}}]
  }

Kibana

In Kibana, go to Stack Management > Alerts and Insights > Rules > Create Rule.
Add a Webhook action with the URL https://your-versus-https-url.ngrok-free.app/api/incidents and this body:

  {
    "name": "High CPU Usage",
    "message": "CPU exceeds 90%",
    "kibanaUrl": "http://kibana.example.com/app/alerts/123"
  }

Sentry

In Sentry, go to Project Settings > Alerts > Create Alert Rule.
Add a Webhook action with the URL https://your-versus-https-url.ngrok-free.app/api/incidents and this payload:

  {
    "data": {
      "issue": {
        "title": "Null Pointer Exception",
        "project": {"name": "Backend"},
        "url": "https://sentry.io/issues/123"
      }
    }
  }

Step 6: Test and Check

Trigger alerts from each tool and look at your Slack channel. You should see messages like:

AWS: 🚨 Job: ETLJob1 run failed
Grafana: 🚨 Node Server1 down
Kibana: ❌ Kibana Alert: High CPU Usage\nMessage: CPU exceeds 90%\nKibana URL: <http://kibana.example.com/app/alerts/123|View in Kibana>
Sentry: 🚨 Sentry Alert: Null Pointer Exception\nProject: Backend\nIssue URL: https://sentry.io/issues/123

Scaling and Improving the Setup

More Channels: Turn on Telegram or Microsoft Teams in config.yaml and add matching templates (e.g., telegram_message.tmpl).
On-Call Support: Enable AWS Incident Manager by adding oncall: enable: true and an awsim_response_plan_arn in the config.
Production Use: Run Versus on a cloud service (e.g., AWS ECS) instead of localhost for better reliability.

Conclusion

Versus turns a confusing mix of alerts into a clear. Its Go templates and webhook support make it a great tool for DevOps teams using multiple monitoring solutions. Whether you’re tracking AWS resources, checking metrics in Grafana, reviewing logs in Kibana, or catching errors in Sentry, Versus ensures your team gets the right details in a consistent format, right when they need.

How to Configure Grafana to Send Alerts to Slack and Telegram

DevOps VN — Tue, 25 Mar 2025 14:01:24 +0000

This guide leverages Versus Incident as an intermediary to route Grafana alerts to both Slack and Telegram with customizable messages, using a real-world JSON payload from Grafana.

Grafana is a powerful platform for monitoring and observability, with robust alerting capabilities. By integrating it with Versus Incident, you can extend its notification options to Slack and Telegram with tailored messages.

In this guide, we’ll configure Grafana to send alerts to Versus Incident based on a sample JSON payload, then forward them to Slack and Telegram.

Prerequisites

A running Grafana instance (version 8.0+ recommended for unified alerting) connected to a data source (e.g., Prometheus).
A Slack workspace with permission to create a bot and obtain a token.
A Telegram account with a bot created via BotFather and a chat ID for your target group or channel.
Docker installed (optional, for easy Versus Incident deployment).

Step 1: Set Up Slack and Telegram Bots

Slack Bot

Visit api.slack.com/apps and click Create New App.
Name your app (e.g., “Grafana Alerts”) and select your Slack workspace.
Under Bot Users, add a bot (e.g., “GrafanaBot”) and enable it.
Go to OAuth & Permissions, add the chat:write scope under Scopes.
Install the app to your workspace and copy the Bot User OAuth Token (starts with xoxb-). Save it securely.
Invite the bot to your Slack channel by typing /invite @GrafanaBot and note the channel ID (right-click the channel, copy the link, and extract the ID).

Telegram Bot

Open Telegram, search for BotFather, and start a chat.
Type /newbot and follow the prompts to name your bot (e.g., “GrafanaAlertBot”).
BotFather will provide a Bot Token (e.g., 123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11). Save it securely.
Create a group or channel, add your bot, and get the Chat ID.

Step 2: Deploy Versus with Slack and Telegram Enabled

Versus Incident will process Grafana’s JSON payload and route alerts to Slack and Telegram.

Create a configuration directory:

   mkdir -p ./config

Create config/config.yaml with the following content:

   name: versus
   host: 0.0.0.0
   port: 3000

   alert:
     slack:
       enable: true
       token: ${SLACK_TOKEN}
       channel_id: ${SLACK_CHANNEL_ID}
       template_path: "/app/config/slack_message.tmpl"

     telegram:
       enable: true
       bot_token: ${TELEGRAM_BOT_TOKEN}
       chat_id: ${TELEGRAM_CHAT_ID}
       template_path: "/app/config/telegram_message.tmpl"

Create a Slack template at config/slack_message.tmpl, tailored to the JSON structure:

   🚨 *Grafana Alert: {{.alerts.[0].labels.alertname}}*

   **Message**: {{.message}}
   **Status**: {{.status}}
   **Instance**: {{.alerts.[0].labels.instance}}
   **Severity**: {{.alerts.[0].labels.severity}}
   **Grafana URL**: <{{.alerts.[0].generatorURL}}|View in Grafana>

   Please investigate this issue.

Create a Telegram template at config/telegram_message.tmpl (using HTML formatting):

   🚨 <b>Grafana Alert: {{.alerts.[0].labels.alertname}}</b>

   <b>Message</b>: {{.message}}
   <b>Status</b>: {{.status}}
   <b>Instance</b>: {{.alerts.[0].labels.instance}}
   <b>Severity</b>: {{.alerts.[0].labels.severity}}
   <b>Grafana URL</b>: <a href="{{.alerts.[0].generatorURL}}">View in Grafana</a>

   Please investigate this issue.

Deploy Versus using Docker:

   docker run -d \
     -p 3000:3000 \
     -v $(pwd)/config:/app/config \
     -e SLACK_TOKEN="your_slack_bot_token" \
     -e SLACK_CHANNEL_ID="your_slack_channel_id" \
     -e TELEGRAM_BOT_TOKEN="your_telegram_bot_token" \
     -e TELEGRAM_CHAT_ID="your_telegram_chat_id" \
     --name versus \
     ghcr.io/versuscontrol/versus-incident

Replace your_slack_bot_token and your_slack_channel_id with Slack values.
Replace your_telegram_bot_token and your_telegram_chat_id with Telegram values.

The Versus Incident API endpoint is available at:

   http://localhost:3000/api/incidents

Use ngrok to expose it to the internet:

   ngrok http 3000 --url your-versus-https-url.ngrok-free.app

Step 3: Configure Grafana Alerts with a Webhook

Grafana’s unified alerting system sends JSON payloads like the one provided. We’ll configure a webhook to send this data to Versus Incident.

Log in to Grafana and go to Alerting > Alert rules.
Create a new alert rule:
- Name: “HighCPUUsage”.
- Data source: Select your data source (e.g., Prometheus) and define a query (e.g., cpu_usage > 90).
- Condition: Set a threshold, such as “value is above 90” over the last 5 minutes.
- Evaluation: Check every 1 minute.

Add a notification channel or contact point:

Type: Webhook.
URL: https://your-versus-https-url.ngrok-free.app/api/incidents.
HTTP Method: POST.
Headers: Add Content-Type: application/json.
Body: Grafana automatically sends a JSON payload like this:

 {
   "receiver": "webhook",
   "status": "firing",
   "alerts": [
     {
       "status": "firing",
       "labels": {
         "alertname": "HighCPUUsage",
         "instance": "server-01",
         "severity": "critical",
         "team": "ops"
       },
       "annotations": {
         "summary": "CPU usage is above 90% on server-01",
         "description": "The CPU usage on server-01 has exceeded 90% for more than 5 minutes."
       },
       "startsAt": "2025-03-25T06:25:00.000Z",
       "endsAt": "0001-01-01T00:00:00Z",
       "generatorURL": "http://grafana.example.com/alerting/1/view",
       "fingerprint": "a1b2c3d4e5f6g7h8"
     }
   ],
   "groupLabels": {
     "alertname": "HighCPUUsage"
   },
   "commonLabels": {
     "alertname": "HighCPUUsage",
     "severity": "critical"
   },
   "commonAnnotations": {
     "summary": "CPU usage is above 90%"
   },
   "externalURL": "http://grafana.example.com/",
   "version": "1",
   "groupKey": "{}:{alertname=\"HighCPUUsage\"}",
   "truncatedAlerts": 0,
   "orgId": 1,
   "title": "[Firing:1] HighCPUUsage (critical)",
   "state": "alerting",
   "message": "CPU usage is above 90% on server-01"
 }

No custom body is needed, as Grafana’s default payload matches Versus Incident’s expectations when paired with the templates.

Save the alert rule and notification settings.

Step 4: Test the Integration

Simulate a Grafana alert using curl with the provided JSON:

curl -X POST http://localhost:3000/api/incidents \
  -H "Content-Type: application/json" \
  -d '{
    "receiver": "webhook",
    "status": "firing",
    "alerts": [
      {
        "status": "firing",
        "labels": {
          "alertname": "HighCPUUsage",
          "instance": "server-01",
          "severity": "critical",
          "team": "ops"
        },
        "annotations": {
          "summary": "CPU usage is above 90% on server-01",
          "description": "The CPU usage on server-01 has exceeded 90% for more than 5 minutes."
        },
        "startsAt": "2025-03-25T06:25:00.000Z",
        "endsAt": "0001-01-01T00:00:00Z",
        "generatorURL": "http://grafana.example.com/alerting/1/view",
        "fingerprint": "a1b2c3d4e5f6g7h8"
      }
    ],
    "groupLabels": {
      "alertname": "HighCPUUsage"
    },
    "commonLabels": {
      "alertname": "HighCPUUsage",
      "severity": "critical"
    },
    "commonAnnotations": {
      "summary": "CPU usage is above 90%"
    },
    "externalURL": "http://grafana.example.com/",
    "version": "1",
    "groupKey": "{}:{alertname=\"HighCPUUsage\"}",
    "truncatedAlerts": 0,
    "orgId": 1,
    "title": "[Firing:1] HighCPUUsage (critical)",
    "state": "alerting",
    "message": "CPU usage is above 90% on server-01"
  }'

Alternatively, trigger a real alert in Grafana and verify the notifications in Slack and Telegram.

Conclusion

By integrating Grafana with Versus Incident, you can send detailed alerts to Slack and Telegram using Grafana’s native JSON payload. The setup is flexible, allowing you to customize templates to extract fields like alertname, instance, and severity.

Versus Incident also supports additional channels (e.g., Microsoft Teams, Email) and integrations like AWS Incident Manager. Check the Versus Incident documentation for more options to enhance your alerting workflow.

The Open-Source On-Call Integration

DevOps VN — Sat, 22 Mar 2025 16:37:47 +0000

This document provides a step-by-step guide to integrate Versus Incident with AWS Incident Manager to make an On Call. The integration enables automated escalation of alerts to on-call teams when incidents are not acknowledged within a specified time.

Prerequisites

Before you begin, ensure you have:

An AWS account with access to AWS Incident Manager.
Versus Incident deployed (instructions provided later).
Prometheus Alert Manager set up to monitor your systems.

Setting Up AWS Incident Manager for On-Call

AWS Incident Manager requires configuring several components to manage on-call workflows. Let’s configure a practical example using 6 contacts, two teams, and a two-stage response plan. Use the AWS Console to set these up.

Contacts

Contacts are individuals who will be notified during an incident.

In the AWS Console, navigate to Systems Manager > Incident Manager > Contacts.
Click Create contact.
For each contact:
Enter a Name (e.g., "Natsu Dragneel").
Add Contact methods (e.g., SMS: +1-555-123-4567, Email: natsu@devopsvn.tech).
Save the contact.

Repeat to create 6 contacts (e.g., Natsu, Zeref, Igneel, Gray, Gajeel, Laxus).

Escalation Plan

An escalation plan defines the order in which contacts are engaged.

Go to Incident Manager > Escalation plans > Create escalation plan.
Name it (e.g., TeamA_Escalation).
Add contacts (e.g., Natsu, Zeref, and Igneel) and set them to engage simultaneously or sequentially.
Save the plan.
Create a second plan (e.g., TeamB_Escalation) for Gray, Gajeel, and Laxus.

RunBook (Optional)

RunBooks automate incident resolution steps. For this guide, we’ll skip RunBook creation, but you can define one in AWS Systems Manager Automation if needed.

Response Plan

A response plan ties contacts and escalation plans into a structured response.

Go to Incident Manager > Response plans > Create response plan.
Name it (e.g., CriticalIncidentResponse).
Define two stages:
Stage 1: Engage TeamA_Escalation (Natsu, Zeref, and Igneel) with a 5-minute timeout.
Stage 2: If unacknowledged, engage TeamB_Escalation (Gray, Gajeel, and Laxus).
Save the plan and note its ARN (e.g., arn:aws:ssm-incidents::111122223333:response-plan/CriticalIncidentResponse).

Define IAM Role for Versus

Versus needs permissions to interact with AWS Incident Manager.

In the AWS Console, go to IAM > Roles > Create role.
Choose AWS service as the trusted entity and select EC2 (or your deployment type, e.g., ECS).
Attach a custom policy with these permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm-incidents:StartIncident",
                "ssm-incidents:GetResponsePlan"
            ],
            "Resource": "*"
        }
    ]
}

Name the role (e.g., VersusIncidentRole) and create it.
Note the Role ARN (e.g., arn:aws:iam::111122223333:role/VersusIncidentRole).

Deploy Versus Incident

Deploy Versus using Docker or Kubernetes. Docker Deployment. Create a directory for your configuration files:

mkdir -p ./config

Create config/config.yaml with the following content:

name: versus
host: 0.0.0.0
port: 3000
public_host: https://your-ack-host.example

alert:
  debug_body: true

  slack:
    enable: true
    token: ${SLACK_TOKEN}
    channel_id: ${SLACK_CHANNEL_ID}
    template_path: "config/slack_message.tmpl"

oncall:
  enable: true
  wait_minutes: 3

  aws_incident_manager:
    response_plan_arn: ${AWS_INCIDENT_MANAGER_RESPONSE_PLAN_ARN}

redis: # Required for on-call functionality
  insecure_skip_verify: true # dev only
  host: ${REDIS_HOST}
  port: ${REDIS_PORT}
  password: ${REDIS_PASSWORD}
  db: 0

Create Slack templates config/slack_message.tmpl:

🔥 *{{ .commonLabels.severity | upper }} Alert: {{ .commonLabels.alertname }}*

🌐 *Instance*: `{{ .commonLabels.instance }}`  
🚨 *Status*: `{{ .status }}`

{{ range .alerts }}
📝 {{ .annotations.description }}  
{{ end }}
{{ if .AckURL }}
----------
<{{.AckURL}}|Click here to acknowledge>
{{ end }}

ACK URL Generation

When an incident is created (e.g., via a POST to /api/incidents), Versus generates an acknowledgment URL if on-call is enabled.
The URL is constructed using the public_host value, typically in the format: https://your-host.example/api/incidents/ack/<incident-id>.
This URL is injected into the alert data as a field (e.g., .AckURL) and becomes available for use in templates.

Create the docker-compose.yml file:

version: '3.8'

services:
  versus:
    image: ghcr.io/versuscontrol/versus-incident
    ports:
      - "3000:3000"
    environment:
      - SLACK_TOKEN=your_slack_token
      - SLACK_CHANNEL_ID=your_channel_id
      - AWS_INCIDENT_MANAGER_RESPONSE_PLAN_ARN=arn:aws:ssm-incidents::111122223333:response-plan/CriticalIncidentResponse
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      - REDIS_PASSWORD=your_redis_password
    depends_on:
      - redis

  redis:
    image: redis:6.2-alpine
    command: redis-server --requirepass your_redis_password
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data

volumes:
  redis_data:

Note: If using AWS credentials, add AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, or attach the IAM role to your deployment environment.

Run Docker Compose:

docker-compose up -d

Alert Rules

Create a prometheus.yml file to define a metric and alerting rule:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'server'
    static_configs:
      - targets: ['localhost:9090']

rule_files:
  - 'alert_rules.yml'

Create alert_rules.yml to define an alert:

groups:
- name: rate
  rules:

  - alert: HighErrorRate
    expr: rate(http_requests_total{status="500"}[5m]) > 0.1
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High error rate detected in {{ $labels.service }}"
      description: "{{ $labels.service }} has an error rate above 0.1 for 5 minutes."

  - alert: HighErrorRate
    expr: rate(http_requests_total{status="500"}[5m]) > 0.5
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "Very high error rate detected in {{ $labels.service }}"
      description: "{{ $labels.service }} has an error rate above 0.5 for 2 minutes."

  - alert: HighErrorRate
    expr: rate(http_requests_total{status="500"}[5m]) > 0.8
    for: 1m
    labels:
      severity: urgent
    annotations:
      summary: "Extremely high error rate detected in {{ $labels.service }}"
      description: "{{ $labels.service }} has an error rate above 0.8 for 1 minute."

Alert Manager Routing Configuration

Configure Alert Manager to route alerts to Versus with different behaviors.

Send Alert Only (No On-Call)

receivers:
- name: 'versus-no-oncall'
  webhook_configs:
  - url: 'http://versus-service:3000/api/incidents?oncall_enable=false'
    send_resolved: false

route:
  receiver: 'versus-no-oncall'
  group_by: ['alertname', 'service']
  routes:
  - match:
      severity: warning
    receiver: 'versus-no-oncall'

Send Alert with Acknowledgment Wait

receivers:
- name: 'versus-with-ack'
  webhook_configs:
  - url: 'http://versus-service:3000/api/incidents?oncall_wait_minutes=5'
    send_resolved: false

route:
  routes:
  - match:
      severity: critical
    receiver: 'versus-with-ack'

This waits 5 minutes for acknowledgment before triggering the AWS Incident Manager Response Plan if the user doesn't click the link ACK that Versus sends to Slack.

Send Alert with Immediate On-Call Trigger

receivers:
- name: 'versus-immediate'
  webhook_configs:
  - url: 'http://versus-service:3000/api/incidents?oncall_wait_minutes=0'
    send_resolved: false

route:
  routes:
  - match:
      severity: urgent
    receiver: 'versus-immediate'

This triggers the response plan immediately without waiting.

Testing the Integration

Trigger an Alert: Simulate a critical alert in Prometheus to match the Alert Manager rule.
Verify Versus: Check that Versus receives the alert and sends it to configured channels (e.g., Slack).
Check Escalation:
Wait 5 minutes without acknowledging the alert.
In Incident Manager > Incidents, verify that an incident starts and Team A is engaged.
After 5 more minutes, confirm Team B is engaged.
Immediate Trigger Test: Send an urgent alert and confirm the response plan triggers instantly.

Conclusion

You’ve now integrated Versus Incident with AWS Incident Manager for on-call management! Alerts from Prometheus Alert Manager can trigger notifications via Versus, with escalations handled by AWS Incident Manager based on your response plan. Adjust configurations as needed for your environment.

If you encounter any issues or have further questions, feel free to reach out!

Replace Opsgenie with this open-source alert router

DevOps VN — Wed, 12 Mar 2025 09:36:33 +0000

Tired of paying $199.5/month (10 users) for Opsgenie? Here’s how teams like yours are slashing costs by 90% while gaining full control over alert routing and templates using Versus Incident – an open-source alternative trusted by DevOps teams managing 10k+ servers.

Why Teams Are Switching

Feature	Opsgenie	Versus Incident
Cost	$2,280/year (10 users)	Free
Custom Templates	Limited	Go Templating
Multi-Channel Routing	Add-on Fees	Built-in
Lock-in Risk	High	None

In this guide, you’ll learn how to route Prometheus Alertmanager alerts to Slack and Telegram using the Versus Incident, while fully customizing alert messages.

Step 1: Redirect Alertmanager to Versus (5 Minutes)

Update your alertmanager.yml to bypass Opsgenie’s webhook:

route:
  receiver: 'versus-incident'
  group_wait: 10s  # Faster than Opsgenie's 30s default!

receivers:
- name: 'versus-incident'
  webhook_configs:
  - url: 'http://versus-host:3000/api/incidents' 
    send_resolved: true  # Get resolution notices for free

Step 2: Deploy Versus Incident (With Multi-Channel Fallback)

Create a configuration folder:

mkdir -p ./config

Create a configuration file config/config.yaml:

name: versus
host: 0.0.0.0
port: 3000

alert:
  slack:
    enable: true
    token: ${SLACK_TOKEN}
    channel_id: ${SLACK_CHANNEL_ID}
    template_path: "/app/config/slack_message.tmpl"

  telegram:
    enable: true
    bot_token: ${TELEGRAM_BOT_TOKEN}
    chat_id: ${TELEGRAM_CHAT_ID}
    template_path: "/app/config/telegram_message.tmpl"

docker-compose.yml – Always-on alerting with Slack + Telegram backup:

version: '3.8'
services:
  versus:
    image: ghcr.io/versuscontrol/versus-incident
    ports:
      - "3000:3000"
    volumes:
      - ./config:/app/config  # Persist templates
    environment:
      SLACK_ENABLE: "true"
      SLACK_TOKEN: "xoxb-..."
      SLACK_CHANNEL_ID: "C123"
      TELEGRAM_ENABLE: "true"  # Failover channel
      TELEGRAM_BOT_TOKEN: "123:ABC"
      TELEGRAM_CHAT_ID: "-456"

Key Advantage: Unlike Opsgenie’s per-channel pricing, Versus lets you blast alerts to unlimited channels for $0.

Step 3: Create Context-Rich Templates

Slack Template (config/slack_message.tmpl):

🔥 {{.commonLabels.severity | upper}} ALERT: {{.commonLabels.alertname}}  
📦 *Cluster*: {{.commonLabels.cluster | default "N/A"}}  
🚨 *Status*: {{.status | title}}  

{{range .alerts}}
⏰ {{.startsAt | formatTime}}  
{{.annotations.description | indent 2}}
{{end}}

{{if .commonLabels.runbook}}📖 <{{.commonLabels.runbook}}|Runbook>{{end}}

Telegram Template (config/telegram_message.tmpl):

🚩 <b>{{.commonLabels.alertname}}</b> ({{.status}})

{{range .alerts}}
🕒 {{.startsAt | formatTime}}
{{.annotations.summary}}
{{end}}

{{if eq .commonLabels.severity "critical"}}🔴 <i>PAGE ON-CALL</i>{{end}}

Step 4: Advanced Routing

Route to different channels based on labels:

# Send database alerts to DBA team
curl -X POST http://versus-host:3000/api/incidents?slack_channel_id=C001DB \
  -H "Content-Type: application/json" \
  -d '{"commonLabels":{"alertname":"PostgresqlDown","team":"dba"}}'

# Send frontend alerts to web-team
curl -X POST http://versus-host:3000/api/incidents?slack_channel_id=C002FE \
  -H "Content-Type: application/json" \
  -d '{"commonLabels":{"alertname":"FrontendLatency","team":"web"}}'

Cost Comparison:

Opsgenie charges $199.5/month (10 users) for basic routing rules – for something Versus does better for free.

Migration Checklist

Redirect Alertmanager webhooks to Versus
Set up multi-channel fallbacks (Slack + Telegram + Email)

Cons

Q: How do we handle on-call schedules?

→ Now we support AWS Incident Manager

Books you should read to become DevOps for beginner

DevOps VN — Fri, 23 Sep 2022 02:55:28 +0000

Hi guys, In this post, I’m going to introduce the books that help me a lot on the path to becoming a Cloud DevOps Engineer from a Full-stack Developer. Hope it is useful for everyone, the books that I recommend go from basic to advance.

Basic

First, the most basic thing that you must know is Linux. Don’t try to become a Linux master (because no one can claim to be a Linux master), we just need to understand the basics about Linux like What is Linux? What’s the difference between Linux and Ubuntu or Centos? Basic Linux commands and Linux host filesystem.

For beginners, I recommend this book for you: Linux for Beginners.

This book will teach you the basics of Linux, the difference between Linux and Ubuntu, the basic Linux command line (CLI), and what CLI is needed? And how to create and manage users on Linux.

If you want to learn deep about Linux, I recommend this book: Ubuntu Linux Unleashed 2021 Edition.

The first step

After you learn the basics about Linux, the next step you need to learn is how to run the programs on different Operating Systems (especially Linux).

It would be best if you learned the basics of NodeJS, JAVA, or .NET. But note we don’t need to learn how to program the language, we just need to learn how to run it, how to install language packages and libraries, and how to build or compile code in runtime.

For example with NodeJS, you just need to learn how to install NodeJS, how to run NodeJS use pm2, how to install NodeJS package using NPM.

For example with JAVA, you need to learn how to install and run JAVA, and how to compile JAVA into a jar file.

Next, to run different programs as easily as possible then we have to learn about Container and Docker. For basic about Docker, I recommend this book: Docker in Action, Second Edition.

In this book, you will learn the most basic about Container and Docker, how to run the container, how to build source code into the container image, and how to use docker-compose and self-host container registry. When you have done, you will have a better overview of how to use Docker to run our programs on different operating systems.

Step on the road to becoming a DevOps

After doing the basics of Linux and Docker, now you can confidently step on the path to becoming a DevOps Engineer.

The next thing you need to learn is how you can automatically deliver our products or programs to clients as easily as possible. The way to do that is called CI/CD, for the basics of CI/CD then I recommend this book: Continuous Delivery with Docker and Jenkins.

This book will show you what you need to set up the CI/CD flow of your program. You will learn the basics of Jenkins and Jenkins Architecture, how to install and set up Jenkins, and how to use Jenkins with Docker.

And along with learning about CI/CD, we also need to learn about Container Orchestration. Container Orchestration is the way to operate and manage thousands of containers in the actual production environment that our customers are using. For beginners to learn the basics of Kubernetes then the book Kubernetes in Action, Second Edition is a great choice.

I love this book, Marko Lukša is a great author that writes about Kubernetes. In this book, Marko Lukša will teach you what’s Kubernetes and how to use it from basic to advanced, and best practices on how to use Kubernetes to deploy applications.

And try to practice more and more about CI/CD and Kubernetes.

Middle

If you finish reading the above books and practice a lot then welcome, you can confidently say you are a Fresher DevOps. The next thing you should learn is how to apply the theory in real work?

Although this is a very difficult job, I also have the following two books to recommend to everyone.

The first book is Managing Kubernetes.

This is a book that will show you how to run Kubernetes in production, how to install Kubernetes in production, how to manage your Kubernetes cluster, and how to backup and restore it.

The second book is Kubernetes Best Practices — Blueprints for Building Successful Applications on Kubernetes.

This is a book that will give you a lot of practical examples, on how to install many applications on the Kubernetes environment, and how to monitor applications.

Become a Good DevOps

Is there any way to make your value better than other DevOps candidates? In my opinion, for our value to be better than the rest of the DevOps candidates, we need to know about Cloud. The future will be Cloud’s playground.

To learn Cloud, we should choose the largest and most used providers, at the moment when I write this article, the most popular Cloud is AWS, and I also like AWS very much.

To start with Cloud, I recommend you two books and a course on Udemy. The first is the book Amazon Web Services in Action, Second Edition.

Throughout this book, you will learn about what AWS is, the basic services of AWS, how to use it, how to design a highly loaded system, and how to design a system that is easy to scale on AWS.

Next is a course that teaches us how to get an AWS certification: Ultimate AWS Certified Solutions Architect Associate.

This course is by a very famous instructor, Stephane Maarek. He will teach you everything you need to know about AWS to take the certification exam. Although it is a certification exam course, after completing the course, we will have the best overview of AWS.

Next is the book about Infrastructure as Code: Terraform in Action.

Usually, when working with AWS, we will use another tool provisioning infrastructure on AWS, and the most popular tool at the time I write is Terraform. The above book will show us all the things we need about Terraform and how to use Terraform with AWS.

Keep Learning

If you can complete all of the above, congratulations, you have become a DevOps Engineer 😁. But the path of learning never ends, to have more knowledge, we must study and learn. These are the books I am reading to improve my knowledge.

The first is a book about monitoring, of course, in our operation then monitoring is indispensable: Monitoring with Prometheus.

This book will teach us how to use Prometheus to monitor the system and Grafana to query, visualize, and alert your metrics system.

Next is Search Engines: Elasticsearch in Action.

This is a book that will teach us about Elasticsearch, although this book teaches Elasticsearch 5.x, from there we can google more to learn about how Elasticsearch 7.x works, which will give us something to look at it in more detail.

Finally is about Event Streaming: Kafka: The Definitive Guide.

When we work with large systems, Event Streaming is indispensable, and Kafka is one of the most powerful tools to do this.

Conclusion

These books help me a lot and hope they will be useful to everyone. DevOps and Cloud are very interesting, although we can never learn all about them, keep learning guys 😁.

If you like my article you can support me by buying me a coffee ☕️, thank you.

Books to learn Kubernetes from beginner to advanced

DevOps VN — Sun, 18 Sep 2022 07:59:20 +0000

Introduction

Kubernetes is a popular tool with DevOps engineers, but learning and using it is difficult for beginners. So in this article, I want to introduce to everyone the books that have helped me learn Kubernetes from scratch.

Basic level

Kubernetes in Action

This book is for those who don't know anything about Kubernetes: Kubernetes in Action, Second Edition.

Kubernetes in Action is a great book that teaches you about Kubernetes by examples, you will learn the basics of Kubernetes in this book. After reading this book, you will know the following concepts:

What is Kubernetes?
Why do we need Kubernetes?
How does Kubernetes manage Containers?
How does Kubernetes scale Containers?
Kubernetes internal architecture.
etc...

Kubernetes Best Practices - Blueprints for Building Successful Applications on Kubernetes

This book is to practice more about Kubernetes: Kubernetes Best Practices.

You will learn how to install various applications using Kubernetes in this book. For example:

How to install the Ingress Controller?
How to install Prometheus?
How to install Elasticsearch, Logstash, and Kibana?
Best practice to manage ConfigMaps and Secrets.
How to build CI/CD system on Kubernetes?
etc...

Middle

After reading the above books, you can use Kubernetes in the production environment. But if you want to understand more about Kubernetes, the next book will help you.

Managing Kubernetes

There are two roles when you work with Kubernetes: Kubernetes Developer and Kubernetes Administrator.

Kubernetes Developers write the manifest files for the application and deploy them on Kubernetes.

Kubernetes Administrators set up, manage, and operate the Kubernetes Cluster so that it can be easily used by the Developers.

The Managing Kubernetes book will show you how to manage Kubernetes.

You will learn:

How to run Kubernetes in Production?
How to manage the Kubernetes cluster?
How to back up the Kubernetes Cluster.
etc...

Kubernetes Operators

Operators are software extensions to Kubernetes that make use of custom resources to manage applications and their components. Operators follow Kubernetes principles, notably the control loop. Kubernetes Operators examples:

Argo Rollout.

Elastic Cloud on Kubernetes (ECK).

Confluent for Kubernetes.

Operators are clients of the Kubernetes API that act as controllers for a Custom Resource, for example:

Just a simple manifest file to deploy Kafka 😁. This is a guidebook for you to design and build the Operators: Kubernetes Operators.

Advanced

These two books are for understanding deep into Kubernetes.

GitOps and Kubernetes

This book will explain the concept of GitOps and how to build CI/CD in Kubernetes: GitOps and Kubernetes.

You will learn:

How to build the CI/CD?
Canary and blue-green deployment.
Security.
ArgoCD, Jenkins X, Flux.
etc...

Core Kubernetes

Finally, a book that will take your understanding of Kubernetes to a whole new level: Core Kubernetes.

This book will teach you the core components of Kubernetes. I haven't finished reading this book yet, but it is a great book.

Conclusion

Hoping these books will be useful to everyone. If you know good books, please recommend them to everyone 😁.

Deep into Container — Build your own container with Golang

DevOps VN — Sun, 07 Aug 2022 15:48:29 +0000

Hi guys, continuing with the series of Deep into Container, we already know that containers are built from Linux Namespaces and Cgroups, and to learn more deeply about it, we're going to learn how to build your own container using Golang.

This article I referenced from Build Your Own Container Using Less than 100 Lines of Go by Julian Friedman and Building a container from scratch in Go by Liz Rice.

This is part four in the series Deep into Container:

Linux namespaces and Cgroups: What are containers made from?
Deep into Container Runtime.
How Kubernetes works with Container Runtime.
Deep into Container - Build your own container with Golang.

Building a Container

Create a file named container.go and write some simple code as follows.

package main

import (
    "os"
)

func main() {

}

func must(err error) {
    if err != nil {
        panic(err)
    }
}

If you are familiar with Docker then you know a command to run the container is docker run <container> <command>, for example:

docker run busybox echo "A"

You will see the container run and print the letter "A", and if you run the following command:

docker run -it busybox sh

The container run and the shell will attach to it.

/ #

If we type a command now, that command is running in the container.

/ # hostname
d12ccc0e00a0

/ # ps
PID   USER     TIME  COMMAND
1     root      0:00 sh
9     root      0:00 ps

The hostname command doesn't print the hostname of the server that prints the hostname of a container, and the ps command print only two processes.

Now we will build a similar container like the above using Golang, update the container.go as follows.

package main

import (
    "os"
)

// docker run <image> <command>
// go run container.go run <command>
func main() {
    switch os.Args[1] {
    case "run":
        run()
    default:
        panic("Error")
    }
}

func run() {

}

func must(err error) {
    if err != nil {
        panic(err)
    }
}

We add a function named run() and in the main function, we use the switch case syntax to check that when we run the program with the flag as run, it will run the run() function. Now when we run the command go run container.go run, it will be similar to when we run docker run.

Next, we update the run() function as follows.

package main

import (
    "os"
  "os/exec"
)

// docker run <image> <command>
// go run container.go run <command>
func main() {
    switch os.Args[1] {
    case "run":
        run()
    default:
        panic("Error")
    }
}

func run() {
    cmd := exec.Command(os.Args[2], os.Args[3:]...)
    cmd.Stdin = os.Stdin
    cmd.Stdout = os.Stdout
    cmd.Stderr = os.Stderr

    must(cmd.Run())
}

func must(err error) {
    if err != nil {
        panic(err)
    }
}

We use the os/exec package to execute user input commands that are stored in the os.Args array, for example, when we type go run container.go run echo "A", then the os.Args will have a value of:

Args[0] = "container.go"
Args[1] = "run"
Args[2] = "echo"
Args[3] = "A"

The value that we need to pass into the exec.Command() we get from the index two of os.Args. The syntax of Command() function as follows.

exec.Command(name string, arg ...string)

The function takes the first argument which is the command it will execute and the remaining values are arguments of that command.

Now, try to run the same command as docker run -it busybox sh with your program.

go run container.go run sh

You will see that it is mostly the same when you run the docker command.

We have successfully taken the first step 😁, but when you type the hostname command, it will print the hostname of our server, not of the container.

# hostname
LAPTOP-2COB82RG

If you type the command to change the hostname in our program, it will affect the outside of the server as well.

# hostnamectl set-hostname container

Type exit and enter, now outside the server, we type the hostname we will see it has been changed.

Our program is currently just running the sh command not the container at all, next, we will go through each step to build the container. As we know the container is built from Linux Namespaces.

Namespaces

Namespaces provide the isolation environment that helps us run a process independent of other processes on the same server. At the time of writing, there are six namespaces as follows,

PID: The PID namespace provides processes with an independent set of process IDs (PIDs) from other namespaces. The PID namespace makes the first process created within it assigned with PID 1.
MNT: Mount namespaces control mount points, and provide you to mount and unmount folders without affecting other namespaces.
NET: Network namespaces create their network stack for the process.
UTS: UNIX Time-Sharing namespaces allow a process has a separate hostname and domain name.
USER: User namespaces create their own set of UIDS and GIDS for the process.
IPC: IPC namespaces isolate processes from inter-process communication, this prevents processes in different IPC namespaces from using.

We will use PID, UTS, and MNT namespaces in our Golang program.

UTS namespace

The first thing we need to isolate is the hostname so that our program has its hostname. Update container.go.

package main

import (
  "os"
  "os/exec"
  "syscall"
)

// docker run <image> <command>
// go run container.go run <command>
func main() {
    switch os.Args[1] {
    case "run":
        run()
    default:
        panic("Error")
    }
}

func run() {
    cmd := exec.Command(os.Args[2], os.Args[3:]...)
    cmd.Stdin = os.Stdin
    cmd.Stdout = os.Stdout
    cmd.Stderr = os.Stderr
    cmd.SysProcAttr = &syscall.SysProcAttr{
        Cloneflags: syscall.CLONE_NEWUTS,
    }

    must(cmd.Run())
}

func must(err error) {
    if err != nil {
        panic(err)
    }
}

To use Linux namespaces in Go, we simply pass the namespace flag we want to use in cmd.SysProcAttr.

cmd.SysProcAttr = &syscall.SysProcAttr{
    Cloneflags: syscall.CLONE_NEWUTS,
}

Now let's try again.

go run container.go run sh

Run the command to change the hostname.

# hostnamectl set-hostname wsl
# hostname
wsl

Type exit and enter, now outside the server, you type the hostname command and you'll see the hostname of the server not change at all. We have completed the next step in building the container 😁.

However for our program to be more like a container, we need to do a few more things. As you can see when we run docker run -it busybox sh and then type hostname it will have its hostname, not like we run the program, and we have to manually type the command to change the hostname. Update container.go.

package main

import (
    "os"
    "os/exec"
    "syscall"
)

// docker run <image> <command>
// ./container run <command>
func main() {
    switch os.Args[1] {
    case "run":
        run()
    case "child":
        child()
    default:
        panic("Error")
    }
}

func run() {
    cmd := exec.Command("/proc/self/exe", append([]string{"child"}, os.Args[2:]...)...)
    cmd.Stdin = os.Stdin
    cmd.Stdout = os.Stdout
    cmd.Stderr = os.Stderr
    cmd.SysProcAttr = &syscall.SysProcAttr{
        Cloneflags: syscall.CLONE_NEWUTS,
    }

    must(cmd.Run())
}

func child() {
    syscall.Sethostname([]byte("container"))

    cmd := exec.Command(os.Args[2], os.Args[3:]...)
    cmd.Stdin = os.Stdin
    cmd.Stdout = os.Stdout
    cmd.Stderr = os.Stderr

    must(cmd.Run())
}

func must(err error) {
    if err != nil {
        panic(err)
    }
}

we add another function named child() and in the run function, we execute the child function by exec.Command.

exec.Command("/proc/self/exe", append([]string{"child"}, os.Args[2:]...)...)

We change the first argument to /proc/self/exe, this command will self-executing the program with a child argument. The child process now runs in isolation UTS namespaces, and we change the hostname with the function syscall.Sethostname([]byte("container")).

go run container.go run sh -> /proc/self/exe child sh -> syscall.Sethostname([]byte("container")) -> exec.Command("sh")

Let's try again.

go run container.go run sh

Typing hostname and you will see your process has its own hostname.

# hostname
container

So we have completed the next step 😁.

Next, try to type the ps command to list the process, and see if is it the same as when we run docker run?

# ps
PID   TTY      TIME     CMD
11254 pts/3    00:00:00 sudo
11255 pts/3    00:00:00 bash
17530 pts/3    00:00:00 go
17626 pts/3    00:00:00 container
17631 pts/3    00:00:00 exe
17636 pts/3    00:00:00 sh
17637 pts/3    00:00:00 ps

Not like at all, the processes you see are processes outside the server.

PID namespace

We will use the PID namespace to create a process with an independent set of process IDs (PIDs). Update container.go as follows.

...
func run() {
 cmd := exec.Command("/proc/self/exe", append([]string{"child"}, os.Args[2:]...)...)
 cmd.Stdin = os.Stdin
 cmd.Stdout = os.Stdout
 cmd.Stderr = os.Stderr
 cmd.SysProcAttr = &syscall.SysProcAttr{
  Cloneflags: syscall.CLONE_NEWUTS | syscall.CLONE_NEWPID,
 }
 must(cmd.Run())
}
...

We just need to add one flag is syscall.CLONE_NEWPID, now let's run again.

go run container.go run sh

# ps
PID   TTY      TIME     CMD
11254 pts/3    00:00:00 sudo
11255 pts/3    00:00:00 bash
17530 pts/3    00:00:00 go
17626 pts/3    00:00:00 container
17631 pts/3    00:00:00 exe
17636 pts/3    00:00:00 sh
17637 pts/3    00:00:00 ps

What? It does not change at all. Why?

When we run the ps program, it will get process information in /proc folder in Linux, let's try.

ls /proc

Now, the filesystem of your process looks the same as the host, because its filesystem is inherited from the current server, let's change that.

MNT namespace

Update container.go as follows.

package main

import (
    "os"
    "os/exec"
    "syscall"
)

// docker run <image> <command>
// ./container run <command>
func main() {
    switch os.Args[1] {
    case "run":
        run()
    case "child":
        child()
    default:
        panic("Error")
    }
}

func run() {
    cmd := exec.Command("/proc/self/exe", append([]string{"child"}, os.Args[2:]...)...)
    cmd.Stdin = os.Stdin
    cmd.Stdout = os.Stdout
    cmd.Stderr = os.Stderr
    cmd.SysProcAttr = &syscall.SysProcAttr{
        Cloneflags: syscall.CLONE_NEWUTS | syscall.CLONE_NEWPID | syscall.CLONE_NEWNS,
    }

    must(cmd.Run())
}

func child() {
    syscall.Sethostname([]byte("container"))
    must(syscall.Chdir("/"))
    must(syscall.Mount("proc", "proc", "proc", 0, ""))

    cmd := exec.Command(os.Args[2], os.Args[3:]...)
    cmd.Stdin = os.Stdin
    cmd.Stdout = os.Stdout
    cmd.Stderr = os.Stderr

    must(cmd.Run())
}

func must(err error) {
    if err != nil {
        panic(err)
    }
}

We use syscall.CLONE_NEWNS flag to create a process with MNT namespaces, and change the filesystem.

syscall.Chdir("/")
syscall.Mount("proc", "proc", "proc", 0, "")

Now, let's run again.

go run container.go run sh

Typing the ps command.

# ps
PID TTY      TIME     CMD
1   pts/3    00:00:00 exe
7   pts/3    00:00:00 sh
8   pts/3    00:00:00 ps

We succeeded 😁.

Conclusion

So we know how to build a simple container using Golang, but in reality, the container will have many other things, like Cgroups to limit the process's resources, create USER namespaces, and mount files from the container to the server, etc…

But basically, the main feature for containers to create an isolation environment is Linux namespaces. If you have any questions or need more clarification, you can ask in the comment section below.

Improve your Golang skill by rewriting popular software

DevOps VN — Tue, 19 Jul 2022 16:01:07 +0000

The fastest way to learn is to learn by doing. If you are unsure what to do to improve your skills and knowledge of the Golang language, then in this article I will introduce some documents that show you how to rewrite popular software that uses Golang.

These tutorials come from the Github repository Build Your Own X. A popular and useful Github repository.

Build BitTorrent Client

From Wikipedia:

BitTorrent is a communication protocol for peer-to-peer file sharing, which enables users to distribute data and electronic files over the Internet in a decentralized manner.

This tutorial below will help you write your BitTorrent software using the Golang language.

Building a BitTorrent client from the ground up in Go.

Build Blockchain/Cryptocurrency

Blockchain is probably not new if we know about cryptocurrency, this has been a scorching topic in recent years. Below is a tutorial series on how to build blockchain using Go.

Building Blockchain in Go.

Build Command-Line Tool

Next, these tutorials on building CLI tools, when writing CLI tools we will have a better understanding of the operating system. Here are four tutorials that show you how to build four different CLI tools.

Visualize your local git contributions with Go.

Build a command line app with Go: lolcat.

Building a CLI command with Go: cowsay.

Go CLI tutorial: fortune clone.

Build Container

The tutorial on how to write your Container Runtime with Go is the best I've read, in this article, the author will explain to us how Containers work, and how to use Golang to build it, this post is great. This is the article that will help you somewhat become more understanding of the Golang language and how Linux OS works. Thank Julian Friedman and Liz Rice.

Build Your Container Using Less than 100 Lines of Go.

Building a container from scratch in Go - Video.

Build Game

Have you ever thought that Golang can be used to write Games? If not, this article will guide you.

Games With Go - Video.

Build Neural Network

If you do AI, you must know Neural Network (I don't know it 😂). Maybe, we often write an AI Model in Python, if you want to challenge yourself, try to build an AI Model with Go.

Build a multilayer perceptron with Golang.

How to build a simple artificial neural network with Go.

Building a Neural Net from Scratch in Go.

Build Shell

The simplest explanation of the shell is a program that takes commands from the keyboard and gives them to the operating system to perform. The below article will guide you on how to write your shell interface using Go, after reading you will have a better understanding of how the shell works.

Writing a simple shell in Go.

In this tutorial, Simon Jürgensmeyer will show you how to re-code a simple shell in less than 80 lines of code.

package main

import (
    "bufio"
    "errors"
    "fmt"
    "os"
    "os/exec"
    "os/user"
    "strings"
)

func main() {
    reader := bufio.NewReader(os.Stdin)
    for {
        path, err := os.Getwd()
        if err != nil {
            fmt.Fprintln(os.Stderr, err)
        }

        hostname, err := os.Hostname()
        if err != nil {
            fmt.Fprintln(os.Stderr, err)
        }

        currentUser, err := user.Current()
        if err != nil {
            fmt.Fprintln(os.Stderr, err)
        }

        path = strings.Replace(path, "/home/"+currentUser.Username+"", "~", -1)

        fmt.Printf("%s@%s:%s$ ", currentUser.Username, hostname, path)

        // Read the keyboad input.
        input, err := reader.ReadString('\n')
        if err != nil {
            fmt.Fprintln(os.Stderr, err)
        }

        if err := execInput(input); err != nil {
            fmt.Fprintln(os.Stderr, err)
        }
    }
}

func execInput(input string) error {
    input = strings.TrimSuffix(input, "\n")

    args := strings.Split(input, " ")

    switch args[0] {
    case "cd":
        if len(args) < 2 {
            return errors.New("path required")
        }

        return os.Chdir(args[1])
    case "exit":
        os.Exit(0)
    }

    cmd := exec.Command(args[0], args[1:]...)

    cmd.Stderr = os.Stderr
    cmd.Stdout = os.Stdout

    return cmd.Run()
}

Build Load Balancer

Load Balancer is probably no stranger to you guys working on the server. Load balancer acts as the traffic division sitting in front of your servers and routing client requests across all servers. Writing your Load Balancer will help you better understand a lot of knowledge about the LB. This is a tutorial on how to build a Load Balancer by yourself using Go.

Let's Create a Simple Load Balancer.

Conclusion

In this article, I just want to introduce you to useful resources for learning and improving my skills, many of the above articles I have not read yet 😁. In addition to Go, in the Github repository I introduced above, there are many other good articles written in different languages, please go and see, very useful for you.

Kubernetes Practice — Logging with Logstash and FluentD by Sidecar Container

DevOps VN — Fri, 15 Jul 2022 16:40:16 +0000

We are going to learn how to use the Sidecar Container pattern to install Logstash and FluentD on Kubernetes for log aggregation.

For any system, log aggregation is very important. When you use Kubernetes to run your application, the log only belongs to one Pod. If that Pod is deleted, the log is also lost.

Therefore, if we want to track system failures, we must have a log aggregation system. At this time, two popular log stacks are ELK (Elasticsearch Logstash Kibana) and EFK (Elasticsearch FluentD Kibana).

To collect logs on each Pod, we use Sidecar Container.

Sidecar Container

Instead of implementing the log collection process on the application containers, we can separate that process to another container to avoid affecting the performance of the application containers. That container is called Sidecar Container.

Sidecar containers are the containers that should run along with the main container in the pod. This sidecar container extends and enhances the application containers in some way.

Logging with Logstash

The original task of Logstash is monitoring logs and transforming them into a meaningful set of fields and eventually streaming the output to a defined destination. However, it has an issue with performance.

So, Elastic has launched Filebeat that use for monitoring logs and streaming the output to a defined destination.

And Logstash acts as an aggregator that ingests data from a multitude of sources, transforms it, and then sends it to your favorite “stash.”

So we’re done with the theory, let’s get to work. First we deploy a Pod with an application container that writes logs to the file /var/log/access.log, and we deploy a sidecar container on the same Pod that runs Filebeat to collect logs and output logs to Logstash.

Create a file named filebeat.cm.yaml to store the Filebeat configuration file.

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  labels:
    component: filebeat
data:
  conf.yaml: |
    filebeat.inputs:
    - type: log
      paths:
        - '/var/log/*.log'
    output:
      logstash:
        hosts: [ "logstash:5044" ]

The input of Filebeat we read from files /var/log/*.log, then we output these logs to Logstash.

Create a file named application.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox
  labels:
    component: busybox
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      component: busybox
  template:
    metadata:
      labels:
        component: busybox
    spec:
      containers:
        - name: busybox
          image: busybox
          args:
            - sh
            - -c
            - >
              while true;
              do
                echo $(date) - filebeat log >> /var/log/access.log;
                sleep 10;
              done
          volumeMounts:
            - name: log
              mountPath: /var/log
        - name: filebeat
          image: elastic/filebeat:7.16.3
          args:
            - -c
            - /etc/filebeat/conf.yaml
            - -e
          volumeMounts:
            - name: filebeat-config
              mountPath: /etc/filebeat
            - name: log
              mountPath: /var/log
      volumes:
        - name: log
          emptyDir: {}
        - name: filebeat-config
          configMap:
            name: filebeat-config

In the Pod above we mount the Filebeat configuration file into the /etc/filebeat/conf.yaml file and use the args to specify that configuration file for Filebeat.

Our application container writes a log to the file /var/log/access.log every 10s. We use emptyDir volumes to share storage between two containers.

Next, we create a file named logstash.cm.yaml to store the Logstash configuration file.

apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash
  labels:
    component: logstash
data:
  access-log.conf: |
    input {
      beats {
        port => "5044"
      }
    }
    output {
      elasticsearch {
        hosts => [ "elasticsearch:9200" ]
      }
    }

Create a Logstash Deployment file named logstash.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: logstash
  labels:
    component: logstash
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      component: logstash
  template:
    metadata:
      labels:
        component: logstash
    spec:
      containers:
        - name: logstash
          image: logstash:7.16.3
          ports:
            - containerPort: 5044
          volumeMounts:
            - name: logstash-config
              mountPath: /usr/share/logstash/pipeline
      volumes:
        - name: logstash-config
          configMap:
            name: logstash
---
apiVersion: v1
kind: Service
metadata:
  name: logstash
  labels:
    component: logstash
spec:
  ports:
  - port: 5044
  selector:
    component: logstash

We mount the configuration file to the folder /usr/share/logstash/pipeline, Logstash will load the configuration files from this folder.

Create Elastichsearch (just for test).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: elasticsearch
  labels:
    component: elasticsearch
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      component: elasticsearch
  template:
    metadata:
      labels:
        component: elasticsearch
    spec:
      containers:
        - name: elasticsearch
          image: elasticsearch:7.16.3
          ports:
            - containerPort: 9200
              name: client
            - containerPort: 9300
              name: nodes
          env:
            - name: JAVA_TOOL_OPTIONS
              value: -Xmx256m -Xms256m
            - name: discovery.type
              value: single-node
          resources:
            requests:
              memory: 500Mi
              cpu: 0.5
            limits:
              memory: 500Mi
              cpu: 0.5
---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  labels:
    component: elasticsearch
spec:
  ports:
  - port: 9200
    name: client
  - port: 9300
    name: nodes
  selector:
    component: elasticsearch

Kibana.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  labels:
    component: kibana
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      component: kibana
  template:
    metadata:
      labels:
        component: kibana
    spec:
      containers:
        - name: kibana
          image: kibana:7.16.3
          ports:
            - containerPort: 5601
---
apiVersion: v1
kind: Service
metadata:
  name: kibana
  labels:
    component: kibana
spec:
  ports:
  - port: 5601
  selector:
    component: kibana

Run apply command to create resources.

kubectl apply -f . --recursive

Use port-forward to access Kibana Dashboard.

kubectl port-forward svc/kibana 5601:5601

Now, go to menu Stack Management > Index patterns and create an index pattern, then go to menu Discover and you’ll see the logs we collected from the busybox container.

Logging with FluentD

FluentD is also a log collection tool like Filebeat and Logstash. It is an open-source data collector, which lets you unify the data collection and consumption for better use and understanding of data.

We can use it as Sidecar Container to collect logs from a Pod.

Create a file named fluentd.cm.yaml to store the Filebeat configuration file.

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  labels:
    component: fluentd
data:
  fluent.conf: |
    <source>
      @type tail
      path /var/log/access.log
      pos_file /tmp/app.logs.pos
      tag app.logs
      <parse>
        @type none
      </parse>
    </source>
    <match app.logs>
      @type elasticsearch
      host elasticsearch
      port 9200
      logstash_format true
      logstash_prefix fluentd
      flush_interval 1s
    </match>

we use the <source> tag to specify where we should collect the logs, then we use the <match> tag to output the log to Elasticsearch.

Next, create a file named application.yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox
  labels:
    component: busybox
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      component: busybox
  template:
    metadata:
      labels:
        component: busybox
    spec:
      containers:
        - name: busybox
          image: busybox
          args:
            - sh
            - -c
            - >
              while true;
              do
                echo $(date) - filebeat log >> /var/log/access.log;
                sleep 10;
              done
          volumeMounts:
            - name: log
              mountPath: /var/log
        - name: fluentd
          image: govtechsg/fluentd-elasticsearch
          volumeMounts:
            - name: fluentd-config
              mountPath: /fluentd/etc
            - name: log
              mountPath: /var/log
      volumes:
        - name: log
          emptyDir: {}
        - name: fluentd-config
          configMap:
            name: fluentd-config

Run apply command to create resources.

kubectl apply -f . --recursive

Fluentd Plugin

It is important to note that in order to output logs to Elasticsearch, we must use the Fluentd Elasticsearch Plugin.

As you can see above we are using the govtechsg/fluentd-elasticsearch container, this container already has the Elasticsearch Plugin.

If you use the fluent/fluentd container, it will give an error that cannot be found @type elasticsearch.

To install the plugin, we can write Dockerfile as follows.

FROM fluent/fluentd:v1.12.0-debian-1.0
USER root
RUN gem install fluent-plugin-elasticsearch --version 5.0.3
USER fluent

Full list of FluentD Plugins https://www.fluentd.org/plugins/all.

Real-world Use Case

Continuing at Medium

Conclusion

So we have learned how to use the Sidecar Container pattern to set up log collection for Pod. ELK and EFK are two very popular log stacks. If you have any questions or need more clarification, you can ask in the comment section below.

DEV Community: DevOps VN

Are You a Software Engineer, or Just an AI Content Paster?

The Pitfall: When Convenience Kills Competence

The Strategy: Use AI to Support, Not Replace

1. Let AI Be Your Teacher

2. Always Check AI’s Work

3. Know When to Work Without AI

Stay Curious and Keep Growing

I Generated Production-Ready Kubernetes Configs in 30 Seconds (Here's How You Can Too)

The Problem with How We Prompt AI

Enter the C.R.A.F.T. Framework

The Before and After That Will Blow Your Mind

Why Context Is Your Secret Weapon

🎯 Include Your "Why"

🔧 Specify Your Tech Stack

📋 Define Your Constraints

📊 Show Data Structures

The Role Revolution

Action Words That Actually Work

Format: Get Exactly What You Need

Real-World Results

Your Next Steps

The Future Is Conversational Infrastructure

A step-by-step guide to building a personalized AI assistant from scratch using Python

Table of Contents

Part 1: The First Spark of Intelligence

Part 2: Giving Your Assistant a Memory

Part 3: Connecting the Brain to the Memory

Part 4: Growing Your Assistant

Part 5: The Responsible AI Owner

How to handle alerts from various tools such as Grafana, Kibana, Sentry, AWS, etc.

Step 1: Create a Unified Alert Template

Step 2: Configure Versus

Step 3: Deploy Versus with Docker

Step 4: Expose Versus with ngrok (Optional)

Step 5: Set Up Webhooks in Your Tools

AWS (SNS)

Grafana

Kibana

Sentry

Step 6: Test and Check

Scaling and Improving the Setup

Conclusion

How to Configure Grafana to Send Alerts to Slack and Telegram

Prerequisites

Step 1: Set Up Slack and Telegram Bots

Slack Bot

Telegram Bot

Step 2: Deploy Versus with Slack and Telegram Enabled

Step 3: Configure Grafana Alerts with a Webhook

Step 4: Test the Integration

Conclusion

The Open-Source On-Call Integration

Prerequisites

Setting Up AWS Incident Manager for On-Call

Define IAM Role for Versus

Deploy Versus Incident

Alert Rules

Alert Manager Routing Configuration

Testing the Integration

Conclusion

Replace Opsgenie with this open-source alert router

Why Teams Are Switching

Step 1: Redirect Alertmanager to Versus (5 Minutes)

Step 2: Deploy Versus Incident (With Multi-Channel Fallback)

Step 3: Create Context-Rich Templates

Step 4: Advanced Routing

Migration Checklist

Cons

Books you should read to become DevOps for beginner

Basic

The first step

Step on the road to becoming a DevOps

Middle

Become a Good DevOps

Keep Learning

Conclusion

Books to learn Kubernetes from beginner to advanced

Introduction

Basic level