DEV Community: Kalio Princewill

How AI Engineers Actually Use Datasets: Test Cases, Edge Cases and Agent Reliability

Kalio Princewill — Mon, 06 Apr 2026 23:32:46 +0000

Most AI agent discussions focus on models. In practice, the model is rarely the problem.

When you build an agent today you are almost certainly not training it. The model is fixed. What determines whether the agent actually works is everything around it: the tools it can call, the prompts that guide it, the logic that decides what it does next.

So when people say "we need more data," they usually do not mean training. They mean better test cases, clearer failure scenarios, and a way to measure whether the agent is behaving correctly.

This article breaks down how to evaluate an AI agent properly: what to test, how to structure realistic scenarios from real world data, how to score the path the agent takes not just the answer it lands on, and how to design adversarial tests that force actual reasoning instead of pattern matching.

Using SRE agents as the concrete example throughout.

What You Are Not Doing vs What You Are

Before anything else, this distinction matters.

What you are NOT doing:

Feeding logs into the model to teach it new things
Fine tuning weights
Changing how the underlying LLM reasons

What you ARE doing:

Using real world logs to construct realistic test scenarios
Grading whether the agent investigates correctly
Exposing edge cases the agent currently fails at
Using those failures to improve prompts, tools, and agent logic
Building a test suite that gets harder as the agent gets better

The model does not improve through this process. What improves is the system around it. Test cases are how you measure that system rigorously instead of guessing.
With that clear, the next question is what you are actually grading.

What You Are Actually Testing

When you test an AI agent you are not checking if the model knows things. The model already knows things. You are checking three specific behaviours.

Does the agent pick the right tools in the right order?
Given a scenario, does it investigate correctly or does it jump straight to conclusions?

Does it stop at the right time?
Does it know when it has found the root cause and stop, or does it keep going in circles?

Can it reason through noise?
If there are red herrings in the data, metrics that look suspicious but are not causal, does it get distracted or stay on the right path?

These are behaviours you grade. Not things you train. And the clearest way to see them in practice is to look at a real agent being built against exactly these constraints.

The SRE Agent As A Case Study

An SRE (Site Reliability Engineering) agent is one that investigates production incidents automatically: it gets an alert, pulls logs and metrics, reasons across the signals, and produces a root cause report.

The OpenSRE project is a good concrete example of this in practice. Their test suite lives in tests/e2e/ and covers Kubernetes and RDS Postgres scenarios. They are building a suite of realistic incident scenarios and checking whether the agent handles them correctly.

You can run the agent directly against a test fixture like this:

opensre investigate -i tests/e2e/kubernetes/fixtures/datadog_k8s_alert.json

That JSON fixture is a synthetic but realistic alert, constructed to represent a specific failure mode with logs, metrics, and context included. The agent runs against it and you check whether the investigation was correct. That is the entire idea. Now let us look at what that fixture actually contains.

What A Test Case Actually Looks Like

A test case has four parts: the input the agent sees, the steps you expect it to take, the answer you expect it to reach, and the red herrings it should notice but not chase.

TEST_CASE = {
    "scenario": "RDS Postgres connection pool exhaustion",
    "input": {
        "logs": """
            2024-01-15 14:23:01 UTC [FATAL] remaining connection slots reserved for
            non-replication superuser connections
            2024-01-15 14:23:01 UTC [ERROR] connection to server failed: FATAL:
            sorry, too many clients already
            2024-01-15 14:23:04 UTC [WARNING] pool wait time exceeded 10000ms
        """,
        "metrics": {
            "db_connections": 498,
            "db_connections_max": 500,
            "latency_ms": 4200,
            "cpu_percent": 45,        # elevated but not the cause
            "memory_percent": 60      # fine
        },
        "alert": "Database latency spike - P95 latency exceeded 4000ms"
    },
    "expected_steps": [
        "check_db_connections",
        "check_active_queries",
        "check_pool_configuration",
        "recommend_pool_size_increase"
    ],
    "expected_root_cause": "connection pool exhaustion",
    "red_herrings": {
        "cpu_percent": "elevated but stable, not the cause of latency"
    },
    "should_stop_after": "check_pool_configuration"
}

The logs and metrics are what the agent sees. The expected steps define the correct investigation path. The red herrings flag what the agent should notice but not chase. The stop condition catches agents that keep digging after the answer is already clear.

The agent runs against this input and you grade whether it got the right root cause, investigated in the right order, and did not get pulled off track by the CPU metric.

Trajectory Scoring

Once you have test cases, you need a way to score them. Getting the right answer is not enough. You want to know if the agent got there the right way.

This matters in practice because an agent that stumbles onto the correct answer after checking ten irrelevant things is not a reliable agent. It got lucky. Trajectory scoring measures the investigation path, not just the conclusion.

def score_trajectory(actual_steps: list[str], expected_steps: list[str]) -> dict:
    score = 0
    penalties = []

    for i, step in enumerate(actual_steps):
        if step in expected_steps:
            expected_position = expected_steps.index(step)
            actual_position = i
            # penalise for investigating out of order
            position_penalty = abs(expected_position - actual_position) * 0.2
            score += max(0, 1 - position_penalty)
        else:
            penalties.append(f"unexpected step taken: {step}")

    # penalise for not stopping when root cause was found
    if len(actual_steps) > len(expected_steps) + 2:
        penalties.append("agent continued investigating after root cause was clear")
        score -= 0.5

    return {
        "score": round(score / len(expected_steps), 2),
        "max_score": 1.0,
        "penalties": penalties,
        "passed": score / len(expected_steps) >= 0.8
    }

# example usage
result = agent.investigate(TEST_CASE["input"])

score = score_trajectory(
    actual_steps=result.steps_taken,
    expected_steps=TEST_CASE["expected_steps"]
)

print(score)
# {"score": 0.85, "max_score": 1.0, "penalties": [], "passed": True}

The function takes two lists: the steps the agent actually took, and the steps you expected it to take.
For each step the agent took, it checks two things: was this step in the expected list at all, and if so, did it happen at roughly the right point in the investigation? If the agent checked check_db_connections first and that was expected first, full credit. If it checked it third when it should have been first, it gets penalised proportionally and scores. After scoring the steps, it checks whether the agent kept going past the point where it should have stopped.

Trajectory scoring handles well-labelled scenarios where you know the expected path. But what happens when the scenario is deliberately designed to mislead?

Adversarial Tests: Forcing The Agent To Reason, Not Pattern Match

Standard test cases check whether the agent handles known scenarios correctly. Adversarial tests go further. They check whether the agent actually reasons or just pattern matches.
The difference matters because production incidents do not arrive cleanly. They arrive with noise, misleading signals, and symptoms that point in the wrong direction. An agent that pattern matches will chase the loudest signal. An agent that reasons will trace the causal chain.
Adversarial tests deliberately inject red herrings to expose which one you have built:

ADVERSARIAL_TEST = {
    "scenario": "Kubernetes OOMKilled - misleading CPU spike",
    "input": {
        "logs": """
            2024-01-15 09:15:22 UTC [WARNING] Container memory usage at 94%
            2024-01-15 09:15:45 UTC [ERROR] OOMKilled: container exceeded memory limit
            2024-01-15 09:15:45 UTC [INFO] Pod restarting...
            2024-01-15 09:16:01 UTC [WARNING] CPU throttling detected on node
        """,
        "metrics": {
            "cpu_throttling_percent": 78,   # looks alarming, is a red herring
            "memory_usage_percent": 94,
            "memory_limit_mb": 512,
            "pod_restarts": 4
        },
        "alert": "High CPU throttling detected"
    },
    "expected_root_cause": "container OOMKilled due to insufficient memory limit",
    "red_herring": "cpu_throttling looks like the main issue but is a downstream symptom",
    "expected_steps": [
        "check_pod_events",
        "check_memory_usage",
        "check_memory_limits",
        "recommend_memory_limit_increase"
    ],
    # agent should NOT go down this path
    "incorrect_path": [
        "check_cpu_throttling",
        "recommend_cpu_limit_increase"
    ]
}

The alert fires on CPU throttling. A pattern-matching agent sees 78% throttling and immediately recommends a CPU limit increase. That is wrong. The CPU throttling is a downstream symptom of the OOMKill restart loop. The real problem is the memory limit being too low. An agent that reasons traces from the OOMKill event back to the memory configuration and stops there.
Now you understand what a test case is, how to score it, and how to stress test the agent against misleading signals. Before you start writing your own, it is worth looking at how others have structured theirs.

Looking At Existing Test Suites Before Building Your Own

Before writing test cases from scratch, you have to look at what others have already built. The structure is often more instructive than the content itself.
The OpenSRE test suite separates scenarios by domain with fixture files containing realistic alert payloads. Reading those fixtures before writing your own will save you several wrong turns on what a well-structured test case should actually contain, what fields matter, how much context to include, and how to frame the expected behaviour clearly enough to grade against.
Two other eval suites worth studying for the structural pattern regardless of your domain:

SWE-bench: how Princeton structured software engineering task evals for coding agents. The input, expected output, graded result pattern maps directly to any agent domain.
AgentBench: benchmark for LLM agents across different environments including OS tasks, database interactions, and web browsing. Useful for seeing how grading works across different action spaces and how to think about pass criteria when the action space is open-ended.

The pattern across all of them is the same: realistic input, defined expected behaviour, graded output. Once you have that pattern clear in your head, the fastest way to build your own scenarios is synthetic.

Synthetic Data: What It Is and When It Plateaus

Synthetic test cases are ones you construct yourself. You write the logs, set the metrics, define the expected answer. You control everything.
This is the right place to start. It is fast, you can cover specific failure modes methodically, and you can design adversarial cases precisely because you decide what the red herrings are.

def generate_synthetic_rds_scenario(failure_type: str) -> dict:
    templates = {
        "slow_query": {
            "logs": [
                "duration: 45231 ms statement: SELECT * FROM orders WHERE status='pending'",
                "autovacuum: found 80000 dead row versions in table orders"
            ],
            "metrics": {"cpu": 35, "db_connections": 45, "latency_ms": 45000},
            "red_herrings": {"db_connections": "normal range, not the cause"},
            "root_cause": "missing index causing full table scan",
            "expected_steps": ["check_slow_queries", "check_query_plans", "recommend_index"]
        },
        "replication_lag": {
            "logs": [
                "replication slot lag: 8GB",
                "WAL sender process waiting for WAL to be archived"
            ],
            "metrics": {"replication_lag_bytes": 8589934592, "cpu": 20, "disk_io": 95},
            "red_herrings": {"cpu": "low, not relevant"},
            "root_cause": "replication lag due to WAL accumulation",
            "expected_steps": [
                "check_replication_status",
                "check_wal_size",
                "check_replica_health"
            ]
        }
    }
    return templates.get(failure_type, {})

The limitation is that synthetic data plateaus. As you add more scenarios the agent improves, but the gains flatten over time. The reason is structural: every scenario you write comes from your own mental model of what can go wrong. You can only write what you can imagine, which means every edge case your synthetic suite does not cover is an edge case your agent has never been tested against.

When gains plateau you have two options: build a different synthetic generator that introduces genuinely new failure patterns, or bring in real world data. Real world cases expose failure modes you never thought to write because they actually happened to someone. That is where the next section comes in.

Where To Get Real World Data For Test Cases

To be clear again: you are not feeding these datasets into the model. You are reading them, understanding the failure patterns, and constructing test cases that reflect what actually happens in production.
The SRE agent is the example we have been using throughout, but the same approach applies to any domain. If you are building an agent that handles customer support tickets, database query optimisation, fraud detection, or any other domain with structured inputs and measurable outcomes, the same process applies: find a labeled dataset in your domain, understand the failure patterns, and turn them into test cases. Here is a list of good dataset sites:

Google Dataset Search
A search engine specifically for datasets. For your domains, search what your agent handles: "customer support tickets", "financial transactions", "medical records."

Kaggle
A large public dataset repository with a lot of labeled data across many domains. It covers finance, healthcare, e-commerce, NLP, and more. Many Kaggle datasets include notebooks showing how others have analysed them, which makes it easier to understand failure patterns before writing test cases.

Once you find a dataset, you load and filter it to find the failure windows, the rows where something actually went wrong.

import pandas as pd

# load a labeled anomaly dataset
df = pd.read_csv("server_machine_dataset.csv")

# filter for labeled failure windows
failure_window = df[df["label"] == 1].head(100)

# the metrics leading up to the failure become your test case input
# the label tells you when the failure occurred
# you construct the expected root cause from the dataset documentation

test_case = {
    "scenario": "server anomaly from SMD dataset machine-1-1",
    "input": {
        "metrics": failure_window[["cpu", "memory", "network_in", "network_out"]].to_dict()
    },
    "expected_root_cause": "network saturation",  # from dataset docs
    "expected_steps": [
        "check_network_throughput",
        "check_active_connections",
        "identify_traffic_source"
    ]
}

One thing worth noting: the expected root cause is not something you derive from the data itself. It comes from the dataset documentation, which for published research datasets will describe what each labeled failure actually was. That documentation is the ground truth your test case is built on. Without it you have inputs but no correct answers to grade against, which means you have data but not a test suite.
The goal across all of this is not to teach the agent. It is to know, with confidence, whether the agent you have built is reliable enough to trust in production.

Implementing Automated Rules-Based Evaluations for LLM Applications

Kalio Princewill — Thu, 05 Feb 2026 13:16:46 +0000

Building software with large language models (LLM) introduces a testing problem that traditional approaches cannot solve. When a function can return different yet equally valid outputs on each invocation, how do you know if it's working correctly?

The standard answer in traditional software is simple: write tests that assert exact outputs. But LLMs are probabilistic systems. Ask them the same question twice and you might get two different phrasings, both correct. Ask them an ambiguous question and you might get a plausible-sounding answer that's completely fabricated (hallucination).

This creates tension. On one hand, LLMs enable applications that would be impossible to build with deterministic code. On the other hand, they behave in ways that make them difficult to test, debug, and deploy with confidence.

This article explores a practical solution: rules-based evaluations integrated into continuous integration pipelines. We will demonstrate this through an AI-powered quiz generator, but the patterns apply to any LLM system where outputs must stay grounded in a specific context.

Why Testing LLM Applications Is Fundamentally Different

Traditional software testing relies on predictability: input X, expect output Y, and assert equality. The test either passes or fails.

LLM applications operate differently. The same prompt can produce multiple valid responses. A request for a quiz about science might correctly return questions about telescopes, physics, or Leonardo da Vinci's inventions depending on what the model selects from the available context.

This probabilistic behavior isn't a bug. But it introduces risks that don't exist in deterministic systems:

Hallucination: The model generates information that sounds plausible but isn't grounded in the provided context. Ask for a quiz about Rome when your dataset only covers Paris, and the model might confidently generate Roman history questions using its pre-training knowledge.
Bias and toxicity: Without constraints, LLMs can produce outputs that reflect societal biases present in their training data or generate harmful content.
Context drift: Even when the model initially behaves correctly, changes to prompts, data, or model versions can introduce regressions that traditional tests won't catch.

These failure modes mean you can't simply write assert output == expected_output. You need evaluation strategies that account for variability while still enforcing correctness.

What Are Evals?

"Evals" are systematic methods for assessing model behavior. While academic benchmarks like MMLU measure general intelligence, they tell you nothing about whether your specific application will hallucinate. A model that scores well on a reading comprehension benchmark might still hallucinate answers when integrated into your customer support bot.

Application-specific evals fill this gap. Instead of testing general capabilities, you test the specific behaviors your application requires:

Does the model only use facts from the provided knowledge base?
Does it refuse requests that fall outside its intended scope?
Does it follow the output format your application expects?

These questions can only be answered by tests you write yourself, tailored to your application's constraints and requirements.

When to Use Automated Evals

Once you understand what evals are, the next question becomes when and how deeply to apply them. Automated evaluations evolve alongside your application, shifting from rapid feedback loops to deep quality assessments.

Development: Rules-based evals provide near-instant feedback on every commit, catching obvious breaks and formatting issues for pennies per run.
Staging: Model-graded evals use "Judge LLMs" to assess nuance and helpfulness on release branches, trading higher costs for deeper insight.
Production: Continuous evals serve as regression detectors, ensuring that prompt tweaks or model updates don't silently degrade performance over time.

Core Evaluation Dimensions for LLM Apps

Regardless of your specific application, most LLM evaluations focus on four dimensions:

Context adherence: Does the output align with the provided context?
Context relevance: Is the retrieved context relevant to the user's query?
Correctness: Does the output align with ground truth or expected behavior?
Bias and toxicity: Does the output contain harmful or offensive content?

Each dimension requires different evaluation techniques. Context adherence might be tested with keyword matching. Bias detection might require a specialized classifier. The implementation depends on your specific constraints.

Case Study: Building a Quiz Generator

To make these concepts concrete, consider an AI-powered quiz generator. The application accepts category requests which include science, geography, or art and returns three quiz questions derived strictly from a predefined dataset.

The critical constraint

The model must never generate questions about subjects outside this dataset. If a user requests a quiz about Rome and the dataset contains no Roman history, the correct behavior is refusal not hallucinated content pulled from pre-training. This constraint makes the application testable. We know exactly what inputs should succeed and which should fail. We can write assertions that verify the model respects these boundaries.

Implementation: Prompt Design

Most LLM behavior is controlled by the prompt. A well-designed prompt establishes rules, provides context, and defines failure modes:

system_message = f"""
Follow these steps to generate a customized quiz for the user.

Step 1: Identify the category from: Geography, Science, or Art

Step 2: Select up to two subjects from the quiz bank that match the category

Step 3: Generate 3 questions using only the facts provided

Additional rules:
- Only use explicit matches for the category
- If no information is available, respond: "I'm sorry I do not have information about that"
"""

This structure breaks generation into discrete steps, lists valid categories explicitly, and defines refusal behavior preventing hallucination.

The LLM Pipeline

LangChain provides composable primitives for building the application:

def assistant_chain(
    system_message=system_message,
    llm=ChatOpenAI(model="gpt-3.5-turbo", temperature=0),
    output_parser=StrOutputParser()):

    chat_prompt = ChatPromptTemplate.from_messages([
        ("system", system_message),
        ("human", "{question}"),
    ])
    return chat_prompt | llm | output_parser

This pipeline serves double duty: it powers the production application and provides a stable interface for automated evaluations.

Writing Rules-Based Evaluations

Rules-based evals check for specific patterns in outputs. Here are three essential patterns:

1. Keyword Matching: Content Validation

Verifies the output contains expected domain terms:

def eval_expected_words(system_message, question, expected_words):
    assistant = assistant_chain(system_message)
    answer = assistant.invoke({"question": question})

    assert any(word in answer.lower() for word in expected_words), \
        f"Expected words {expected_words} not found in output"

# Test case
def test_science_quiz():
    question = "Generate a quiz about science."
    expected_subjects = ["davinci", "telescope", "physics", "curie"]
    eval_expected_words(system_message, question, expected_subjects)

This assertion is deliberately loose. It passes if *any* expected word appears, accommodating the model's freedom to select different subjects from the valid set. You're not testing for exact phrasing, you're verifying that the model chose appropriate content.

2. Refusal Testing: Hallucination Prevention

Verifies the model declines out-of-scope requests:

def evaluate_refusal(system_message, question, decline_response):
    assistant = assistant_chain(system_message)
    answer = assistant.invoke({"question": question})

    assert decline_response.lower() in answer.lower(), \
        f"Expected refusal with '{decline_response}', got: {answer}"

# Test case
def test_refusal_rome():
    question = "Help me create a quiz about Rome"
    decline_response = "I'm sorry"
    evaluate_refusal(system_message, question, decline_response)

When asked about Rome a topic outside the dataset the model should respond with the specified refusal phrase. If it generates a quiz instead, the test fails, signaling that the prompt's guardrails aren't working.

3. Format Validation: Structural Correctness

Verifies the output follows expected structure using regex patterns:

import re

def eval_quiz_format(system_message, question, expected_questions=3):
    """Verify the quiz follows delimiter-based format"""
    assistant = assistant_chain(system_message)
    answer = assistant.invoke({"question": question})

    # Pattern to match "Question 1:####", "Question 2:####", etc.
    pattern = r'Question\s+\d+:####'
    matches = re.findall(pattern, answer)

    assert len(matches) == expected_questions, \
        f"Expected {expected_questions} questions in format 'Question N:####', " \
        f"found {len(matches)}"

# Test case
def test_science_quiz_format():
    question = "Generate a quiz about science."
    eval_quiz_format(system_message, question, expected_questions=3)

LLMs can "drift" and start returning different formats. If your quiz generator is supposed to use delimiters and suddenly returns markdown or plain text, downstream parsing breaks. Format enforcement prevents the model from returning markdown-wrapped or JSON-formatted output when plain text is expected:

def eval_output_type(system_message, question, should_be_plain_text=True):  
    """Verify output format matches expectations"""  
    assistant = assistant_chain(system_message)  
    answer = assistant.invoke({"question": question})  

    if should_be_plain_text:  
        # Should NOT be wrapped in markdown code blocks  
        assert not answer.strip().startswith('\`\`\`'), \\  
            f"Expected plain text, got markdown-wrapped output"  

        # Should NOT start with JSON/dict characters  
        assert not answer.strip().startswith('{'), \\  
            f"Expected plain text, got JSON-like output"

# Test case  
def test_output_is_plain_text():  
    question = "Generate a quiz about art."  
    eval_output_type(system_message, question, should_be_plain_text=True)

This catches a common LLM failure mode: returning content wrapped in markdown code fences (text ...) or unexpectedly switching to JSON format.

Format tests catch issues that keyword tests miss:

Format drift: Model updates or prompt changes cause output structure to shift
Integration failures: If your frontend expects JSON and the LLM returns markdown, your app breaks
Faster debugging: When a test fails, you immediately know whether it's a content issue or a format issue.

Integrating Evals into Continuous Integration

Manual testing doesn't scale. As teams grow and development accelerates, the discipline required to run tests before every commit erodes. This is even more critical for LLM applications because:

Prompt changes are easy to make but hard to validate without testing
Model behavior can shift with version updates
Context changes affect outputs in non-obvious ways

By integrating evals into CI, every change is automatically validated before it reaches production.

The GitHub Actions Workflow

Every push and pull request triggers automated evaluation:

name: Run Evaluations

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  run-evals:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run evaluation tests
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: pytest test.py -v

The OpenAI API key is stored as a repository secret, keeping credentials secure while allowing tests to make real API calls. When you push code, GitHub Actions automatically runs your evaluation suite:

The workflow shows all tests passing. Any test failure blocks the merge, preventing broken behavior from reaching production.

Limitations of Rules-Based Evaluations

Rules-based evaluations have clear boundaries. They can verify patterns and keywords but cannot assess:

Factual accuracy beyond keyword presence: The test verifies "telescope" appears but not whether the telescope facts are correct
Output quality: A response might contain correct keywords but be poorly phrased or confusing
Subtle hallucinations: The model might include expected keywords while inserting fabricated details between them

The Path Forward

For production systems, extend this foundation with:

Model-graded evaluations: Use a second LLM to assess subjective qualities (helpfulness, clarity, tone)
Bias and toxicity detection: Integrate specialized classifiers
A/B testing: Compare prompt variations with data-driven metrics
Production observability: Monitor outputs continuously; some failure modes only emerge at scale

The strategy: Run rules-based evals on every commit, model-graded evals on release branches, and continuous monitoring in production.

Conclusion

LLM applications don't have to be opaque or untestable. By combining explicit constraints, semantic evaluations, and automated execution, you transform probabilistic models into reliable system components. The workflow is straightforward and repeatable:

Define constraints in prompts
Write tests that verify those constraints (keyword, format, refusal)
Run evaluations automatically on every change
Use rules-based evals for speed, model-graded evals for depth

The result becomes a software you can iterate on quickly, deploy confidently, and maintain sustainably even when it's powered by probabilistic models. See full implementation here (https://github.com/iamkalio/llmops-eval-ci)

Understanding Vectors and Vector Search: How Vector Search Understands What You Really Mean

Kalio Princewill — Mon, 20 Oct 2025 09:42:33 +0000

Introduction

Imagine you're an engineer looking to optimise your project's database. You might start by searching for "B-Tree indexing" or "database query optimisation." A traditional search engine, operating on keyword-matching principles, would scan its index for documents containing those exact phrases.

But what if the most groundbreaking article on the topic never uses those words, instead referring to the concept as "SQL tuning strategies" or "efficient data retrieval patterns"? With a simple keyword search, that invaluable resource would remain invisible. This highlights the fundamental limitation of traditional search: it matches words, not ideas. It fails to understand context, intent, or the subtle semantic relationships that connect concepts.

This conflict between human intent and machine literalism creates a clear need for a more intelligent, intuitive way to discover knowledge.

In this blog, we'll discuss everything related to vector search, starting with the basic concepts and then moving on to more advanced techniques. Let's begin by overviewing vectors and embeddings.

Vectors and Embedding

To understand how a machine can grasp the concept of "similarity," we first need to understand how it represents information. Humans use words and images, but computers speak in numbers. Modern AI lies in its ability to translate our rich, unstructured world into a mathematical format it can process. This translation is achieved through a concept known as vector embeddings.

What is Vector Embedding?

Vector embedding is a numerical representation of an object: be it a word, a sentence, a complete document, an image, or a piece of audio. These numbers aren't random; they are generated by a machine learning model trained to capture the object's true semantic meaning and context. This process effectively maps data into a high-dimensional space where related items are positioned closer together.

The relationships captured within this high-dimensional space are so precise that they even support arithmetic operations. A classic demonstration of this is the equation: vector('king') - vector('man') + vector('woman') ≈ vector('queen').
This reveals how the model has learned to associate abstract concepts like gender and royalty in a calculable way.

These numerical representations are technically known as dense vectors, arrays in which most values are non-zero. It's this dense structure that allows them to encode such a rich amount of semantic information, making them the foundational element for any vector search system.

The Vectorisation Process

Creating these embeddings is a structured process known as vectorisation, which turns raw data into meaningful numerical representations. It follows a clear and methodical path outlined in the section.

Data Preparation: Raw data is messy. The process begins by cleaning and standardising the raw data. This step ensures the model receives high-quality input, whether that means removing irrelevant characters from text or resizing an image to a uniform dimension.
Model Selection: The heart of the vectorisation process is the embedding model. An embedding model, pre-trained on vast datasets, is selected to act as a translator. Popular models include BERT (Bidirectional Encoder Representations from Transformers) or Sentence-BERT, which are designed to understand the nuances of text. Similarly, CLIP (Contrastive Language--Image Pre-training), trained on vast pairings of images and their textual descriptions, is used for interpreting the content of images.
Generation: The prepared data is fed into the selected model. After processing the input through its complex layers, the model outputs the final vector embedding, the data's unique mathematical fingerprint that captures its semantic meaning.
Storage & Indexing: Finally, the newly created vectors are loaded into a specialised vector database, such as Pinecone, Chroma, etc. This system is purpose-built to store, manage, and index millions of these high-dimensional vectors for fast search and retrieval.

The Engine: How Vector Search Works

With our data translated into vectors and stored, the search process can begin. The core principle is quite intuitive: it's all about finding the closest matching vector in a high-dimensional space(a mathematical space defined by many features, like hundreds or thousands of axes, where similar items (vectors) are placed close together). This section breaks down how the engine finds the most relevant results with remarkable speed.

a. The Fundamental Principle: Finding Similarity
When you make a query, whether it's text or an image, it's also converted into a vector using the very same model. The system's job is then to find the vectors in the database that are the closest "neighbors" to your query vector. This process of finding the most similar items based on proximity is known as a Nearest Neighbor (NN) search. The original items corresponding to these neighboring vectors are the results you see.

b. Measuring "Closeness": Distance Metrics
To find the "closest" neighbors, the system needs a way to measure the distance or similarity between two vectors. While there are many methods, two are predominantly used:

Euclidean Distance: This is the most straightforward approach. It measures the straight-line distance between two vector points in the high-dimensional space, much like finding the distance between two cities on a map. A smaller distance means the items are more similar.
Cosine Similarity: Instead of distance, this metric measures the angle between two vectors. It focuses on their orientation, not their magnitude. A smaller angle results in a similarity score closer to 1, indicating a strong match. This is especially useful for text, as it can recognise that a short headline and a long article are about the same topic if they point in the same conceptual direction.

c. The Challenge of Scale: Why Brute-Force Isn't Enough
Calculating the distance between a few vectors is easy. But what about a database with millions or even billions of them? Comparing your query vector to every single one is known as a brute-force search (native search). While this method is perfectly accurate, it becomes incredibly slow and computationally expensive at scale, making it impractical for the real-time results we expect from modern applications.

d. The Solution for Speed: Approximate Nearest Neighbor (ANN)
To overcome this performance bottleneck, vector search employs a clever trade-off: Approximate Nearest Neighbor (ANN). Instead of finding the exact nearest neighbors, ANN algorithms use efficient indexing techniques to find vectors that are almost the nearest, and they do it exponentially faster.

For most applications, this compromise is ideal. It delivers highly relevant, near-perfect results in milliseconds, avoiding the impossible computational cost of a brute-force search and making vector search a practical, powerful tool.

Vector Search in Action: Real-World Applications

The true impact of this engine is seen in the growing number of applications it powers. By understanding meaning, vector search makes our digital experiences more intuitive, relevant, and intelligent.

Semantic Search

This is the most direct application. Instead of being limited by keywords, you can find what you mean, not just what you type. For example, an e-commerce site can interpret a search for "something warm to wear on a cold hike" and return results for fleece jackets, thermal layers, and wool socks, even if those exact words aren't in the product descriptions.

Recommendation Systems

Ever wonder how Spotify discovers your next favourite artist or Netflix suggests the perfect movie? These platforms often use vector search. They create a vector representing your unique taste based on your activity. The recommendation engine then searches for items which include songs, movies, or products with vectors that are closest to your taste profile, delivering highly personalised suggestions.

Image & Visual Search

Vector search allows for content-based retrieval. Instead of trying to describe a pattern or style with words, you can use an image as your query. A user could upload a photo of a chair they like and instantly find visually similar chairs from an online furniture catalog, a task that would be nearly impossible with keywords alone.

Retrieval-Augmented Generation (RAG)

This is a critical application for improving Large Language Models (LLMs) like ChatGPT. LLMs are powerful but can be out-of-date or invent incorrect facts ("hallucinate"). RAG uses vector search as a fact-checker. When you ask a question, the RAG system first uses vector search to find the most relevant, factual documents from a private or updated knowledge base. It then gives that information to the LLM along with your question, instructing it to generate an answer based only on the provided facts. This makes the LLM's responses dramatically more accurate and trustworthy.

Hybrid Search

Hybrid search offers the best of both worlds. It combines the contextual understanding of vector search with the precision of traditional keyword search. For example, when searching for a specific product like an "iPhone 15 Pro case," the keyword "iPhone 15 Pro" is essential. Hybrid search ensures exact matches are prioritised while also using vector similarity to find the most relevant styles and features, delivering the most accurate results.

Conclusion

Ultimately, vector search represents a fundamental shift in how we interact with information. It moves us beyond literal keyword matching into a more fluid, human-like understanding of context and meaning. Powered by vector embeddings and the speed of ANN algorithms, this technology is the engine behind a new generation of smarter applications. It's how search tools finally learn to understand intent, how recommendation engines predict our needs, and how AI can be grounded in fact. It's the technology that allows applications to stop just matching words and start understanding our world.

References

https://www.elastic.co/what-is/vector-search

https://www.coveo.com/blog/what-is-vector-search/

https://www.oracle.com/africa/database/vector-search/

https://www.ibm.com/think/topics/vector-search

https://medium.com/@myscale/what-is-vector-search-7d39b7e4d77a

https://weaviate.io/blog/vector-search-explained

Load Testing a Scalable AWS Application Using Grafana k6

Kalio Princewill — Wed, 28 May 2025 09:06:00 +0000

In cloud-native applications, performance under load is a critical consideration. Ensuring applications can handle expected traffic and unexpected surges is key to maintaining a positive user experience and system stability. Load testing is essential in validating the resilience and scalability of modern applications.

This tutorial will demonstrate how to use Terraform to set up a scalable application infrastructure on Amazon Web Services (AWS). After the infrastructure has been provisioned, we will use Grafana K6 to conduct a variety of load tests, and we'll execute smoke, average, and spike load tests on the application. We will analyse the performance metrics captured during these tests, such as latency, throughput, and success rates, to understand how the architecture handles different traffic patterns and how its components, like the Auto Scaling Group and Application Load Balancer, contribute to its ability to scale and maintain reliability under stress.

Prerequitises

To effectively follow this guide, ensure you have the following prepared:

AWS Account. You'll need an active account, which you can sign up for on the AWS website if you don't have one.
Configured AWS CLI. Install the AWS Command Line Interface from the official AWS CLI installation guide and configure it with your credentials by following the AWS CLI configuration guide.
Terraform CLI Installed: Download and install the Terraform CLI from the official HashiCorp Terraform installation page.
Grafana k6 Installed: Install the k6 load testing tool by downloading it from the k6.io website.
AWS SSH Key Pair: Create an SSH key pair in your AWS region for EC2 access via the EC2 console, and note its name for your Terraform setup.
Basic Terraform Understanding: Familiarise yourself with Terraform's core concepts through the Terraform documentation.
Sufficient IAM Permissions: The AWS identity (user or role) that Terraform uses must have permissions to create and manage resources for this project, primarily involving services like VPCs, EC2 instances, ALBs, Auto Scaling Groups, Security Groups, and Subnets. While a broad policy like AdministratorAccess can be used for initial learning in a personal account (use with caution), the best practice for any environment is to apply the principle of least privilege. This involves creating a custom IAM policy with only the specific permissions needed for Terraform to manage these resources. You can learn how to do this by reviewing the AWS guide on creating IAM policies. If Terraform encounters permission errors during execution, the error messages will typically guide you on what specific permissions are missing.

Overview of the Scalable Infrastructure

Before we dive into provisioning, let's understand the architecture we'll be building on AWS. This setup is designed for scalability and high availability, ensuring our application can handle varying loads effectively. The key components, as illustrated in the diagram below, include:

Amazon Web Services (AWS) Cloud: Our infrastructure resides entirely within the AWS cloud, leveraging its robust and scalable services.
Virtual Private Cloud (VPC): We'll establish a custom VPC, which acts as a private, isolated section of the AWS cloud. This provides a secure network environment for our resources.
Internet Gateway (IGW): An IGW is attached to our VPC to allow communication between resources in our VPC and the internet. This is crucial for users to access our application.
Public Subnets across Multiple Availability Zones (AZs): Within our VPC, we will configure public subnets. To ensure high availability and fault tolerance, these subnets will be distributed across multiple Availability Zones (e.g., AZ a, AZ b, AZ c). If one AZ experiences an issue, our application can continue running in other AZs.
Application Load Balancer (ALB): The ALB serves as the single point of contact for clients and automatically distributes incoming application traffic across multiple targets, such as EC2 instances, in different Availability Zones. This enhances application availability and fault tolerance.
EC2 Instances: These are virtual servers in the AWS cloud where our simple web application will run. The application itself will be deployed using EC2 user data scripts.
Auto Scaling Group (ASG): The ASG is the core of our scalability. It automatically adjusts the number of EC2 instances running our application based on predefined conditions (like CPU utilisation or network traffic). If the load increases, the ASG launches more instances; if the load decreases, it terminates instances to save costs.
Target Group: The Application Load Balancer uses this group to route requests to one or more registered targets, which in our case are the EC2 instances managed by the Auto Scaling Group. The ALB checks the health of instances in the target group and only sends traffic to healthy instances.
EC2 Security Group: This acts as a virtual firewall for our EC2 instances, controlling inbound and outbound traffic. We'll configure it to allow traffic from the Application Load Balancer and necessary administrative access (e.g., SSH, if needed for debugging, though not directly used by the load test traffic itself).

This architecture ensures that incoming internet traffic passes through the IGW to the ALB. The ALB then distributes this traffic across healthy EC2 instances running in public subnets across different AZs. The Auto Scaling Group monitors these instances and scales the fleet in or out based on demand, providing both resilience and cost-effectiveness. Our Terraform scripts will define and provision all these components.

Provisioning the infrastructure with Terraform

This section outlines the steps in provisioning your application's infrastructure on AWS using Terraform. Terraform allows you to define your infrastructure as code, making the provisioning process repeatable, predictable, and versionable.

Cloning the infrastructure repository

The first step is to obtain the Terraform code that defines your AWS infrastructure.

1.Open your terminal.

2.Navigate to the directory where you want to clone the repository.

3.Clone the repository containing the Terraform configuration. Run

  git clone https://github.com/iamkalio/infra-load-testing

4.Change into the cloned repository directory:

  cd infra-load-testing/

This repository contains the Terraform configuration files (.tf files) that describe the desired state of your AWS resources.

Provisioning your infrastructure

Once you have the infrastructure code locally, you can use Terraform to provision the resources on AWS. Ensure you have the AWS CLI configured with appropriate credentials and permissions to create resources in your desired region.

1.Initialise your working directory: To initialise the directory and download the necessary provider plugins, run this command:

  terraform init

On successful initialisation, the .terraform and .terraform.lock.hcl files would be added to your directory.

2.Review the execution plan: Before making any changes to your infrastructure, review the plan that Terraform will execute. To see which resources will be created, modified, or destroyed, run:

  terraform plan

Carefully examine the output to ensure that Terraform intends to make the changes you expect.

3.Apply the configuration: Apply the configuration to provision the resources on AWS. Run this command to begin the process:

  terraform apply

Terraform will prompt you to confirm the action. Type yes and press Enter to proceed. Terraform will then create the defined infrastructure resources in your AWS account.

Pasting the ALB DNS name into your browser lets you view your active application.

Note the ALB DNS from the output

After a successful terraform apply, Terraform will output the values of any defined outputs in your configuration. It is common practice to output the DNS name of the Application Load Balancer (ALB), as this is the entry point for traffic directed towards your application running on the provisioned infrastructure.

Look for the outputs section in the console output after terraform apply completes. Identify and note down the DNS name listed for your ALB ( alb_dns_name). This is the URL you will use to access your deployed application once it's running on the provisioned infrastructure.

To retrieve this output value at any time after provisioning, run:

  terraform output alb_dns_name

Showcasing the infrastructure that is created on AWS

After successfully running terraform apply, the infrastructure defined in your code will be active in your AWS account. You can verify and explore the created resources using the AWS Management Console or the AWS CLI.

The infrastructure provisioned includes:

Virtual Private Cloud (VPC): A logically isolated network for your resources.

Subnets: Divisions within your VPC (e.g., public subnets for load balancers and private subnets for application instances and databases). In this case, it is a public subnet.

Internet Gateway: Allows communication between your VPC and the internet (usually associated with public subnets).

Route tables: Control the routing of network traffic within your VPC and to the internet.

Security groups: Act as virtual firewalls, controlling inbound and outbound traffic for your instances.

Application Load Balancer (ALB): Distributes incoming application traffic across multiple targets, such as EC2 instances or containers.

ALB target group: A logical grouping of targets (e.g., EC2 instances) that the ALB routes traffic to.

Compute resources: EC2 instances, ECS tasks, or EKS pods where your application will run.

Auto Scaling group: Manages the desired number of healthy instances for your application.

Running the load tests

Now that your application infrastructure is provisioned, you can create a load test script using k6 to simulate user traffic. This script will define the behaviour of virtual users accessing your application through the Application Load Balancer (ALB). Our load testing process will include a smoke test, an average load test, and a spike test to evaluate the application's performance across different scenarios.

Performing a smoke test

Let's begin with a quick smoke test. Smoke test is a lightweight load test designed to verify that your application is running, accessible, and responds correctly to basic requests. It's a crucial first step to catch any fundamental issues early on.

You will need a place to save your k6 test scripts. It's a good practice to keep your load test scripts separate from your infrastructure code.

1.Navigate to a suitable location outside of your Terraform infrastructure directory.

2.Create a directory for your load test scripts. You can name it load-tests

  mkdir load-tests

3.Change into the load-tests directory:

  cd load-tests

4.Create a new file named smoke-test.js.

5.Populate this file with the following k6 script. Remember to replace http://YOUR_ECB_ALB_DNS_NAME/ with the actual DNS name of your ALB.

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
    vus: 3, // Key for Smoke test. Keep it at 2, 3, max 5 VUs
    duration: '1m', // This can be shorter or just a few iterations
};

export default () => {
    // Replace 'http://YOUR_ECB_ALB_DNS_NAME/' with your actual public IP and port
    const urlRes = http.get('http://YOUR_ECB_ALB_DNS_NAME/');
    check(urlRes, { 'status is 200': (r) => r.status === 200 });
    sleep(1);
};

This script sets up a basic test scenario. It uses import statements for the necessary k6 modules. The options object configures the test to use a small number of virtual users (vus: 3) for a short duration: '1m'. The default function defines the actions each virtual user will perform: making an HTTP GET request to your application's URL, using check to ensure the response status is 200, and pausing with sleep(1).

To run the script, open your terminal, ensure you are in your load-tests folder, and execute the script:

k6 run smoke-test.js

After the test completes, k6 will output results summarising performance. For this smoke test, with 3 virtual users running for approximately 1 minute, the results confirmed the application's basic functionality and responsiveness:

HTTP request duration: The average request duration was 238.68ms, with 95% of requests completing within 279.67ms. The maximum duration was 852.69ms.
Checks and failures: All 143 checks succeeded, resulting in a 100.00% success rate and 0.00% HTTP request failures.
Throughput: The test generated a total of 143 HTTP requests at a rate of approximately 2.36 requests per second.

These results demonstrate a healthy and responsive application under light load. Concurrently, observing your AWS CloudWatch metrics during the test will show the request count received by your Application Load Balancer (ALB).

Performing an average load test

Next, simulate an average level of traffic that your application might typically handle. This test runs over a longer duration and gradually increases the load to sustained levels, providing insights into performance under expected conditions.

1.Create a new file named average-load-test.js in your load-tests folder.

2.Add the following k6 script to it, making sure to replace the placeholder URL with your actual target (the ALB DNS name).


import  http  from  'k6/http';

import { sleep, check } from  'k6';

export  const  options  = {

   // Key configurations for avg load test in this section

   stages:  [

{ duration:  '5m', target:  100 }, // traffic ramp-up from 1 to 100 users over 5 minutes.

{ duration:  '30m', target:  100 }, // stay at 100 users for 30 minutes

{ duration:  '5m', target:  0 }, // ramp-down to 0 users

   ],

};

export  default () => {

   const  urlRes  =  http.get('http://YOUR_ALB_DNS_NAME/');

   check(urlRes, { 'status is 200': (r) =>  r.status  ===  200 });

   sleep(1);

};

What does this script do?

This script uses the stages option to define different phases of the test, simulating a more realistic traffic pattern over time:

Ramp-up (5 minutes): The number of virtual users linearly increases from the default (usually 1) up to target: 100.
Sustained load (30 minutes): The test maintains a constant load of target: 100 virtual users. This is the core of the average load test, measuring performance under steady, typical traffic.
Ramp-down (5 minutes): The number of virtual users decreases from 100 down to target: 0, gracefully ending the test.

The default function remains the same as the smoke test, performing the basic request and status check for each user.

3.Execute the average load test script from your load-tests folder:

  k6 run average-load-test.js

Average load test results:

This test ran for approximately 40 minutes and aimed to simulate 100 virtual users. The results show 168,556 total HTTP requests were made, with 99.99% of checks succeeding. The average HTTP request duration was 245.49ms, with 95% of requests completing within 301.52ms. Notably, the test recorded a very high maximum request duration of 1 minute and 8 seconds, and some Request Failed warnings indicated timeouts. Despite these outliers, the overall success rate for HTTP requests was 0.00% failed (representing 2 failures out of over 168,000 requests), indicating the application largely handled the sustained load.

Concurrently, CloudWatch metrics for "Total requests acknowledged" by your ALB showed the traffic pattern of ramp up, sustained load, and ramp down, confirming the load balancer successfully received the requests. The total requests recorded by CloudWatch over this period exceeded 307,000, reflecting the cumulative traffic handled by the ALB.

Perform a spike load test

Finally, conduct a spike test. This simulates a sudden, dramatic surge in traffic. It's designed to reveal how your application behaves under extreme bursts of load and its ability to recover.

1.Create a new file named spike-load-test.js in your load-tests folder.

2.Add the following k6 script, replacing the placeholder URL with your target (ALB DNS name).

import  http  from  'k6/http';

import { sleep, check } from  'k6';

export  const  options  = {

   // Key configurations for spike in this section

   stages:  [

{ duration:  '2m', target:  2000 }, // fast ramp-up to a high point

       // No plateau

{ duration:  '1m', target:  0 }, // quick ramp-down to 0 users

   ],

};

export  default () => {

   // Replace 'http://YOUR_ALB_DNS_NAME/' with your actual ALB DNS name

   const  urlRes  =  http.get('http://YOUR_ALB_DNS_NAME/');

   check(urlRes, { 'status is 200': (r) =>  r.status  ===  200 });

   sleep(1);

};

The spike load script uses stages to create a sudden, sharp increase and decrease in virtual users:

Rapid ramp-up (2 minutes): The number of virtual users quickly increases from the default up to a very high target: 2000. This simulates a sudden traffic event.
Rapid ramp-down (1 minute): The number of users drops just as quickly back to target: 0. There is no sustained load phase in a typical spike test.

The default function remains the same, performing the request and status check.

3.Execute the spike test script from your load-tests folder:

  k6 run spike-load-test.js

This test completed relatively quickly, running for approximately 3 minutes. Under such a rapid and high surge in traffic, the test successfully simulated up to 2000 virtual users. The results show 138,618 total HTTP requests were made, with a 100.00% success rate for all checks and 0.00% HTTP request failures. The average HTTP request duration was 383.01ms, with 95% of requests completing within 646.53ms. The maximum duration observed was 4.29 seconds, indicating some requests experienced noticeable delays during the peak load. The system sustained a high throughput of approximately 766 requests per second during the test.

CloudWatch metrics for "Total requests acknowledged" by your ALB clearly showed a sharp peak of over 120,000 requests during the spike, followed by a rapid drop-off. This pattern mirrored the intense traffic load generated by k6, confirming that the ALB successfully handled the sudden surge in traffic.

Analysing Performance

The k6 load tests provide insight into how the provisioned infrastructure performs under varying traffic.

The architecture works by having the ALB distribute incoming requests across multiple instances. To handle increased traffic, the ASG automatically adjusts the number of instances running your application. Automatic scaling is driven by CloudWatch metrics like requests per target or CPU utilisation. When these metrics cross defined thresholds, CloudWatch alarms trigger scaling policies in the ASG. This tells the ASG to launch new instances. These new instances automatically register with the ALB, and the load is spread further.

This scaling process directly impacts the performance metrics observed in your k6 tests in several ways. As the ASG adds more instances, the load on each instance decreases. This allows requests to be processed faster, reducing overall latency, especially during traffic spikes. With more capacity available, instances are less likely to become overwhelmed. This reduces errors and dropped connections, leading to a higher percentage of successful requests even under significant load. Increased instances also mean the system can handle a higher total volume of requests per second sustainably, known as throughput. The ALB's health checks also ensure traffic only goes to healthy instances, contributing to overall reliability.

The k6 test results, particularly from the average and spike tests, demonstrate how effectively the ASG responds to handle the load. By scaling out, the architecture maintains performance and high reliability even during bursts of traffic. This dynamic adjustment of resources based on demand is a key benefit of the scalable infrastructure provisioned by Terraform.

Best practices for load testing applications

To ensure effective load testing and derive meaningful insights from your applications, follow these best practices:

Always test in a staging or test environment: Avoid running load tests directly on production systems unless necessary and with extreme caution. Use an isolated environment that closely mirrors your production setup.
Use realistic traffic patterns: Design your k6 scripts to mimic real user behaviour as closely as possible, including request types, request frequency, and user concurrency.
Start small and gradually increase the load: Begin with a low number of virtual users and gradually ramp up the load. This helps you identify bottlenecks incrementally and observe how your system scales.
Monitor AWS metrics alongside k6 output: Combine the client-side metrics from k6 with server-side metrics from AWS CloudWatch. This provides a holistic view, showing how your infrastructure (ALB, EC2/ECS/EKS, ASG, RDS) responds to the load.
Consider chaos testing for resilience validation: Once stability under load is confirmed, introduce controlled failures (e.g., terminating instances) to validate your system's resilience and recovery capabilities.

Cleaning up your provisioned resources

After completing your load tests and analysis, it is crucial to tear down the infrastructure you provisioned with Terraform to avoid ongoing AWS charges.

1.Navigate back to your Terraform infrastructure directory.

2.Destroy the provisioned resources by running the terraform destroy command:

    terraform destroy

3.Terraform will display a plan of all resources that will be destroyed. Type yes and press Enter to confirm the destruction of your AWS infrastructure.

This command will safely de-provision all the AWS resources that Terraform created, ensuring no unnecessary costs are incurred.

Conclusion

In this guide, we demonstrated an approach to validating the performance and scalability of cloud-native applications. We covered the entire process, including setting up a cloud infrastructure on AWS using Terraform, performing various load tests with Grafana k6, from basic smoke tests to more intensive average and spike scenarios, and finally, analyzing the performance metrics to understand how this architecture effectively handles diverse traffic conditions, especially through its auto scaling capabilities driven by CloudWatch.

The key takeaway is that performance under load is paramount for modern applications. Implementing a scalable infrastructure with components like auto scaling groups is critical for ensuring reliability, consistent latency, and high throughput. Combining regular load testing with robust monitoring is essential to confirm your application can dynamically adapt to demand and provide a seamless user experience.

Provisioning an AWS EC2 Instance with Terraform

Kalio Princewill — Sat, 24 May 2025 20:27:43 +0000

Introduction

Infrastructure as Code (IaC) is becoming a key part of modern cloud engineering. It lets developers and DevOps engineers provision and manage cloud resources consistently and efficiently. Among the most popular IaC tools, Terraform stands out for its declarative syntax, strong community support, and multi-cloud capabilities.

This article provides a step-by-step guide to help you provision your first EC2 instance on AWS using Terraform. Whether you're a beginner exploring Terraform for the first time or someone looking to solidify your AWS provisioning workflow, this guide walks you through the entire process, from setting up your AWS environment to writing and applying Terraform configuration files. By the end, you'll understand how to securely configure access, provision an EC2 instance, and clean up resources while adhering to best practices.

Prerequisite

To get the most out of this article, you must have the following:

An AWS Account: You need an active AWS account. If you don't have one, sign up on the AWS website.
AWS CLI: Install the AWS Command Line Interface from the official AWS CLI installation guide.
Terraform CLI Installed: Download and install the Terraform CLI from the official HashiCorp Terraform installation page. Verify your installation by running terraform --version in your terminal.
Terraform extension (optional): Install VS Code and the official Terraform extension for improved syntax highlighting and features.

Setting Up Your AWS Environment

Before Terraform can provision resources on AWS, it needs to authenticate with your AWS account. You'll generate an Access Key ID and a Secret Access Key using AWS Identity and Access Management (IAM) for this purpose. Follow these steps:

Create an IAM User

To securely manage access to your AWS resources, begin by creating a dedicated IAM user that Terraform will use for authentication. This will ensure that you avoid using your root account credentials.

1.Log in to your AWS Management Console. In the search bar, type IAM and select the IAM service.

2.On the left sidebar, click on Users, then select Create User.

3.Enter a preferred username (e.g.terraform-admin) and click Next.

4.On the Set Permissions page, attach the AdministratorAccess policy directly to the user. For this tutorial, we're using AdministratorAccess for simplicity, but in a production environment, always apply the Principle of Least Privilege by granting only the minimum necessary permissions. If you already have an IAM group with the required permissions, you can add the new user to that group instead.

5.Review the permissions, click Next, and then Create User. Your IAM user is now set up.

Create an Access Key

With the new IAM user in place, the next step is to generate the Access Key ID and Secret Access Key, which will allow Terraform to authenticate and interact with AWS services.

1.Navigate back to the IAM service and click on Users in the left sidebar.

2.Select the newly created user (terraform-admin) to open its details page.

3.Go to the Security Credentials tab.

4.In the Access Keys section, click on Create Access Key.

5.When prompted to select a use case, choose Command Line Interface (CLI).

6.Copy and securely store the Access Key ID and Secret Access Key displayed. After this step, AWS will not show the secret key again.

Configure AWS CLI on Your Local Machine

The next step is to configure your local machine to authenticate with AWS using the AWS Command Line Interface (CLI). This allows Terraform to communicate with AWS services seamlessly.

1.Open your terminal.

2.Run the following command on your terminal


aws configure

3.Enter the following details when prompted:

AWS Access Key ID: Paste the Access Key ID you copied.
AWS Secret Access Key: Paste the Secret Access Key you copied.
Default region name: Enter us-east-1 (or your preferred AWS region).
Default output format: Enter json.

Your local machine is now configured to authenticate with AWS using the AWS CLI, allowing Terraform to communicate seamlessly with AWS services.

Creating your Terraform configuration for EC2 deployment

This section guides you through writing your Terraform configuration to provision an EC2 instance on AWS. You'll set up your project, define the AWS provider, create an SSH key pair, configure a security group, define your EC2 instance, and add useful outputs and variables.

Project Setup

1.Create a new directory for your Terraform project. Open your terminal and run:


mkdir terraform-project && cd terraform-project

Generate an SSH Key Pair

To securely connect to your EC2 instance using SSH, you need an SSH key pair. You'll generate this on your local machine and then upload the public key to AWS through Terraform.

1.Open your terminal.

2.Generate a new SSH key pair by running:


ssh-keygen -t ed25519 -f ~/.ssh/my-ec2-key -C "my-ec2-key"

This command creates an Ed25519 key pair, which is a modern and highly secure.

-f ~/.ssh/my-ec2-key flag specifies the file path and name for your private key (my-ec2-key) and its corresponding public key (my-ec2-key.pub).
-C "my-ec2-key" flag adds a descriptive comment to the key, making it easier to identify later.

3.When prompted for a passphrase, you can press Enter to leave it blank for simplicity in this tutorial. However, in a production environment, always use a strong passphrase to protect your private key.

This command generates two files in your ~/.ssh directory:

my-ec2-key: This is your private key. Keep this file absolutely secure and never share it.
my-ec2-key.pub: This is your public key. You will upload the contents of this file to AWS through Terraform.

Defining Your Terraform Configuration Files

To provision the EC2 instance, you'll organise your Terraform configuration into several core files, each serving a specific purpose:

Define Terraform Variables

Variables make your Terraform configurations flexible and reusable. Define them upfront to easily customise your deployment without altering the main resource definitions.

Create a new file named variables.tf in your terraform-project directory.
Add the following content to variables.tf:


variable  "aws_region" {

 description =  "The AWS region to deploy resources in."

 type =  string

 default =  "us-east-1"  # Set your default region here

}

variable  "instance_type" {

 description =  "The type of EC2 instance."

 type =  string

 default =  "t2.micro"  # Set your default instance type here

}

aws_region: Specifies the AWS region for resource deployment.
instance_type: Defines the type of EC2 instance to launch.

Define the AWS Provider and Key Pair

This file will configure Terraform to interact with AWS and upload your generated public SSH key to the AWS Key Pairs service.

Create a new file named provider.tf in your terraform-project directory.
Add the following content to provider.tf:


terraform {

   required_providers {

     aws = {

       source  = "hashicorp/aws"

       version = "~> 5.0"  # Specify a compatible version (e.g., 5.x)

     }

   }

 }

  provider "aws" {

   region = var.aws_region  # Use the region defined in variables.tf

 }

  resource "aws_key_pair" "ec2_key" {

   key_name   = "my-ec2-key"  # This is the name AWS will use for your key

   public_key = file("~/.ssh/my-ec2-key.pub")  # Path to your public key file

 }

terraform block: Declares the required AWS provider and its version.
provider "aws" block: Configures the AWS provider, using the aws_region variable for consistency.
resource "aws_key_pair": This resource uploads the content of your my-ec2-key.pub file to AWS, making it available for new EC2 instances. The key_name is how you'll refer to it in AWS.

Create the Security Group

A security group acts as a virtual firewall for your EC2 instance, controlling both inbound (ingress) and outbound (egress) network traffic.

1.Create a new file security_group.tf in your terraform-project directory.
2.Add the following configuration to security_group.tf:


resource  "aws_security_group"  "ec2_security_group" {

name= "terraform-ec2-sg"

description = "Allow SSH and HTTP access for Terraform EC2 instance"

ingress {

  description = "Allow SSH from my IP"

  from_port = 22

  to_port = 22

  protocol= "tcp"

  cidr_blocks = ["0.0.0.0/0"]

}

ingress {

  description = "Allow HTTP from anywhere"

  from_port = 80

  to_port = 80

  protocol= "tcp"

  cidr_blocks = ["0.0.0.0/0"]

}

egress {

  from_port = 0

  to_port = 0

  protocol= "-1"  # Allow all outbound protocols

  cidr_blocks = ["0.0.0.0/0"]

}

tags = {

  Name  =  "Terraform-EC2-SecurityGroup"

}

}

ingress blocks: Define inbound rules.
egress block: Allows all outbound traffic from the instance.
tags: Apply a tag for easy identification in the AWS console.

Define the EC2 Instance and AMI Data Source

This main.tf file will contain the core definition of your EC2 instance. It also includes a data source to dynamically fetch the latest Ubuntu AMI.

1.Create a new file main.tf in your terraform-project directory.

2.Add the following content to main.tf:


# Data source to get the latest Ubuntu AMI

data  "aws_ami"  "latest_ubuntu_ami" {

most_recent = true

filter {

  name = "name"

  values = ["ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*"]

}

filter {

  name = "virtualization-type"

  values = ["hvm"]

}

owners = ["099720109477"] # Canonical's AWS account ID

}

# Define the EC2 instance resource

resource  "aws_instance"  "my_first_ec2" {

ami = data.aws_ami.latest_ubuntu_ami.id  # Use the dynamically fetched AMI ID

instance_type = var.instance_type # Use the instance type defined in variables.tf

key_name= aws_key_pair.ec2_key.key_name # Reference the key pair created in provider.tf

vpc_security_group_ids = [aws_security_group.ec2_security_group.id] # Attach the security group

tags = {

  Name  =  "MyFirstTerraformEC2"

}

}

data "aws_ami": This data source automatically retrieves the ID of the most recent Ubuntu 22.04 LTS AMI.
resource "aws_instance": This block defines the EC2 instance itself, using the ami from the data source, instance_type from variables, and referencing the key_name and security_group created in their respective files.

Define the Outputs

Outputs are useful for displaying important information about your deployed resources after Terraform applies the configuration.

1.Create a new file outputs.tf in your terraform-project directory.

2.Add the following content to outputs.tf:


output "instance_public_ip" {

 description = "Public IP address of the EC2 instance."

 value       = aws_instance.my_first_ec2.public_ip

}

These outputs will display the public IP address of your EC2 instance once it's provisioned, making it easy to connect to your instance.

Initialising and Applying Your Configuration

After writing your Terraform configuration files, the next step is to initialise your working directory, validate your code, review the execution plan, and finally apply the changes to provision your AWS resources.

Initialise Your Terraform Project

The terraform init command prepares your project's working directory by downloading necessary provider plugins and setting up the backend for state management. Run this command first whenever you start a new Terraform project or clone an existing one.

1.Open your terminal and navigate to your project directory if you are not already there:


cd terraform-project

2.Run the initialisation command


terraform init

Validate Your Terraform Configuration

Validating your Terraform code before attempting to create resources is a good practice. The terraform validate command checks your configuration for syntax errors and internal consistency.

To validate your configuration, run:


terraform validate

If there are any errors, Terraform will provide detailed messages to help you fix them.

Review the Execution Plan

Before making any changes to your AWS environment, always generate and review an execution plan using terraform plan. This command shows you exactly what Terraform will create, modify, or destroy.


terraform plan

Terraform compares your desired state (defined in your .tf files) with the current state of your AWS resources and proposes a set of changes to achieve the desired state.

Apply Your Terraform Configuration

Once you are satisfied with the execution plan, use terraform apply to provision the resources in your AWS account.

Apply the configuration to provision resources, run:


terraform apply

This command executes the actions outlined in the terraform plan. Terraform will interact with the AWS API to create, update, or delete resources as necessary.

[

Verifying Your EC2 Instance

Confirm your EC2 instance is active in your AWS account following a successful terraform apply. This tutorial focuses on verification using the AWS Console, though the AWS CLI is also an option.

The AWS Management Console provides a visual way to confirm your instance's status.

1.Log in to your AWS Console.

2.In the search bar, type EC2 and select the EC2 service when it appears. This will take you to the EC2 Dashboard.

3.On the left-hand navigation pane, under Instances, click on Instances.

4.Locate your newly created EC2 instance. You should see an instance with the Name tag MyFirstTerraformEC2 (or whatever name you assigned in your main.tf).

5.Check the Instance state column. It should show Running.

6.Select your instance and review its details in the lower pane, including its Public IPv4 address, Instance ID, and associated Security Groups, to confirm everything matches your Terraform configuration.

Cleaning up your provisioned resources

After you've finished experimenting with your EC2 instance, it's crucial to clean up the resources you've provisioned. This prevents unnecessary AWS charges and keeps your account tidy. Terraform makes this process simple with a single command.

The terraform destroy command removes all resources defined in your current Terraform configuration.

1.Open your terminal.

2.Navigate to your project directory if you are not already there:

    cd terraform-project

3.Initiate the destruction process:

terraform destroy

Terraform will calculate all the resources that it previously created and are still managed by your configuration, and it will propose to destroy them. This includes your EC2 instance, security group, and the uploaded key pair.

And the instance has been terminated when verified on AWS Console.

Best practices for working with Terraform

You've successfully provisioned and destroyed your first EC2 instance with Terraform! As you continue your Infrastructure as Code journey, consider these best practices and areas for further exploration.

Version Control

Always store your Terraform configuration files (.tf files) in a version control system like Git. This allows you to track changes, collaborate with others, and easily revert to previous versions if needed.

Git Ignore Sensitive Files

Prevent sensitive information and Terraform's internal working files from being committed to your Git repository. Create a .gitignore file in your project root and add the following entries:

# .gitignore

.terraform/

*.tfstate

*.tfstate.backup

.terraform.lock.hcl

.terraform contains downloaded provider plugins and other internal Terraform files. You don't need to commit these.
*.tfstate is your Terraform state file. If you're not using remote state (which is recommended for teams), keep this out of version control as it can contain sensitive information and is crucial for Terraform's operation.
*.tfstate.backup backup state files don't need to be pushed as well.
.terraform.lock.hcl locks provider versions to ensure consistent builds, but it's typically fine to commit. The outline asks to ignore it if not using remote state, so I've included it.

Security

Security is paramount in cloud environments.

Never commit sensitive data directly: Avoid hardcoding AWS access keys or other secrets directly into your .tf files, especially if your repository is public or shared. Instead, use secure methods like AWS environment variables (as leveraged in this tutorial), AWS Secrets Manager, or a dedicated secret management tool like HashiCorp Vault.
Principle of Least Privilege: Always grant Terraform (or the IAM user/role it uses) only the minimum necessary permissions to perform its tasks. For this tutorial, we used AdministratorAccess for simplicity, but for production environments, define specific IAM policies that restrict actions to only the required resources.

Modularization

As your infrastructure grows, your Terraform configurations can become large and complex. Consider breaking down your configurations into reusable modules. Modules allow you to encapsulate and reuse common infrastructure patterns (e.g., a standard VPC setup, a common EC2 instance configuration) across different projects or environments.

Remote State

For team collaboration and production deployments, use remote state backends (e.g., Amazon S3, HashiCorp Cloud Platform, Terraform Cloud). Remote state safely stores your terraform.tfstate file in a shared, versioned, and often encrypted location. This enables multiple team members to work on the same infrastructure simultaneously without state conflicts and provides a reliable source of truth for your infrastructure's current state.

Conclusion

In this tutorial, you have successfully set up your AWS environment, written your first Terraform configuration, provisioned an EC2 instance, checked that it was deployed, and cleaned up the resources. You've experienced the core Terraform workflow: initialising your project, planning changes, applying them to create infrastructure, and finally, destroying it to avoid unintended costs. This exercise has equipped you with the building blocks for managing AWS resources using Infrastructure as Code (IaC).

The steps covered, from IAM user creation and SSH key management to defining resources like security groups and EC2 instances, form the bedrock of more complex deployments. As you continue to explore Terraform, remember the best practices discussed: leverage version control, secure your sensitive data, adhere to the principle of least privilege, and consider modularising your code and using remote state for collaborative or production environments

Automating User Creation and Management with Bash Scripting: A Detailed Guide.

Kalio Princewill — Fri, 05 Jul 2024 11:31:25 +0000

Introduction:

Linux, being a multi-user system, empowers administrators to create users and groups, each with specific permissions. When users are assigned to groups, they acquire the group's permissions. Effective user and group management is essential for system administration, from small personal servers to extensive enterprise networks. Streamlining the user creation process is crucial for ensuring security, organization, and efficiency, and Bash scripting provides a powerful means of automating these tasks.

This guide will walk you through creating a Bash script for managing user accounts on a Linux system.

The problem

Your company has employed many new developers. As a SysOps engineer, it's your responsibility to efficiently manage user accounts and their associated permissions. To streamline this process, you are tasked with writing a Bash script that automates the setup and configuration of new user accounts based on predefined criteria.

User and Group Definition: Users and their groups are defined in a text file that will be supplied to the script as an argument.
Home Directory Creation: A dedicated home directory should be created for each user
Secure Password Storage: User passwords should be stored securely in a file with path /car/secure/user_passwords.txt
Logging Actions: Logs of all actions should be logged to /var/log/user_management.log for auditing and troubleshooting purposes.
File Access Control: Only the owner, in this case root, should be able to access the user_password.txt file, ensuring that unauthorised access is prevented.
Error Handling: Errors should be gracefully handled

Prerequisites

To begin this tutorial, ensure you have the following:

A Linux system with administrative access.
Basic familiarity with Linux commands.

The Script:

The script, create_user.sh, automates user account creation, password encryption, group assignment, and logging activities on a Linux system. The goal is to streamline user management and ensure that all actions are securely recorded and auditable. Below is a detailed breakdown of each section of the script, including its purpose and functionality.

The shebang.

 #!/bin/bash

This specifies the type of interpreter script will be run with. Since it is a "bash" script, it should be run with the Bourne Again Shell (Bash) interpreter. Also, some commands in the script may not be interpreted correctly outside of Bash.

Defining File Paths and Variables
Next, create some environment variables that will hold the text_file.txt, the logs file path (var/log/user_management.log), and password file path (/var/secure/user_passwords.csv).

LOG_FILE="/var/log/user_management.log"
PASSWORD_FILE_DIRECTORY="/var/secure"
PASSWORD_FILE="/var/secure/user_passwords.txt"
PASSWORD_ENCRYPTION_KEY="secure-all-things"
USERS_FILE=$1

LOG_FILE: specifies where the log entries will be stored. Logging is crucial for tracking actions and diagnosing issues.
PASSWORD_FILE_DIRECTORY: defines the directory for storing user passwords securely.
PASSWORD_FILE: is the file where encrypted passwords will be saved.
PASSWORD_ENCRYPTION_KEY: is used to encrypt user passwords.
USERS_FILE: takes the first script argument, which should be the path to a file containing user data.

Next, create this file path and give them the right permission using the command below.

touch $LOG_FILE
mkdir -p /var/secure
chmod 700 /var/secure
touch $PASSWORD_FILE
chmod 600 $PASSWORD_FILE

Details of the command in the codeblock are:
touch $LOG_FILE

touch: This command is used to create an empty file or update the access and modification times of an existing file.
$LOG_FILE: This variable holds the path("/var/log/user_management.log") to the log file.

mkdir -p /var/secure

mkdir: This command is used to create directories.
-p: This option allows the creation of parent directories as needed. If the directory already exists, it will not return an error.
/var/secure: This is the directory path to be created.

chmod 700 /var/secure

chmod: This command is used to change the permissions of files or directories.
700: This permission setting means:
7: The owner has read, write, and execute permissions.
0: The group has no permissions.
0: Others have no permissions.
/var/secure: The directory whose permissions are being modified.

chmod 600 $PASSWORD_FILE

chmod: This command changes the permissions of a file.
600: This permission setting means:
6: The owner has read and write permissions.
0: The group has no permissions.
0: Others have no permissions.

Checking for Sudo Privileges

if [ "$(id -u)" != "0" ]; then  
    echo "This script must be run with sudo. Exiting..."   
    exit 1
 fi

This code ensures the script runs with root privileges by checking if the Effective User ID (EUID) is zero. In Linux, an EUID of 0 corresponds to the root user. If the script is not executed with administrative rights, it exits with an error message. This is because the script will perform actions that require elevated privileges, such as creating users and groups, setting passwords, creating files, etc.

Validating Arguments
Next to ensure that the script is properly executed with an input file provided as an argument, this conditional check terminates the script if no argument is detected. Using $#to represent the number of arguments passed when executing the script, it validates whether no argument ($# equals zero) or more than one argument ($# is greater than or equal to 2) is given. If either condition is met, an error message is displayed, halting the script's execution.

if [ $# -eq 0 ]; then   
   echo "No file path provided."   
   echo "Usage: $0 <user-data-file-path>"   
   exit 1 
fi

Also to verify if the provided file exists. If the file does not exist, the script exits with an error message.

if [ ! -e "$USERS_FILE" ]; then
  echo "The provided user's data file does not exist: $USERS_FILE"
  exit 1
fi

Function Definitions
Next to ensure reusability and modularity in your code, copy and paste the several utility functions to your code
Check for Installed Package:

is_package_installed() {
  dpkg -s "$1" >/dev/null 2>&1
}

This function checks if a package is installed on your machine using dpkg. It suppresses output and returns a status code.
Encrypt Password:

encrypt_password() {
  echo "$1" | openssl enc -aes-256-cbc -pbkdf2 -base64 -pass pass:"$2"
}

The commnad uses openssl to encrypt a password. The password is encrypted using AES-256-CBC and a provided encryption key.
Set Bash as Default Shell:

set_bash_default_shell() {
  local user="$1"
  sudo chsh -s /bin/bash "$user"
}

This function sets Bash as the default shell for the specified user.
Installing Required Packages:
To ensure you have the necessary packages to run the script, add the code-block

if ! is_package_installed openssl; then
  echo "openssl is not installed. Installing..."
  sudo apt-get update
  sudo apt-get install -y openssl
fi

if ! is_package_installed pwgen; then
  echo "pwgen is not installed. Installing..."
  sudo apt-get update
  sudo apt-get install -y pwgen
fi

with the command above you have defined variables for the file paths corresponding to the log file, password file, and input file. Additionally, you have ensured that the script is being executed with superuser privileges.

The subsequent step involves iterating through each line in the input text file, separating these lines by usernames and groups, and then using the Bash script to create the users and assign them to their respective groups.
To set up these users and groups, incorporate the following commands into your create_users.sh script:

for line in "${lines[@]}"; do
    # Remove leading and trailing whitespaces
    line=$(echo "$line" | xargs)

    # Split line by ';' and store the second part
    IFS=';' read -r user groups <<< "$line"

    # Remove leading and trailing whitespaces from the second part
    groups=$(echo "$groups" | xargs)

    # Create a variable groupsArray that is an array from spliting the groups of each user
    IFS=',' read -ra groupsArray <<< "$groups"

    # Check if user exists
    if id "$user" &>/dev/null; then
        echo "User $user already exists. Skipping creation."
        continue
    fi

    # Generate a 6-character password using pwgen
    password=$(pwgen -sBv1 6 1)

    # Encrypt the password before storing it
    encrypted_password=$(encrypt_password "$password" "$PASSWORD_ENCRYPTION_KEY")

    # Store the encrypted password in the file
    echo "$user:$encrypted_password" >> "$PASSWORD_FILE"

    # Create the user with the generated password
    sudo useradd -m -p $(openssl passwd -6 "$password") "$user"

    # Set Bash as the default shell
    set_bash_default_shell "$user"

    # loop over each group in the groups array
    for group in "${groupsArray[@]}"; do
        group=$(echo "$group" | xargs)

        # Check if group exists, if not, create it
        if ! grep -q "^$group:" /etc/group; then
            sudo groupadd "$group"
            echo "Created group $group"
        fi

        # Add user to the group
        sudo usermod -aG "$group" "$user"
        echo "Added $user to $group"
    done

    echo "User $user created and password stored securely"
done

The command above does the following:
Iterating Through Lines in the File

for line in "${lines[@]}"; do

This command initiates a loop that iterates over each line in the lines array. Each element of the array lines, representing a line from the input file, is processed one by one and assigned to the line variable.

Removing Leading and Trailing Whitespace

line=$(echo "$line" | xargs)

Here, echo "$line" | xargs removes any leading and trailing whitespace from the current line. This ensures that any extra spaces at the beginning or end of the line are trimmed off.

Splitting the Line into User and Groups

IFS=';' read -r user groups <<< "$line"

This line splits the linevariable into two parts based on the ; delimiter. The first part is assigned to user, and the second part is assigned togroups. IFS=';'sets the Internal Field Separator to ;, which defines how the line is split.

Trimming Whitespace from Groups

groups=$(echo "$groups" | xargs)

This command removes any leading and trailing whitespace from the groupsvariable. This is necessary to ensure that the group names are clean and free from extra spaces.

Splitting Groups into an Array

IFS=',' read -ra groupsArray <<< "$groups"

Here, IFS=',' sets the Internal Field Separator to ,. The read -ra groupsArray splits the groups string into an array groupsArray based on commas, with each element representing a group.

Checking if User Already Exists

if id "$user" &>/dev/null; then
    echo "User $user already exists. Skipping creation."
    continue
fi

This block checks if a user with the given user name already exists. The id "$user" &>/dev/null command attempts to fetch user details, and if successful (i.e., the user exists), it outputs nothing to /dev/null. If the user exists, a message is printed, and the continue command skips the rest of the loop for this user and moves to the next iteration.

Generating a 6-Character Password

password=$(pwgen -sBv1 6 1)

This command generates a secure, random 6-character password using the pwgen tool. The -s flag ensures a secure password, -B avoids ambiguous characters, and -v1 specifies the length and quantity (1 password).
Encrypting the Password

encrypted_password=$(encrypt_password "$password" "$PASSWORD_ENCRYPTION_KEY")

This line calls the encrypt_password function to encrypt the generated password. The encrypted password is stored in the encrypted_passwordvariable. The encryption key used is stored in the PASSWORD_ENCRYPTION_KEY variable.

Storing the Encrypted Password

echo "$user:$encrypted_password" >> "$PASSWORD_FILE"

This command appends the username and encrypted password to the PASSWORD_FILE. The >> operator ensures that the data is added to the end of the file without overwriting existing content.
Creating the User with the Generated Password

sudo useradd -m -p $(openssl passwd -6 "$password") "$user"

This line creates a new user with the username stored in user. The -m option creates the user's home directory. The password is hashed using openssl passwd -6 with the generated password, ensuring it is securely stored.

Setting Bash as the Default Shell

set_bash_default_shell "$user"

This function call sets Bash as the default shell for the new user. The set_bash_default_shell function is assumed to be defined elsewhere in the script to change the user's shell to Bash.

Adding User to Specified Groups

for group in "${groupsArray[@]}"; do
    group=$(echo "$group" | xargs)

This loop iterates through each group in the groupsArray array. Each group is cleaned of leading and trailing whitespace.
Checking and Creating Group if Necessary

if ! grep -q "^$group:" /etc/group; then
    sudo groupadd "$group"
    echo "Created group $group"
fi

For each group, this block checks if the group exists in the /etc/group file. If not, the groupadd command creates the group, and a confirmation message is printed.

Adding User to the Group

sudo usermod -aG "$group" "$user"
echo "Added $user to $group"

This command adds the user to the specified group using usermod -aG. The -aG option appends the user to the supplementary group without affecting membership in other groups. A message is printed confirming the addition.

With this, you’ve successfully developed a script that efficiently handles user and group management on your system. You can find the complete script on the Github Repo

Verifying Script Functionality

To verify that your script is functioning correctly, execute it in a terminal that supports Linux commands. Use the following command in your terminal to run the script:

Firstly, make your script executable, run the command below,

chmod +x create_users.sh

then, run the script,

./create_user.sh ./text_file.txt

In Conclusion

This article details the procedure for automating the creation of users and groups through a script. It incorporates several assumptions and compromises to balance security with usability. As an administrator, you can now leverage this script to streamline user onboarding within your organisation.

Deploying a Static Website on AWS EC2 with Nginx: A Comprehensive Guide

Kalio Princewill — Tue, 02 Jul 2024 08:16:43 +0000

Introduction:

AWS (Amazon Web Services) is a comprehensive cloud computing platform provided by Amazon, offering a wide range of services including computing power, storage, and databases. It enables businesses and developers to access and use scalable and cost-effective cloud resources on-demand.

What is AWS EC2?
Amazon Elastic Compute Cloud (EC2) is a core service within AWS that provides on-demand virtual machines (instances). You can select from a diverse range of instance types to match your specific computational needs. AWS EC2 enables elastic scaling, allowing you to adjust compute power up or down based on requirements.

This guide will concentrate on AWS EC2, demonstrating how to use its virtual machine features to set up a server environment tailored for hosting your static website with Nginx web server.

Prerequisites

To begin this tutorial, the following are good to have:

To complete this project you should have a fully verified AWS account, to get on visit here.
Also a good to have would be basic knowledge of linux commands.
Basic understanding of HTML and CSS

The complete the project above i will walk you through as step by step procedure:

Creating an AWS EC2 instance:

Firstly, sign into your AWS console and click on the EC2 under the Compute section:

Next, to launch an EC2 instance click on the Launch Instance button

Name and tags
Next, give your EC2 Instance a desired name,

Application and OS images
For this project select Ubuntu as the application image for the instance.

Amazon Machine Image (AMI)
Select the Ubuntu Server and free tier, you don’t want to spend money when it can be free.

Key-Value Pair: Key-value pairs are used in instance metadata to securely configure access, retrieve instance-specific information, and automate initial setup tasks.
Next, create a new key pair and choose the .pem format.

Give the Key pair a name and ensure the fields are selected, this should download a file on your browser.

Network Settings:
Network settings configure firewall rules to control the traffic to your instance. In this section, enable options for Allow SSH traffic from Anywhere, Allow HTTPS traffic from the Internet, and Allow HTTP traffic from the Internet.
Click on the launch Instance button to continue.

After a successful launch, you should see the screen below

click on the instances button on the left sidebar to view your running instance, select the checkbox of your instance to see more details on the instance as it’s displayed below.

To use your instance, click on the Connect button, this should open the screen below,

Click the orange connect button to launch the terminal of your EC2 instance Virtual machine.

Once connected you will see your EC2 ubuntu terminal.

Installing Nginx

What’s Nginx?
Nginx is a free, open-source web server known for its high performance and efficient resource usage. It excels in reverse proxy, load balancing, and caching, while also providing HTTPS server capabilities. Designed for maximum performance and stability, Nginx can also serve as a proxy for email protocols like IMAP, POP3, and SMTP, making it a versatile tool for building robust and scalable web applications.

Firstly to operate as a root user, run the command on your terminal

sudo -i

update the package list on your instance by running

apt-get update

When the package is fully updated, install Nginx by running the command

apt-get install nginx

Once that is installed, to check the status of nginx web server with this command

service nginx status

To access nginx template html run

cd /var/www/html

Then run view the nginx template

curl localhost

Alternatively, to view the default nginx template, copy the PublicIP address at the bottom of the page and open on a new browser

Customising your Nginx server
At the same file directory to view the current files in the directory, run

ls

Now to change the template of your nginx server, run

nano index.nginx-debian.html

using your arrow key clean up the content

Copy and paste the Html and css file below

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Kalio Princewill</title>
    <style>
    body, html {
    margin: 0;
    padding: 0;
    height: 100%;
    display: flex;
    justify-content: center;
    align-items: center;
    background: linear-gradient(135deg, #6B73FF 0%, #000DFF 100%);
    font-family: 'Montserrat', sans-serif;
}

.container {
    display: flex;
    justify-content: center;
    align-items: center;
    height: 100%;
    width: 100%;
}

.glass {
    background: rgba(255, 255, 255, 0.1);
    border-radius: 10px;
    box-shadow: 0 4px 30px rgba(0, 0, 0, 0.1);
    backdrop-filter: blur(10px);
    -webkit-backdrop-filter: blur(10px);
    padding: 20px 40px;
    border: 1px solid rgba(255, 255, 255, 0.3);
    text-align: center;
}

h1, a {
    color: #fff;
    margin: 0;
    font-size: 1em;
}
    </style>
</head>
<body>
    <div class="container">
        <div class="glass">
            <h1>Hello world, l just deployed a my status site on AWS EC2</h1>
        </div>
    </div>
</body>
</html>

After you have adding your desire html and css, to save the file use Control + O , then Enter finally use Control + X to exit the the editor.
Refresh your public IP address to see the final result

Your static website is now online and accessible!

In summary

This guide provides a step-by-step tutorial on how to deploy a static website using an Nginx web server on an AWS EC2 instance. It begins by introducing AWS EC2, a service for on-demand virtual machines, and outlines the prerequisites such as an AWS account, basic Linux knowledge, and understanding of HTML and CSS. The guide details the process of creating and configuring an EC2 instance, including selecting an Ubuntu image, setting up network security rules, and establishing a key pair for secure access. It then covers the installation and configuration of the Nginx web server on the EC2 instance. The final steps involve customizing the Nginx template to host the static website, resulting in a live and accessible web page.

Building a Cryptocurrency News Aggregator with Xata and Cloudinary

Kalio Princewill — Wed, 23 Nov 2022 16:04:39 +0000

Cryptocurrencies are digital assets that are traded on exchanges or pair-to-pair, they vary in value and are susceptible to market news. People who trade these digital assets have to pay attention to news to be informed of market movements because positive news tends to drive up prices while negative news tends to drive down prices.

Navigating through numerous news sources is challenging for traders as some of these sources publish simultaneously or at various times of the day. This challenge would be well-solved by a news aggregator, which would bridge the information gap by gathering market news and trends on one website that traders could access.

In this tutorial, I will be discussing how to build a cryptocurrency news aggregator application with React while implementing tools like Cloudinary and Xata.

At the end of this tutorial, your website should look like the below image.

Tools

I will be using the tools below for this project.

React.js: React is a front-end JavaScript toolkit that is free and open source for creating user interfaces using UI components.
Cloudinary: Cloudinary is a platform users deliver, manage, and deliver photos and videos for websites and apps.
Xata: Xata is a serverless data platform with a spreadsheet-like user interface and an infinitely scalable data API. Xata also functions as a free-text search engine.

Prerequisites

Readers should have the following before building:

A text editor
Node installed on your local computer
Basic knowledge of Reactjs
An understanding of how APIs work
Cloudinary account

Getting Started

In building our cryptocurrency news aggregator, we will create a react app, add features to it, proceed to integrating Cloudinary to manage media upload and Xata for database management.
If you would like to get started with the code right away, you can visit the GitHub repository here.

Create react app

On your computer, create a project folder and open it up with your text editor.

In the project directory, open up a terminal and run the below commands


npx create-react-app news-aggregator

cd news-aggregator

npm start

The npm start command is used to start the react application on a development server in your web browser. Open localhost:3000 in your browser and you should see the below image

Now your react application is ready, it’s time to setup Cloudinary.

Cloudinary Setup

As you already know, Cloudinary is used for media management, and we will be integrating it into the application to manage crypto images on the watch list.

Before configuring Cloudinary in your React app, install the Cloudinary React.js SDK in your project using the command below

npm i @cloudinary/url-gen @cloudinary/react

In the root of your project, create a .env file on the project folder and include your Cloudinary Cloud name. You should find the cloudName by going to your Cloudinary dashboard.

Register the cloudname as follows in your .env file.

cloudName: 'demo’
Using Cloudinary component for global use in your App.js file by adding the following:

import React from 'react'
    import {AdvancedImage} from '@cloudinary/react';
    import {Cloudinary} from "@cloudinary/url-gen";

    const App = () => {
      // Create a Cloudinary instance and set your cloud name.
      const cloud = new Cloudinary({
        cloud: {
          cloudName: 'demo'
        }
      })

NB: Replace demo with your cloudname from cloudinary

Creating Components

By now you should have an src folder in your project directory which was created from react.
Inside the folder, create a component folder which will contain the following files

latest.js
navigation.js
news.js
options.js
watchlist.js

These component files are structures for the application.

Latest.js

Create a latest.js file and paste the below into it

import React from "react";

export default function TheLastest(){
    return (
        <h1 className="latest">
            Latest News...
        </h1>
      )
}

Navigation.js

Create a navigation.js file and paste the below into it

import React from "react";

export default function Nav(){
    return (
        <div>
        <nav>

        <img src="https://cdn4.iconfinder.com/data/icons/summer-line-5/48/sea_lighthouse_ocean_water_beach-256.png"  className="logoimg"/>

        <div>
            <h2 className='logoname'>
            LightCrypto
        </h2>
        <p className="newstor">News Aggregator</p>

        </div>

        <h2 className='search'>
            <input type= "search" placeholder="search" className = "searchtab" />
        </h2>
        </nav>
    </div>
    )
}

News.js

Create a news.js file and implement the below configuration as well

import React from "react";

export default function News(props){
    return (

        <div className="newstab">
            <div>
            <p className="Arrival">
                {props.timestamp}
            </p>
            </div>
        <div className="the-news">

        <h3 className="the-heading">
            <a href="{link}" alt="" className="sourceList">
        {props.heading}
        </a>
        </h3>
        <h4 className="the-preview">
        source: {props.source}
        </h4>
        </div>
        </div>
    )
}

Options.js

Create a options.js file and paste the below into it

import React from 'react';
export default function Options(){
    return(
        <div className='optionsArea'>
            <div className='optionButton'>
        <h3 className='home'>
            Home
        </h3>
        <h3 className='calender'>
            <a href= "https://coinmarketcal.com/en/" className='calender' >Calender</a>
        </h3>
        <h3 className='aboutUs'>
            About Us
        </h3>
        <h3>
            <a href= "https://medium.com/@kalio_" className='blog'>Blog</a>
        </h3>
        </div>
        <div className='socails'>
            <h3 className='socailhead'>Socials</h3>                   
                <div className='twitter'><img src="https://cdn4.iconfinder.com/data/icons/various-icons-2/476/YouTube.png" alt='youtube' className='twitter-img' /><a href='https://www.youtube.com/channel/UCGu6O9-vlYHWZT6SFIpp4Kw' className='twitterr'>Youtube</a></div>
                <div className='twitter'><img src="https://cdn3.iconfinder.com/data/icons/social-network-flat-3/100/Discord-256.png" alt='discord' className='twitter-img' /><a href='https://twitter.com/Kalio_Prince' className='twitterr'>Discord</a></div>
        </div>
        </div>
    )
}

Watchlist.js

Create a watchlist.js file and add the following code snippet

https://gist.github.com/kalio007/38a25e73127d1cd62b0d8258c432c31a

Configuring Mock API

Fully fledged news aggregator applications would have API integration to fetch news from numerous news platforms or sources. Since this is not an advanced or complex application at the moment, we would be implementing a mock API which does the job as well. This is how an actual API integration works but the only difference here is we are using a mock version.
The mock API here would be used to fetch news from various sources on its arrival and serve the news to users.

The API will be arranged in the following manner:

id
arrival
link
heading
source

Return to the src folder and create an Api.js file src/Api.js and paste the below snippets into it.
https://gist.github.com/kalio007/96d1db56335143ce7af7c04593e53286

Feel free to play around with the API by adding your own information

Open your App.js file and replace the existing configuration with the codes snippet below. We are importing the components we developed previously into App.js and setting up the API to get it working.

https://gist.github.com/kalio007/da8bb215d19dcb4d7cb611616e7fb696

Formatting the CSS styling

I will be using basic CSS to style the application. paste the code snippet below.
https://gist.github.com/kalio007/9cc3453c3e81cb457c64563742989b1e

After completing the above steps, restart the application with the npm start command and your application be ready.

Xata Setup

For this application to manage the database of your application, you need to create an account with Xata by clicking here. It is completely free to create an account and should take about a minute.

On your Workspace, create a database by clicking add a database which we would be querying from our application.
Run the below on your local terminal

npm install typescript @types/node @types/react -D

Next on your terminal install the Xata CLI globally

npm i -g @xata.io/cli

To login into your Xata, run he Auth login on your terminal

xata auth login

Create an API key for Xata by clicking here. To initialize Xata run the following command

xata init

Copy and paste the codes below to your main src file to query the database

import { FC } from 'react';
    import { XataClient } from '../util/xata';

    type Props = Awaited<ReturnType<typeof getServerSideProps>>['props'];

At this point your application is complete, and the necessary tools have been implemented as I earlier discussed.

Conclusion

This tutorial explains how to build a cryptocurrency news aggregator web application, which I implement Cloudinary to manage media assets and Xata for database management.

Resources

You can visit Cloudinary’s official documentation to understand how to get started.
To get started with Xata as well, a good place to visit is their official documentation
Xata has uploaded YouTube videos that walk you through the implementation of the tool in your projects and explain how Xata simplifies development.