DEV Community: Emilien Lancelot

Build Your MCP Client: Because Copy-Pasting JSON Isn’t Engineering

Emilien Lancelot — Thu, 12 Jun 2025 23:21:15 +0000

MCP is a lot more complex than AI influencers would have you believe. Sure, if by “understanding MCP” you mean putting together a JSON file that points to the server, then I guess you’re all set. And if you’re working on nuclear power plants, do let me know where, I’d rather not spend my holidays nearby.

What you’ll get from this article

I’ll explain the architecture and protocols used by MCP clients.
Then I’ll give you the complete code of an HTTP MCP client and server. The client can connect to any HTTP MCP server and list tools. It won’t use an LLM as YOU will be the LLM.^^

I. The two types of MCP clients

There are 2 types of MCP servers… Ideally, your client should be able to deal with both types.

1. STDIO clients

I’ll try to keep this short: DO NOT DO THAT

For some unknown, stupid reason, Anthropic pushed for a strange server communication (transport) protocol: STDIO

This doesn’t rely on any HTTP socket bound to your localhost to interact, but instead uses the f*cking filesystem…

We have built client/server software using localhost for 40 years. But hey… Let’s use the FS instead this time. This way our servers can’t be exposed on the internet, and no clients can connect except the ones on the same machines! YEAY

Very future-proof!

For a client to use this stupid implementation, you first have to set the filepath of your server in the global MCP config.

So your mcp.json config should look like this:

{
"servers": {
    "todoist": {
        "url": "https://mcp.pipedream.net/aabc9cef-a206-545f-8094-1b9fgdf09203/todoist"
    },
    "todo_local": {
      "command": "uv",
      "args": [
        "run",
        "--with",
        "mcp[cli]",
        "mcp",
        "run",
        "/path/to/your/server.py"
      ]
    }
  }
}

This config tells the MCP client to look at the file /path/to/your/server.py which contains the source code of your server. By reading its content, it will discover the different tools that can be “served”.

When the client starts, its first task is to boot the server. YES. You read that right. The client starts the server… It makes total sense! (No, it doesn’t)

When the client exits, it must stop the server process that was forked during boot.

To summarize: DON’T WRITE STDIO SERVERS

As for the client, you can use a software bridge to port the stupid STDIO transport to HTTP (better). For example, mcp-proxy does exactly that. But you can find many more on Google. This way, you only have to care about one transport mechanism and not two.

2. HTTP clients

Obviously, HTTP clients deal with HTTP servers. It’s way better than STDIO as you will be able to expose servers on the internet, allowing remote clients to connect from anywhere.

This is what the future will look like. We’ll all have our own LLM connecting to remote MCP servers and take actions on our behalf. Like ordering food, booking flights, etc.

As if things weren’t complicated enough, I also have to specify the exact HTTP protocol used by these servers: SSE

Never heard of SSE?

It’s Server-Sent Events, meaning that the server can push messages to the clients without the client having to request anything. It’s like an HTTP socket but unidirectional (server => client only).

What’s dumb, IMHO, is that your client makes a POST request to get the list of available tools, and the server responds using SSE. So it’s kind of weird to use a protocol designed to push data to the client proactively… only when it’s explicitly asked for data.

I may see a point in the future where MCP servers will send some data to the client out of the blue, and the client could react to it. But as of today, it feels very strange to ask for data and retrieve it using SSE.

How it works

Also, the SSE protocol specifies that each message should end with a double \n and start with data:.

So, for example:

data: message1\n
\n
data: message2\n
\n

So, what happens if the server sends some unknown unstructured data containing any number of consecutive line endings?

It will go BOOM is what will happen.

You could send the message as a base64 string to fix this. But this would imply that the client can recognise base64 and decode it. Something that isn’t going to happen anytime soon unless it is specified in Anthropic’s MCP spec. So we’ll see… Just keep in mind that the server can’t send random stuff, as it might break the client's ability to read messages.

II. Creating a demo server

To create our client, we first need a basic server implementing the MCP spec. This way we’ll have the ability to learn what are the correct JSON responses.

You can follow this good tutorial. It’s using NodeJS, but it’s very easy to understand and use.

I’ll add everything here for posterity:

# Install the MCP framework globaly (there's a CLI)
npm i -g mcp-framework

# Create the boilerplate code of an MCP weather server
mcp create weather-http-server --http --port 1337 --cors

cd weather-http-server

# Add a tool to the server
mcp add tool weather

Update src/tools/WeatherTool.ts with this content:

import { MCPTool } from "mcp-framework";
import { z } from "zod";

interface WeatherInput {
  city: string;
}

class WeatherTool extends MCPTool<WeatherInput> {
  name = "weather";
  description = "Get weather information for a city";

  schema = {
    city: {
      type: z.string(),
      description: "City name to get weather for",
    },
  };

  async execute({ city }: WeatherInput) {
    // In a real scenario, this would call a weather API
    // For now, we return this sample data
    return {
      city,
      temperature: 22,
      condition: "Sunny",
      humidity: 45,
    };
  }
}

export default WeatherTool;

# Build the project
npm run build

# Start the server
npm start

# Your HTTP MCP server is now running at http://localhost:1337/mcp

You can try to connect with existing clients like Claude or Copilot. Simply set the above server URL in the MCP global config file, and you should be good to go.

III. Building an MCP client

Now that I’ve stopped ranting about what’s dumb in MCP and we have a working server, let’s write the client.

But because I want to iterate rapidly, I will not include any LLM in this piece of code. This means that doing the tool calling and parsing will be done manually. DON’T PANIC. The longest methods are not the ones that interest us the most. It’s only parsing.

#!/usr/bin/env python3
"""
Simple MCP (Model Context Protocol) HTTP Client
Connects to remote MCP servers via Streamable HTTP transport and provides a basic chat interface.
"""

import json
import requests
import uuid
import time
from typing import Dict, List, Any, Optional
from dataclasses import dataclass
from urllib.parse import urljoin


@dataclass
class Tool:
    name: str
    description: str
    input_schema: Dict[str, Any]


class MCPClient:

    def __init__(self, server_url: str):
        self.server_url = server_url.rstrip('/')
        self.session = requests.Session()
        self.tools: Dict[str, Tool] = {}
        self.initialized = False
        self.session_id: Optional[str] = None

    def initialize(self) -> bool:
        """Initialize connection with the MCP server"""
        try:
            # Initialize the protocol
            init_result = self._make_request("initialize", {
                "protocolVersion": "2025-03-26",
                "capabilities": {
                    "tools": {}
                },
                "clientInfo": {
                    "name": "yacana-mcp-client",
                    "version": "0.1.0"
                }
            })

            server_info = init_result.get('serverInfo', {})
            print(f"Connected to MCP server: {server_info.get('name', 'Unknown')} v{server_info.get('version', 'Unknown')}")

            # List available tools
            tools_result = self._make_request("tools/list")
            tools = tools_result.get("tools", [])

            for tool_info in tools:
                tool = Tool(
                    name=tool_info["name"],
                    description=tool_info["description"],
                    input_schema=tool_info["inputSchema"]
                )
                self.tools[tool.name] = tool
                print(f"Available tool: {tool.name} - {tool.description}")

            if not tools:
                print("No tools available on this server")

            self.initialized = True
            return True

        except Exception as e:
            print(f"Failed to initialize MCP client: {e}")
            return False

    def _make_request(self, method: str, params: Dict[str, Any] = None) -> Dict[str, Any]:
        """Make a JSON-RPC request to the MCP server using Streamable HTTP transport"""
        request_id = str(uuid.uuid4())
        payload = {
            "jsonrpc": "2.0",
            "id": request_id,
            "method": method
        }
        if params:
            payload["params"] = params

        headers = {
            "Content-Type": "application/json",
            "Accept": "application/json, text/event-stream"
        }

        # Add session ID header if we have one
        if self.session_id:
            headers["Mcp-Session-Id"] = self.session_id

        try:
            response = self.session.post(
                self.server_url,
                json=payload,
                headers=headers,
                timeout=30
            )
            response.raise_for_status()

            # Check if server returned a session ID during initialization
            if method == "initialize" and "Mcp-Session-Id" in response.headers:
                self.session_id = response.headers["Mcp-Session-Id"]
                print(f"Server assigned session ID: {self.session_id}")

            # Handle different response types
            content_type = response.headers.get('content-type', '').lower()

            if 'application/json' in content_type:
                result = response.json()
                if "error" in result:
                    raise Exception(f"MCP Error: {result['error']}")
                return result.get("result", {})

            elif 'text/event-stream' in content_type:
                # Handle SSE response - for simplicity, we'll read the first JSON response
                # In a production client, you'd want to properly handle the SSE stream
                return self._handle_sse_response(response)

            else:
                # If no specific content type, try to parse as JSON
                try:
                    result = response.json()
                    if "error" in result:
                        raise Exception(f"MCP Error: {result['error']}")
                    return result.get("result", {})
                except:
                    raise Exception(f"Unexpected response format: {response.text}")

        except requests.exceptions.HTTPError as e:
            if e.response.status_code in [404, 405]:
                print("This server is too old and doesn't fully support SSE but probably a mix of HTTP+SEE. This is not supported by this client...")
            raise Exception(f"HTTP Error {e.response.status_code}: {e.response.text}")

    def _handle_sse_response(self, response) -> Dict[str, Any]:
        """Handle Server-Sent Events response (simplified implementation)"""
        # This is a simplified SSE parser - in production you'd want a proper SSE client
        lines = response.text.split('\n')
        for line in lines:
            line = line.strip()
            if line.startswith('data: '):
                try:
                    data = json.loads(line[6:])  # Remove 'data: ' prefix
                    if "result" in data:
                        return data["result"]
                    elif "error" in data:
                        raise Exception(f"MCP Error: {data['error']}")
                except json.JSONDecodeError:
                    continue

        raise Exception("No valid JSON-RPC response found in SSE stream")


    def call_tool(self, tool_name: str, arguments: Dict[str, Any]) -> Dict[str, Any]:
        """Call a tool on the MCP server"""
        if not self.initialized:
            raise Exception("Client not initialized")

        if tool_name not in self.tools:
            raise Exception(f"Tool '{tool_name}' not available. Available tools: {list(self.tools.keys())}")

        result = self._make_request("tools/call", {
            "name": tool_name,
            "arguments": arguments
        })

        return result

    def get_available_tools(self) -> List[str]:
        """Get list of available tool names"""
        return list(self.tools.keys())

    def get_tool_info(self, tool_name: str) -> Optional[Tool]:
        """Get information about a specific tool"""
        return self.tools.get(tool_name)

    def disconnect(self):
        """Explicitly disconnect from the server"""
        if self.session_id:
            try:
                headers = {"Mcp-Session-Id": self.session_id}
                self.session.delete(self.server_url, headers=headers, timeout=5)
            except:
                pass  # Ignore errors during cleanup
        self.session.close()


def detect_tool_calls(message: str) -> List[Dict[str, Any]]:
    """
    Detect tool calls in the format: @tool_name({"param1": "value1", ...})
    Returns a list of dicts: {"name": ..., "arguments": ...}
    """
    import re
    import json

    tool_calls = []
    pattern = r'@([\w-]+)\((\{.*?\})\)'

    matches = re.finditer(pattern, message)
    for match in matches:
        tool_name = match.group(1)
        args_str = match.group(2)
        try:
            arguments = json.loads(args_str)
            if not isinstance(arguments, dict):
                raise ValueError("Arguments must be a JSON object")
        except Exception as e:
            print(f"Failed to parse arguments for @{tool_name}: {e}")
            raise Exception("Invalid param. Retry.")
        tool_calls.append({
            "name": tool_name,
            "arguments": arguments
        })

    return tool_calls


def main():
    """Main chat loop"""
    print("=== Yacana MCP Client ===")

    # Get server URL from user
    server_url = input("Enter MCP server URL (e.g., https://mcp.deepwiki.com/mcp): ").strip()
    if not server_url:
        server_url = "https://mcp.deepwiki.com/mcp"

    # Initialize client
    client = MCPClient(server_url)

    try:
        if not client.initialize():
            print("Failed to connect to MCP server. Exiting.")
            return

        print("\n=== Chat Loop Started ===")
        print("Type your messages below. Use @tool_name(arg1=\"value1\") to call tools.")
        print("Type 'quit' to exit, 'tools' to list available tools.\n")

        while True:
            try:
                user_input = input("\n> ").strip()

                if user_input.lower() in ['quit', 'exit']:
                    break

                if user_input.lower() == 'tools':
                    print("\nAvailable tools:")
                    if not client.get_available_tools():
                        print("  No tools available")
                    else:
                        for tool_name in client.get_available_tools():
                            tool = client.get_tool_info(tool_name)
                            print(f"  - {tool_name}: {tool.description}")
                            # Show required parameters
                            schema = tool.input_schema
                            if 'properties' in schema:
                                required = schema.get('required', [])
                                props = schema['properties']
                                params_info = []
                            for k, v in props.items():
                                param_type = v.get('type', 'any')
                                required_marker = "*" if k in required else ""
                                params_info.append(f"{k}({param_type}){required_marker}")
                            print(f"    Parameters: {', '.join(params_info)}")
                    continue

                if not user_input:
                    continue

                # Detect tool calls in the message
                try:
                    tool_calls = detect_tool_calls(user_input)
                except Exception as e:
                    continue

                if tool_calls:
                    print("\nDetected tool calls:")
                    for call in tool_calls:
                        tool_name = call["name"]
                        arguments = call["arguments"]

                        print(f"  Calling {tool_name} with {arguments}")

                        try:
                            result = client.call_tool(tool_name, arguments)
                            print(f"  Result: {json.dumps(result, indent=2)}")
                        except Exception as e:
                            print(f"  Error calling {tool_name}: {e}")
                else:
                    # In a real implementation, you would send this to your LLM
                    # and let the LLM decide whether to call tools
                    print("Assistant: I received your message. In a full implementation, this would be processed by an LLM that could decide to call tools.")
                    print("For now, use @tool_name({\"param_name\": \"value\"}) syntax to test tool calls.")

            except KeyboardInterrupt:
                break
            except Exception as e:
                print(f"Error: {e}")

    finally:
        client.disconnect()
        print("\nGoodbye!")


if __name__ == "__main__":

    main()

DON’T PANIC!

You’ll be okay! I know it feels like a loooot of code. But most of it is to perform the task of the LLM programmatically.

Let me explain, and you’ll be fine. I promised.

Starting from the top:

init(): The constructor takes the server URL as input. initialize(): This method will try to connect to the server and send greetings. Upon success, it will request available tools from the server.
_make_request(): This method is used to interact with the server. It’s used to do the initial connection and list available tools. Also, we use request to store the session token that identifies our client.
_handle_sse_response(): This method will parse the response from the server. The SSE protocol starts with the bytes “data: ” so we strip this from the string. Also, the message delimiters are \n\n so we can go through it line by line. Each line should be a different message.
call_tool(): This method will ask the MCP server to call a tool by its name and with the parameters you gave using the client’s CLI. detect_tool_calls(): This function should be entirely replaced by classic LLM “function calling”. For this demo, I parse the tool name and its arguments programmatically by following the convention@tool_name({"arg_name": "value1"}). However, this should be replaced by a “function call” as specified by OpenAI spec’s.
main(): The main chat loop. I hope you survived. Open the code in your favorite Python IDE, and I’m sure it will look less scary!

IV. Using the client

1. Connecting to our local demo server

Now that we have a client and a server. Let’s connect them!

python3 client.py
=== Demo MCP Client ===
Enter MCP server URL (e.g., https://mcp.deepwiki.com/mcp): http://localhost:1337/mcp
Server assigned session ID: 03da4b6c-124a-4274-8e25-8b53e41a5e37
Connected to MCP server: weather-http-server v0.0.1
Available tool: example_tool - An example tool that processes messages
Available tool: weather - Get weather information for a city

=== Chat Loop Started ===
Type your messages below. Use @tool_name(arg1="value1") to call tools.
Type 'quit' to exit, 'tools' to list available tools.


> @weather({"city": "paris"})

Detected tool calls:
  Calling weather with {'city': 'Paris'}
  Result: {
  "content": [
    {
      "type": "text",
      "text": "{\"city\":\"Paris\",\"temperature\":22,\"condition\":\"Sunny\",\"humidity\":45}"
    }
  ]
}

> ^C
Goodbye!

In the above output, we connected to http://localhost:1337/mcp which is our local server.

We can see that the server sent information about the available tools. We ask for the weather tool with the argument “Paris”: @weather({"city": "Paris"}) . Then the server answers with the fake response, which should be added to the LLM context if we had one.

2. Connecting to a real remote MCP server

Do you know DeepWiki? It’s a tool that scans GitHub repositories and allows you to chat with an LLM about the repo’s content. I hate to say this because it was made by the guys who made the Devin crap. But this time it works quite well…

They have a fully working MCP server with 3 tools. Let’s try to connect to it.

python3 client.py
=== Yacana MCP Client ===
Enter MCP server URL (e.g., https://mcp.deepwiki.com/mcp): https://mcp.deepwiki.com/mcp
Server assigned session ID: 68d8083a519da262c385212ad6aeb08c91733479d484af2ad7065fa9614678e2
Connected to MCP server: DeepWiki v0.0.1
Available tool: read_wiki_structure - Get a list of documentation topics for a GitHub repository
Available tool: read_wiki_contents - View documentation about a GitHub repository
Available tool: ask_question - Ask any question about a GitHub repository

=== Chat Loop Started ===
Type your messages below. Use @tool_name(arg1="value1") to call tools.
Type 'quit' to exit, 'tools' to list available tools.


> tools

Available tools:
  - read_wiki_structure: Get a list of documentation topics for a GitHub repository
    Parameters: repoName(string)*
  - read_wiki_contents: View documentation about a GitHub repository
    Parameters: repoName(string)*
  - ask_question: Ask any question about a GitHub repository
    Parameters: repoName(string)*, question(string)*

> @ask_question({"repoName": "rememberSoftwares/yacana", "question": "What is yacana?"})

Detected tool calls:
  Calling ask_question with {'repoName': 'rememberSoftwares/yacana', 'question': 'What is Yacana?'}
  Result: {
  "content": [
    {
      "type": "text",
      "text": "Yacana is an open-source, task-driven multi-agent framework designed to simplify the creation of LLM-powered applications. It focuses on chaining tasks together rather than solely on agents, offering a flexible and intuitive API for both beginners and advanced AI users. \n\n## Core Concepts\n\nThe Yacana framework revolves around two primary classes: `Agent` and `Task`.\n\n### Agent\nAn `Agent` in Yacana represents an LLM (Large Language Model) instance. You can create agents using either `OllamaAgent` for local models or `OpenAiAgent` for OpenAI-compatible APIs. \n\n*   **`OllamaAgent`**: Used for interacting with Ollama inference servers. It requires a `name` and `model_name`, and can optionally take a `system_prompt` and `endpoint`. \n*   **`OpenAiAgent`**: Used for interacting with OpenAI-compatible APIs. It also requires a `name` and `model_name`, along with an `api_token` and `endpoint`. \n\nAgents maintain their own conversation history, allowing for contextual interactions across multiple tasks. \n\n### Task\nA `Task` is the fundamental execution unit in Yacana. It encapsulates a prompt and is executed by an `Agent`. \n\n*   To create a task, you provide a `prompt` (the instruction for the LLM) and an `agent` instance. \n*   The `solve()` method of a `Task` executes the prompt using the assigned agent and returns a `GenericMessage` object, which contains the LLM's response in its `content` attribute. \n\n## Key Features\n\nYacana offers several features to enhance LLM interactions:\n\n*   **Task Chaining**: Tasks can be linked together by reusing the same agent instance, allowing for multi-turn conversations where each task builds upon the agent's history. \n*   **Tool Calling**: Yacana enables LLMs to invoke Python functions (tools) to perform specific actions. This is a core strength, allowing even small open-source models to use tools effectively. \n    *   Tools are defined using the `Tool` class, specifying a `tool_name`, `function_description`, and a `function_ref` (the Python function to be called). \n    *   Unlike other frameworks, tools are made available at the `Task` level, not permanently assigned to agents, reducing \"noise\" and improving relevance. \n*   **Structured Output**: This feature allows you to define a Pydantic model for the LLM's response, ensuring type-safe JSON outputs. \n*   **Streaming Responses**: Yacana supports token-by-token processing of LLM responses, useful for real-time user feedback. \n*   **Multi-Agent Collaboration**: The framework supports orchestrating conversations between multiple agents, enabling complex problem-solving scenarios. \n\n## Installation\nYou can install Yacana using pip: `pip install yacana` \n\n## Example Usage\n\nHere's a basic example demonstrating agent creation and task solving:\n```

python\nfrom yacana import OllamaAgent, Task\n\n# Create agent\nagent = OllamaAgent(\n    \"Research Assistant\", \n    \"llama3.1:8b\",\n    system_prompt=\"You are a helpful research assistant\"\n) \n\n# Execute tasks with chaining\nTask(\"What is machine learning?\", agent).solve() \nclarification = Task(\"Explain that in simpler terms\", agent).solve().content \n\nprint(f\"Simplified explanation: {clarification}\") \n\n# Check conversation history\nagent.history.pretty_print() \n

```\n\n## Notes\nYacana was initially designed for Ollama but now supports any OpenAI-compatible endpoints.  The framework is open-source under the MIT license. \n\nWiki pages you might want to explore:\n- [Quick Start Guide (rememberSoftwares/yacana)](/wiki/rememberSoftwares/yacana#2.1)\n- [Advanced Features (rememberSoftwares/yacana)](/wiki/rememberSoftwares/yacana#6)\n\nView this search on DeepWiki: https://deepwiki.com/search/what-is-yacana_e1a73f0f-7ad2-4c53-8028-9513cca5c37e\n"
    }
  ]
}

> ^C
Goodbye!

In the above output, we connected to https://mcp.deepwiki.com/mcp. During initialization, we retrieve and print the list of available tools.

We use the tools command to print information about the tools parameters. We then call the ask_question tool with two parameters: repoName and question. The MCP server answers our question.

V. Conclusion

MCP is not that simple, and the protocol will evolve. I guess that the STDIO transport will slowly die. Hopefully.

For authentication, you can use the headers member variable to deal with different authorisation flows.

In the meantime, if you want to have fun with LLMs locally or otherwise, go try Yacana!

It’s an agent framework that focuses on simplicity with powerful features. It will soon have MCP support. ;-)

I’ll update this article when I get deeper into this.

Making VLLM work on WSL2

Emilien Lancelot — Fri, 17 Jan 2025 13:24:54 +0000

Running a small llama3 model for demonstration purposes on WSL2 using VLLM

➡️ Read this article on Medium for a better experience.

1. Requirements

WSL version 2
Python: 3.8–3.12
GPU ABOVE GTX1080 (Did not manage to make it work on 1080 as it told me the hardware was too old. :-( Ollama is less picky).

2. Preflight checks

Checking NVCC

In WSL, do:

nvcc --version

▶️ Command not found?

Fixing NVCC: Nvidia Drivers Installation

Visit Nvidia's official website to download and install the Nvidia drivers for WSL. Choose Linux > x86_64 > WSL-Ubuntu > 2.0 > deb (network)

Follow the instructions provided on the page.

Add the following lines to your .bashrc:

export PATH="/usr/local/cuda-12.6/bin:$PATH"
export LD_LIBRARY_PATH="/usr/local/cuda-12.6/lib64:$LD_LIBRARY_PATH"

⚠️ ⚠️ ⚠️ Check the content of "/usr/local" to be sure that you do have the "cuda-12.6" folder. Yours might have a different version.

Reload your configuration and check that all is working as expected

source ~/.bashrc
nvcc --version
nvidia-smi.exe

ℹ️ "nvidia-smi" isn't available on WSL so just verify that the .exe one detects your hardware. Both commands should displayed gibberish but no apparent errors.

3. Creating the environment

python3 --version # copy the version
conda create -n myenv python=3.10 -y # Update the python version with your own

▶️ Don't have conda?

mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm ~/miniconda3/miniconda.sh
source ~/miniconda3/bin/activate

conda create -n myenv python=3.10 -y # Update the python version with your own

4. Installing VLLM

Installing VLLM:

pip install vllm

Trying to start the inference server with a tiny LLM:

vllm serve facebook/opt-125m

▶️ Runtime crash of VLLM?

    from torch._C import *  # noqa: F403
    ^^^^^^^^^^^^^^^^^^^^^^
ImportError: /home/xxxx/vllm_serve/lib/python3.11/site-packages/torch/lib/../../nvidia/cusparse/lib/libcusparse.so.12: undefined symbol: __nvJitLinkComplete_12_4, version libnvJitLink.so.12

Well, this is where sh*t hits the fan. I recommend that you try the fix below. However, if it doesn't work, I can only wish you good luck. It's another one of those technologies that have error messages written by a depressive data scientist. So just scroll to the top of the stacktrace and hope for the best. Google is your friend. Have faith.

Potential VLLM serve fix:

python -m pip uninstall torch torchvision torchaudio
python -m pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu121
vllm serve facebook/opt-125m # Should be working now...

https://github.com/pytorch/pytorch/issues/111469

5. Running VLLM

Let's try with a tiny Facebook LLM.

Create an account on Hugging Face and then create a api-key.
Then go to the model page you want to try out.
For us: https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct
At the top of the page there will be some kind of form you must apply to.

When done, you'll get an email 10 minutes later telling you you've got access. 🎉

To connect to the VLLM server try this piece of code:

from vllm import LLM
from huggingface_hub import login

login("<REPLACE-ME-WITH-YOUR-HUGGING-FACE_TOKEN") # Replace this !

# Load the LLama 3 8B model
model = LLM("meta-llama/Llama-3.2-1B-Instruct")

from transformers import AutoTokenizer

# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B-Instruct")

# Prepare the input message
messages = [{"role": "user", "content": "What is the capital of France?"}]
formatted_prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

# Generate the output
output = model.generate(formatted_prompt)
print(output)

Run it and in all the mess you'll see:

text='The capital of France is Paris.'

Serving using OpenAI style

VLLM is OpenAI compatible to some extent. We can use this ability to use the OpenAI client instead of the Hugging Face one.

from openai import OpenAI
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="token-abc123",
)

completion = client.chat.completions.create(
  model="meta-llama/Llama-3.2-1B-Instruct",
  messages=[
    {"role": "user", "content": "Hello!"}
  ]
)

print(completion.choices[0].message)

Should output something like:

ChatCompletionMessage(content='Hello! How can I assist you today?', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=[])

6. Manage installed models

List models

ls ~/.cache/huggingface/hub
models--facebook--opt-125m  version.txt

Delete models

rm -rf ~/.cache/huggingface/hub/models--<model-name>

In need of a simple yet effective multi-agent framework with working tool calling for any LLMs ?
I've got you covered! Go try Yacana!

HF.

The long road to an Agentic framework that Won’t Burn You Out

Emilien Lancelot — Wed, 25 Sep 2024 22:11:12 +0000

▶️▶️ Read this article on Medium.◀️◀️

I. The Leak That Sparked Open Source Innovation

Like most of us, I started my AI journey with a simple llama.cpp wrapper. It was a simple chat loop with no memory between messages but… IT SPOKE!

And that it did very well. This was an exciting perspective that pulled us back in time. In 2008 exactly. At a time when the first IronMan aired in theaters and speaking AI was to become one of the most trending things in movies.

Since then we have had many speaking "smart" assistants on smartphones. Even though it was fun, I must have used it no more than 10 minutes in total during the 9 years I owned my Galaxy-S4.

The human AI…But speaking AIs continued spawning, each time getting us closer to a functional Jarvis. But the more we would come close to it, the more it fainted. Showing limitations that killed the experience quicker than it took us to deactivate the power-hungry functionality. No Google Home or Alexa would save us from dumb AIs.

But then it happened!

ChatGPT came out of nowhere and a new spark was created. A shimmering light showed us a glimpse of hope. Hope that we didn't know was to be a new revolution, a gift we weren't ready to receive.

But only a spark it was. And to get us all back in the game we needed a fire!

Hopefully, someone took the responsibility and lit the blaze.

Some guy, which we still don't know anything about, leaked the first Large Language Model (LLM) on 4Chan. It was Facebook's, for which I won't cry as they have fed on us for far too long. This long-coming reckoning was to be the spark that lit the entire forest inside developers! We could boot an LLM directly on our computers! We would speak to it, and it responded. And damn it was good. At the same time, image generation was on the loose, too, with Midjourney and the first open source models being released every week. Not one day had passed since a new web UI was created. Not one day had passed without one new model iteration being revealed.

The Jarvis dream was back, and this time, it was real!

II. I tested them all…

To use an LLM you need 3 things:

An LLM
An inference server to run the above LLM
A way to interact with the above inference server

As an inference server, I started with LMstudio which had the good taste of displaying the amount of RAM needed to run the model. A fitting functionality that showed once more that successful tech is driven by small details.

I then switched to Ollama because, as a developer, I'm more into command lines. Also, the performance was a bit better, and layers would be loaded in RAM by the software itself. Less to configure yourself is always better as you don't have infinite time to deal with the ever-changing landscape of AI.

Both LMstudio and Ollama had chat integration but what to do with it?

The Jarvis ideal could only be achieved if we had a way to programmatically interact with the LLM.

CrewAI

Many of us started there. The notion of Agents embodying one instance of an LLM had us started with CrewAI for one reason. A reason that existed since the dawn of time!

It was EASY!

A simple and straightforward definition of an agent

CrewAI had this straightforward agent definition that allowed developers to start prototyping quickly. It mostly used multi-turn chat, which let us believe that LLMs speaking with themselves would solve all our problems, including world hunger.

Unfortunately, making them speak was not an issue. But getting things done was a whole other fight, and CrewAI just didn't deliver. Prompts were too complex for our small 8B LLMs that had reasoning troubles. I guess that plugging CrewAI with a frontier model like ChatGPT yielded better results, but the open source community couldn't use CrewAI as performance was just too bad.

LangGraph

LangGraph reinvented how to deal with agents.

It was said that "compiling a graph of predicted prompts would help guide the LLM toward truthness" and was the way to go with our dumb open source LLMs. And they were right! At least the conceptual idea was good.

Example of a compiled graph

It shed light on how bad the performances of open source LLMs were and how much guidance they needed to actually achieve computable results.

But the issue with LangGraph was the sheer complexity of it. Also, the bad documentation didn't help. It was outdated, missed parts, and had many wrong code snippets.

On the graph side, the hardcoded links lacked flexibility as software development needs room to evolve during its creation cycle!

Modifying the graph representation of possible interactions was a lot of work, and prompt engineering pseudo-science made the whole thing exhausting. Programming doesn't like hard links between things. Programming is about iterating a lot and having things stitched to each other will inevitably slow you down.

In the end, I raged-quit LangChain as it was too difficult to learn and the documentation was too unhelpful.

The Tool Calling disaster

Things started to look bad when we discovered that chatting programmatically with an LLM was cool but didn't achieve anything. To create something with our LLM it had to interact with the exterior world. Not just write poems or bad jokes.

This is why "tool calling" was created. It allowed LLMs to call 'classic' programming functions. For instance, it could get the weather of a city by calling a real weather API or accessing files on your computer. The possibilities were endless.

This was the start of something awesome and we all went looking for a way to make this real.

The tool-calling mechanism was first available in chatGPT. It had this silly name of "function calling" that made us all believe that chatGPT could magically call a programming function.

We were wrong. 'Function calling' was merely a way of outputting text in a standardized way. The way in question was JSON.

The LLM would produce a known JSON architecture that was parsed by the software running the LLM. Recognizing the pattern it would call the given function with the right parameters.

OpenAI's "function calling" JSON to trigger a tool call

The idea was good. But the JSON architecture was not. When tried with small 8B models it failed to output the JSON correctly hence making the tool calling impossible. 😭

▶️ Therefore, the community was stuck with text-spewing LLMs that couldn't enter the tool-calling league of frontier models.

Following this, new models came with specific training on tool calling to help them output the correct JSON hence splitting models into two categories. The one with tool-calling abilities and the one without.

The one without became of no interest and the one that could, still didn't have this good of a success rate. Better than basic LLMs but not that great... Calling a function was still a complex endeavor.

CrewAI was incapable of calling any tools. Making it simple but useless.
LangGraph on the other hand could output the JSON but not call the function by itself. It would have been your job to parse the JSON and call it yourself. Also, for opensource models, it only worked with Ollama but only had partial support because… Why care about opensource ?

In my quest to find the best framework I tested many others: Autogen, BabyAGI, AutoGPT, Langoid, etc. None that had a correct Agent implementation and tool-calling support for local open source LLMs. None.

My very own Jarvis dream was about to go to rest once again as I would never plug anything into chatGPT. As a strong open source believer and even though I love chatGPT I'm not letting it access my emails, my domotic, my life… It must be open source or nothing. And nothing it was…

III. The solution

On the blink of dispair some unknown tool came up. It took the form of a llama in a deep dark sky. A constellation of small sparkles that may light the path ahead after all!

Yacana

Github: https://github.com/rememberSoftwares/yacana
Read The Docs: https://remembersoftwares.github.io/yacana

I. Installation

All it took was a simple pip install:

pip install yacana

II. Agents

The 'Agents' concept is the human way of speaking to LLMs. We have anthropomorphized it so that it feels more natural. Yacana understands that and allows creating Agents with a name (and an optional system_prompt to set their behavior)

⚠️Just a heads-up: for now, Yacana only supports Ollama. If you don't have it installed already, it's only one command away.

from yacana import Agent

agent1 = Agent("Experience book writter", "llama3.1:8b")

The second parameter is the LLM model name used by Ollama.

III. Tasks

Chaining prompts together was an idea of LangGraph. Even though no graphs are available in Yacana we do get the same feeling of linking prompts together but not hard linking them!

We can soft link tasks with something we, as developers, have done all our lives: programming! No fancy web UIs needed! To chain two tasks together you simply have to call them one after the other. This might look silly right now, but you'll soon see why it's not.

from yacana import Task

task1 = Task("Write a few lines about the deep blue sky", agent1)
task2 = Task("Change the subject to 'ocean'", agent1)

task1.solve()
task2.solve()

$ python writer.py

Output of our two tasks that the agent solved

Yacana shows prompts in green and the LLM answer in purple. This makes tracking conversations a bit easier.

Let's do some basic programming!

So, Yacana doesn't provide graphs. But you don't need graphs! You've got something way better! Object Oriented Programming (OOP)!

We'll start by making an Animal base class:

class Animal:
    def __init__(self, name):
        self.name = name

    def make_sound(self):
        raise NotImplementedError("Subclass must implement abstract method")

This will be the parent class for the next two classes Cat and Dog :

# Subclass Cat
class Cat(Animal):
    def make_sound(self):
        return f"{self.name} says: Meow!"
# Subclass Dog
class Dog(Animal):
    def make_sound(self):
        return f"{self.name} says: Woof!"

Instantiating our pets:

cat = Cat("Whiskers")
dog = Dog("Buddy")

print(cat.make_sound())
print(dog.make_sound())

🔼 Output of this very complexe program...

Now let's change our hardcoded pets to AI cyber pets!

CyberCat and CyberDog playing Doom via Bluetooth

from yacana import Agent, Task

# Parent class
class Animal:
    def __init__(self, name, system_prompt):
        self.agent: Agent = Agent(name, "llama3:8b", system_prompt=system_prompt)

    def make_sound(self):
        raise NotImplementedError("Subclass must implement abstract method")

# Subclass Cat
class CyberCat(Animal):

    def make_sound(self):
        task = Task("Do a cat noise", self.agent).solve()
        return task.content

# Subclass Dog
class CyberDog(Animal):

    def make_sound(self):
        task = Task("Do a dog noise", self.agent).solve()
        return task.content

# Example usage
cyber_cat = CyberCat("Whiskers", "You are a cat called Whiskers")
cyber_dog = CyberDog("Buddy", "You are a dog called Buddy")

print(cyber_cat.make_sound())
print(cyber_dog.make_sound())

We now have

An Agent defined in the Animal class.
Two Cyber AI pets CyberCat and CyberDog that each take a name and a system_prompt.
The .make_soud() method now has the task of generating a sound based on the sytem_prompt representing itself.

ℹ️ The default logging can be deactivated to get a cleaner output.

Tool calling

What's great with cyber pets is that they can buy their kibble themselves!

Let's make a tool that simulates calling a web API to order more kibble.

def order_kibble(animal: str, weight_grams: int) -> str:
    # Validation
    if animal.lower() != "cat" and animal.lower() != "dog":
        raise ToolError("Animal parameter can only be one word which describes your class like 'cat' or 'dog'.")
    if not isinstance(weight_grams, int):
        raise ToolError("Parameter weight_grams mut be an integer.")

    # Making a fake call to https://order-kibble.com

    # Fake API return
    if weight_grams > 500:
        return "To much kibble was ordered for one day. Please order less or you'll become a fat pet..."
    else:
        return f"Successfully ordered {weight_grams} grams of {animal.lower()} kibble on order-kibble.com!"

The important parts of this function are:

The function name will be used by Yacana so choose something meaningful.
Duck typing the arguments: Yacana will use this information to infer the types of each parameter for the function call.
Implementing validation and raising meaningful errors: Same as classic server-side validation we cannot trust the LLM to send us valid data. You should at least check for types. If something is wrong raise a meaningful ToolError(...) exception message that the LLM will use to try and call the tool again!
If it's a success then return a meaningful string relevant to the tool call. Note that the final result of the Task will be the Tool return value.

Creating a Tool from a function is as simple as choosing a tool name, description, and function reference:

Tool("Kibble_order",
     "Takes your animal type (cat, doc, etc) and the quantity of kibble to order as input and places an order.",
     order_kibble)

Now let's add a .run() method to our Animal class. Each time it's called it loses 100 calories (max 200) and when reaching 0 calories the pet orders a kibble refill by itself. Good boy!

def run(self):
    # Running!
    running_task_output: str = Task("Do running noises.", self.agent).solve().content

    # Losing calories
    self.calorie -= 100

    if self.calorie <= 0:
        # Need to eat! Let's order some kibble!
        tool_output: str = Task("You are hungry. Order kibble.", self.agent, tools=[self.kibble_order_tool]).solve().content
        print(tool_output)
        self.calorie = 200

    return running_task_output

Full code below before making our pets run:

from yacana import Agent, Task, Tool, ToolError

def order_kibble(animal: str, weight_grams: int) -> str:
    # Validation
    if animal.lower() != "cat" and animal.lower() != "dog":
        raise ToolError("Animal parameter can only be one word which describes your class like 'cat' or 'dog'.")
    if not isinstance(weight_grams, int):
        raise ToolError("Parameter weight_grams mut be an integer.")

    # Making a call to https://order-kibble.com

    # Fake API return
    if weight_grams > 500:
        return "To much kibble was ordered for one day. Please order less or you'll become a fat pet..."
    else:
        return f"Successfully ordered {weight_grams} grams of {animal.lower()} kibble on order-kibble.com!"

# Parent class
class Animal:
    def __init__(self, name, system_prompt):
        self.agent: Agent = Agent(name, "llama3:8b", system_prompt=system_prompt)
        self.kibble_order_tool: Tool = Tool("Kibble_order", "Takes your animal type (cat, doc, etc) and the quantity of kibble to order as input and places an order.", order_kibble)
        self.calorie = 200

    def make_sound(self):
        raise NotImplementedError("Subclass must implement abstract method")

    def run(self):
        # Running!
        running_task_output: str = Task("Do running noises.", self.agent).solve().content

        # Losing calories
        self.calorie -= 100

        if self.calorie <= 0:
            # Need to eat! Let's order some kibble!
            tool_output: str = Task("You are hungry. Order kibble.", self.agent, tools=[self.kibble_order_tool]).solve().content
            print(tool_output)
            self.calorie = 200

        return running_task_output

# Subclass Cat
class CyberCat(Animal):

    def make_sound(self):
        task = Task("Do a cat noise", self.agent).solve()
        return task.content

# Subclass Dog
class CyberDog(Animal):

    def make_sound(self):
        task = Task("Do a dog noise", self.agent).solve()
        return task.content

Let's make our dog run twice so that its calories reach 0 and it needs to refill!

cyber_dog = CyberDog("Buddy", "You are a dog")

cyber_dog.run()
cyber_dog.run()

Let's break down the output:

🔼 The dog ran twice as expected.

🔼 As the calories reached 0 it triggered the tool call to order kibble. Here you can witness Yacana's Tool-calling magic! But something went wrong 😨! Can you spot the issue??
…
The weight_grams value was sent as a string : {"annimal": "dog", "weight_grams": "500"} Although, it was ducked typed as an int!

🔼 Lastly, we can see that Yacana got our raise() message about the parameter type and retried calling the tool.

The LLM took into account the new information and this time, weight_grams was sent as an integer and the tool was called successfully!

Our cyber dog placed its order! 🐶

In the end, Yacana does not provide a static graph. However, it does integrate quite nicely into standard development practices. From Yacana's only POV this is how it is conceptually created:

Graph nodes version of Yacana

IV. Going further

We'll wrap up here for this introduction to Yacana. However, there are many more functionalities to explore and many use cases other than dystopic cyber pets to imagine. From routing to multi-turn chat and history management, the framework has many things to propose.

Consider giving a like to this article and following me on Medium and dev.to so you won't miss the coming series of articles about Yacana.

Have fun with the framework and read the doc: https://remembersoftwares.github.io/yacana

Calling code with local LLM is a hoax

Emilien Lancelot — Mon, 20 May 2024 16:26:02 +0000

Having a local LLM spewing text is good. But what you need is the LLM to execute YOUR code!

Introduction

Is calling tools even doable? Sure chatGPT makes it easy. But what of your local LLMs. In this article, we'll be trying multiple agent frameworks with tool-calling capabilities and see if our local LLM can use them.

My configuration is:

RTX4090 with 32GB of RAM

Using the following LLMs for testing:

llama3:8b
dolphin-mixtral:8x7b-v2.7-q4_K_M
mistral:latest

Powered locally by Ollama.

I. AutoGPT

AutoGPT is a framework that seems nice. It has a cool CLI and a flutter UI to create agents from the browser. Its main purpose is to work with your local stuff (documents, audio, videos, etc)

BUT

It mostly relies on chatGPT or any proprietary LLM providers to do the heavy lifting. At least that's how I understand it.

Using local models

Here you can find the configuration file where we must set our config.

We must trick AutoGPT to use the Ollama endpoint like it's chatGPT.

## OPENAI_API_KEY - OpenAI API Key (Example: sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx)
OPENAI_API_KEY="helloworld"

...

## OPENAI_API_BASE_URL - Custom url for the OpenAI API, useful for connecting to custom backends. No effect if USE_AZURE is true, leave blank to keep the default url
# the following is an example:
OPENAI_API_BASE_URL=http://localhost:11434/v1

...

## SMART_LLM - Smart language model (Default: gpt-4-turbo)
SMART_LLM=dolphin-mixtral:8x7b-v2.7-q4_K_M

## FAST_LLM - Fast language model (Default: gpt-3.5-turbo)
FAST_LLM=mistral:latest

This should do the trick.

./autogpt.sh run

value is not a valid enumeration member; permitted: 'text-embedding-ada-002', 'text-embedding-3-small', 'text-embedding-3-large', 'gpt-3.5-turbo-0301', 'gpt-3.5-turbo-0613', 'gpt-3.5-turbo-16k-0613', 'gpt-3.5-turbo-1106', 'gpt-3.5-turbo-0125', 'gpt-3.5-turbo', 'gpt-3.5-turbo-16k', 'gpt-4-0314', 'gpt-4-32k-0314', 'gpt-4-0613', 'gpt-4-32k-0613', 'gpt-4-1106-preview', 'gpt-4-1106-vision-preview', 'gpt-4-0125-preview', 'gpt-4-turbo-2024-04-09', 'gpt-4', 'gpt-4-32k', 'gpt-4-turbo', 'gpt-4-turbo-preview', 'gpt-4-vision-preview' (type=type_error.enum; enum_values=[<OpenAIModelName.EMBEDDING_v2: 'text-embedding-ada-002'>, <OpenAIModelName.EMBEDDING_v3_S: 'text-embedding-3-small'>, <OpenAIModelName.EMBEDDING_v3_L: 'text-embedding-3-large'>, <OpenAIModelName.GPT3_v1: 'gpt-3.5-turbo-0301'>, <OpenAIModelName.GPT3_v2: 'gpt-3.5-turbo-0613'>, <OpenAIModelName.GPT3_v2_16k: 'gpt-3.5-turbo-16k-0613'>, <OpenAIModelName.GPT3_v3: 'gpt-3.5-turbo-1106'>, <OpenAIModelName.GPT3_v4: 'gpt-3.5-turbo-0125'>, <OpenAIModelName.GPT3_ROLLING: 'gpt-3.5-turbo'>, <OpenAIModelName.GPT3_ROLLING_16k: 'gpt-3.5-turbo-16k'>, <OpenAIModelName.GPT4_v1: 'gpt-4-0314'>, <OpenAIModelName.GPT4_v1_32k: 'gpt-4-32k-0314'>, <OpenAIModelName.GPT4_v2: 'gpt-4-0613'>, <OpenAIModelName.GPT4_v2_32k: 'gpt-4-32k-0613'>, <OpenAIModelName.GPT4_v3: 'gpt-4-1106-preview'>, <OpenAIModelName.GPT4_v3_VISION: 'gpt-4-1106-vision-preview'>, <OpenAIModelName.GPT4_v4: 'gpt-4-0125-preview'>, <OpenAIModelName.GPT4_v5: 'gpt-4-turbo-2024-04-09'>, <OpenAIModelName.GPT4_ROLLING: 'gpt-4'>, <OpenAIModelName.GPT4_ROLLING_32k: 'gpt-4-32k'>, <OpenAIModelName.GPT4_TURBO: 'gpt-4-turbo'>, <OpenAIModelName.GPT4_TURBO_PREVIEW: 'gpt-4-turbo-preview'>, <OpenAIModelName.GPT4_VISION: 'gpt-4-vision-preview'>])

Seems like it's not... The model name MUST be a proprietary name like "GPT4-turbo" or any other from the above list. Unfortunately, my models are not named like that.

Now to see If it could go a bit further with a fake (but compliant) model name I set "GPT4-turbo" and ran again.

./autogpt.sh run
2024-05-19 16:03:01,937 ERROR  Invalid OpenAI API key! Please set your OpenAI API key in .env or as an environment variable.
2024-05-19 16:03:01,938 INFO  You can get your key from https://platform.openai.com/account/api-keys

It doesn't like my API key. I've tried many different keys. It won't go further.

Clonclusion on autoGPT

To fix the model names you could create a custom model in Ollama called GPT4-turbo which you would base on any local model you already have. It's just a way to rename your model and trick AutoGPT. But that wouldn't fix the API key error.

Also as mentioned HERE you could maybe duplicate the OpenAi model provider file from AutoGPT and remove any non-compliant parts. But I'm unsure how to perform such an operation.

The documentation doesn't have anything about using local models and doesn't mention calling tools.

In the end, I don't think that AutoGPT is ready for local model use and you should wait and hope the paradigm shifts toward a more local approach.

II. LangChain & LangGraph

Langchain has been at the core of many projects since the beginning of the AI gold rush. Why it's not the king already is probably because of its complex syntax that many developers don't have the time to learn.

Langchain has a way of using the most obscure Python functionalities and makes you feel like you never have read Python code before.

For instance:

chain = prompt | model | outputparser
chain.invoke("Question.")

The LCEL system of Python uses pipes ("|") to string things up. This is rendered possible in Python by overriding Python's __or__ magic method. in other words, Langchain overrides operators like you would in C++.

But did we really need this kind of idea? I'll let you make up your mind…

Now about using local models:

Langchain has 2 plugins:

Ollama chat: Allows you to chat with an LLM
Ollama-functions: Allows an LLM to answer using a specific formatting output. For instance, if you want your LLM to answer as JSON or as YAML then you can define the format type, keys, and values type that you expect.

Beware with the "function calling" capability ! It's a troll from OpenAI… Worst feature naming possible! It doesn't calls functions like "using tools" would. It's only about formating the output of the LLM.

=> Now, what about tool calling (aka executing real code locally)?

Well… The Ollama plugin doesn't have this functionality…

@tool
def multiply(first_number: int, second_number: int):
    """Multiplies two numbers together."""
    return first_number * second_number

model = ChatOllama(model="mistral:latest")
model_with_tools = model.bind_tools([multiply]) # <== Binding tool here

Running this will output:

ChatOllama doesn't have a method bind_tools()

And I can confirm it doesn't… So we're F*****.

Conclusion on Langchain & Langraph

I'm a bit disappointed. As this framework powers many others like CrewAI etc I thought that it would have a nice integration with local tools. In the end, it's not that great. It's just a complicated mess that doesn't fix our main concern.

III. Rivet

I must say that I love this one! It's kind of new but has tremendous potential in the future.

It's some kind of IDE for LLM interactions that uses a canvas to create an execution diagram (DAG). It can run in the browser but you can also export the DAG and run it as code to empower your software.

Look at this! How cool it looks! There is an Ollama plugin so you can use it locally.

Just make sure to click the 3 dots in the top right and change the executor to "node" otherwise it might not run.

Unfortunately, I didn't find a way to call custom tools and the documentation is quite lacking anyway. It's clearly a project that needs to be watched for further updates!

Conclusion on Rivet

Cool software! Free and open source. Love the canvas system that feels like what LangGraph should have.

Still requires tool calling before being useful. But have fun with it if you have a chatGPT account.

IV. AutoGen

One of the best candidates on this list. Autogen is backed up by one of the largest tech companies out there.

I have done the tutorial and I must say… I don't understand most of what I'm doing! If the first pages are okay, the situation rapidly gets out of control and then you would need an AI agent framework to explain to you how all of this works.

However, it does have all you need and supports Ollama out of the box:

code_writer_agent = ConversableAgent(
    "code_writer_agent",
    system_message=code_writer_system_message,
    llm_config={"config_list": 
      [{"model": "dolphin-mixtral:8x7b-v2.7-q4_K_M",
      "api_key": "hello world",
      "base_url": "http://127.0.0.1:11434/v1"}]},
    code_execution_config=False,
)

The best functionalities are IMO:

Generating code on the fly and executing it
Call tools (aka calling your code)
Human input

But does tool calling works ?

Still doesn't… Only OpenAI-compatible tools calling LLMs can use this. So Ollama + Mistral won't make the cut. However, the code generation and execution thingy works quite well. Also, note that calling LangChain tools is not supported.

Available chat mechanisms you can use

Two chats pattern: Two LLMs speak to each other to complete the task

Sequential chats: Tasks will be evaluated in the order you specified

This is starting to get complicated. The carryover mechanism which contains the context accumulated over the multiple conversations is a hard concept to grasp. And why is each task still is a conversation between 2 agents ?? And why is it A=>B, A=>C, A=>D, A=>E ? Why always start with A ? God knows.

Group chat: Don't expect an explanation...

This is when things get out of hand! One agent seems to be the brain, installing some kind of hierarchy between the agents. If the concept is appealing, the examples from the documentation are not really helpful.

It also supports the last trending prompting stuff like:

ReAct: Allows to decompose actions and make a plan. Then it tries to follow each step and if things go wrong it makes another plan and starts again. It's all about creating context that has a semantic meaning to the LLM and helps it focus on what it should do right now.
Reflection: It's kind of like ReAct but with an emphasis on its own output. After "speaking" it will ask itself "Is this correct ?". And it seems that iterating over its own answers yields better results.

As always, "better results" means "fewer hallucinations" as this is the main issue with LLMs.

AutogenStudio

Also if you don't want to mess with code you can download the AutoGenStudio software that allows you to define agents without the need of coding. It's an interesting piece of software but doesn't really help you grasp the core functionality of the framework.

Conclusion on AutoGen

AutoGen clearly has a bright future in front of it. As it's made by Microsoft, we can only hope that they won't pull the plug on it or make it an OpenAI only software.

However, still, no tool calling is available with local LLMs. :-(

V. CrewAI

Another excellent piece of software.

If the documentation is okay and the framework simple it does have a few issues.

On the brightside :

Ollama support
LangChain tools calling
Custom tools calling
Human input

On the dark side:

Tool calling still isn't working!
Human input doesn't always trigger
Low consistency with infinite loops
Bugs
Soooooo many prompts to write

Available chat mechanisms you can use

There are "sequential" and "hierarchical". Sequential will allow your LLMs to go through the tasks in the order you choose. Hierarchical on the other side will create a ghost agent that automatically decides which one of your agents should be triggered using its description.

Hierarchical would be great if only it worked. There are constant errors about agents that can find their co-workers. It rapidly gets tedious.

The Framework proposes 3 types of classes:

First, you have the Agents which have the following prompts bound to them:

A role: What it does for a living
A goal: What it should do in the team
A backstory: A story of its life…

writer = Agent(
  role='Writer',
  goal='Write a fake anecdote using a number.',
  backstory='An experienced writer with vivid imagination.',
  llm=ollama_mistral,
  verbose=True
)

Then you have the Tasks that also have prompts:

description: What should the task do
expected_output: The output that is expected of this task

teacher_task = Task(
  description='Decompose the arithmetic operations.',
  expected_output='A consise list of operation to execute',
  agent=teacher
)

Finally, you have the Tools:

Tools can be bound to Agents to give them capabilities. But for some reason, they can also be bound to tasks… Which I don't think makes much sense.

@tool("sleep")
def my_sleep(nb_seconds: int) -> str:
    """Will sleep the amount of specified seconds provided as a number"""
    print(nb_seconds)
    return time.sleep(nb_seconds)

I like having the @tool decorator. You simply have to pass a string that describes your tool and the LLM should know if it should use it or not.

In the end, you'll have so many prompts to write that you'll lose yourself.

Does this prompt belong to a task or an agent? Does this tool belong to this agent or this one? Or maybe the tool should be bound to the task itself… So many questions but so few answers as the CrewAI documentation is quite scarce!

Conclusion on crewAI

If you wish to have agents speak to each other then it's the simpler framework out there. Besides having to many prompts to write it's quick and easy. However, calling tools don't work so we still have the same issue.

Also, the consistency is quite bad. Often will you see you're agents going into infinite loops.

A note on the constant Youtube AI trend bullshit: Haven't you noticed how many YouTubers have made videos on Agent frameworks. The subject is always about writing about stupid AI trends and making poor RAG systems. Well, that's because there is currently not much you can do as calling local tools isn't a thing right now. Except if you use chatGPT, Grok, or Claude.

VI. Conclusion of all conclusions

We're screwed.

Honestly, it's time that we get a way to better integrate low-cost LLMs in our applications. Calling tools is the way to go but needs a simpler architecture and one that doesn't rely on openAI's complex format.

What use of small models like PHI would we have on our mobile devices if the only thing it can do is spew text and can't integrate any of it with our applications?

If I have made mistakes or overseen anything please let me know in the comments.

Any idea how to get local code executed is good to know. Please advise in the comment section !

Thx for reading. Leave a thumbs up if you liked this article. ❤

In need of an agent framework that works with both local and remote inference servers?

▶️ Checkout Yacana

Shielding Your Kubernetes Network: Mastering iptables for Enhanced Security

Emilien Lancelot — Sun, 11 Feb 2024 14:38:32 +0000

I. Introduction

Would you like to hear the good news or the bad news first?

Let's begin with the downside. If you've solely relied on securing the internal cluster network with NetworkPolicy during your recent on-premise K8S installation, you've missed out on 50% of the security measures. Many threats originate from within, but they could just as easily infiltrate from outside. Bummer… I know!

Hey! At least you did half of it! I'm sure that some people reading this blog are like "Huh what are NetworkPolicies anyway ?". Those cannot be saved… May god have mercy on their souls.

Now to the good news part! Securing incoming requests to your cluster can also be achieved and I've got you covered!

Why you should shield your cluster from the exterior world?

As mentioned previously, numerous threats may arise from within. For instance, the act of pulling and executing a rogue Docker image could potentially corrupt your nodes, propagate across applications, and ultimately compromise your entire infrastructure.

⬆️A container that seems fishy; Something doesn't look right…🤨

Let's not dwell on that frightening idea any longer and address our specific concerns!

Kubernetes is composed of masters, workers and ETCD nodes. All these components are working from inside virtual machines (VMs) that are part of a network.

⬆️Network around Kubernetes components

The network that connects these VMs has 3 purposes:

Allow each VM (K8S component) to communicate with each other (In green 📗)
Allow developers to interact with Kubernetes to deploy their payloads (In blue 📘)
Allow engineers to connect to the underlying VMs that support K8S for debugging and stuff (In orange📙)

Securing this network is of paramount importance as one of the VMs could get infected and spread to all others.

On one hand, if your VMs are part of a private network then you might feel safe but in reality, it's private until it's not… And that's when sh*t hits the fan!

On the other hand, if your Kubernetes cluster is available directly from the internet then you really should read this tutorial carefully as all your nodes might currently be screaming H4CK ME PLEASE!

⬆️Hackers attacking exposed VMs that have no protection from incoming requests

There are only two ways to access a system. Physically or from a network endpoint. So unless you find yourself tied up to your chair with a man equipped with a malevolent USB key, the primary threat will definitely come from the network.

⬆️If that's you then you and your clusters are in deep trouble!

In general, hackers search for open ports on the exposed VMs. As most ports are dedicated to a specific program or protocol, for instance, port 22 is always SSH, listing opened ports gives a good idea of what runs on the target. With this list, the hacker can then search for an exploit and hack his way in. So the fewer open ports, the better.

Benefits of limiting opened ports on underlying VMs:

protects your infrastructure from infection
slows propagation of malware in case of infection
allows better control of what (custom) software runs on the VMs

What technology to use for restricting incoming requests and logging them

On Linux, there are many ways to secure incoming network requests. It's called "firewalls". You may have heard of "firewallD" or "UFW" (Uncomplicated Fire Wall (it's a lie)). The truth is, I can't just give you the instructions for all existing firewalls. Moreover, I did tests using a few and didn't quite achieve adequate results and broke the Kubernetes cluster quite a few times.

In the end, I think that the best technology to use when it comes to networks and you don't have one hundred rules to write is: "iptables"!

I'm sure you've heard of it. It's older than you. It was created during the Jurassic era but is still relevant today.

⬆️iptables hype during Jurassic

Even better, most firewalls are simple overlays of iptables, manipulating their rules for you like a wand and dark magic.

But fear not. Old doesn't (always) mean that it sucks. I have mastered the beast and if you read thoroughly the next sections I'm sure that I can pass on this knowledge to you as it was passed to me by an old but wise sage.

II. How to use iptables dark magic?

A network metaphor that you'll love

When a happy network packet is received by a VM it goes through the iptables administration. The packet will go through about 7 different rooms before reaching its destination.

⬆️The process of a network packet

Each room above is called a "Table" in iptable jargon

Each room is responsible for evaluating the network packet and decides if it can continue the process onto the next room or gets kicked out!

To achieve this selection each room has multiple desks to which the packet must report. Each room has about 1 to 5 desks depending on the room type.

Those desks are called "chains" in iptable jargon

Sat to those desks, are overly suspicious old ladies analyzing packets. They have a list of rules that a packet criteria can match and take action if it does.

⬆️Meet "Ms. Input" with her suspicious look. No weird packets at her desk will go unnoticed by this expert eye…

If the packet matches one of the criteria (rules) then two things can happen:

It can be ACCEPTED immediately and go to the next room even if there are other desks left in the room. Its process can continue! Wishing it good luck with the next room!
It can be REJECTED and gets killed on the spot! 😱

IMPORTANT CONCEPT HERE

=> By default, if the packet doesn't match any rules at any desks, it will still arrive at destination! This behaviour is called "default accept". When we finish setting our rules to match the packets we trust , we will change the default behaviour to "default deny". This means that if a packet didn't match any rules at any desks it will be killed by default when leaving the last desk available!

But, good news traveler!

One positive aspect is that we're solely focused on one room and one desk! And that's fantastic - do you know why? Because I don't come close to understanding what half the other old ladies do and neither will you (or you wouldn't be reading this tutorial in the first place).

So let's focus on the only room and desk that interests us. It's the Filter room!

⬆️In red is the only room of iptables that we will cover

Understanding the filter table and INPUT chain

Let's lose the metaphor for a bit and repeat what is what. A room is called a "table" which the binary "iptables" gets its name from. And each desk is called a "chain". You'll get used to it.

For some reason, chains are always written in caps. I guess that the guy who invented iptables was so old and deaf that people constantly yelled at him so he could hear them. Hence the cap! It's just a theory though…

The filter table (room) is all about filtering... That you may have figured out and you're right, WP! That's perfect because we want to filter incoming packets!

The Filter table has 3 default chains (desks)

INPUT
FORWARD
OUTPUT

⬆️We are only interested in Mrs. Input and her list of rules

And do you know what chain (desk) a paquet would go through first when entering the Filter table? The…. THE…… INNNNNPUUTTT chain. Yes, yes. Well done.

The chain concept is easy to grasp. It's like functions in programming but instead of containing actual code they contain rules. The INPUT, FORWARD and OUTPUT chains are the default ones but you can create your own chains! You can have your own desk and an old lady just for your use case! Isn't that great? You just have to call this new chain from the INPUT chain like a programming function would!

Checking our current configuration

Let's see how are our iptables are configured using the following command:

iptables -xvnL

⬆️The content of our "filter" table and 4 chains

At first, this might be a bit overwhelming with information but if you decompose it chain by chain it's quite easy. It's composed of:

The 3 default chains of the filter table: INPUT, FORWARD and OUTPUT
A custom chain called DOCKER added by the docker engine

We can see that the INPUT chain is empty. There are no rules so no old suspicious lady will look at our packets… ☹️

The FORWARD chain on the other hand has a lot of rules but OUTPUT has none. Then the custom DOCKER one has only 2.

ℹ️ Remember that the Filter table is displayed by default because it's the most commonly used. However, you can list other tables if you wished using "-t <table_name>".

Order of evaluation of rules

That's a simple one! All incoming packets are evaluated by the rules from top to bottom.

⬆️For example, using the FORWARD chain

In the example above, we can look at the FORWARD chain that has 14 rules. Each packet will undergo evaluation against rules 1 through 14 unless it matches one of these rules before reaching the 14th (If so it will either be ACCEPTED or REJECTED!).

The Macdonald Default Policy

Macdo's commercials are always like "Come as you are". Well… By default, iptables is the same. If the incoming packet doesn't match any rules and attains the last rule available then it will continue its journey whether it's legit or not. So, by default, iptables is useless, L.O.L.

But, it's good news because we don't want to mess things up with a wrong rule. It's only when we feel we are ready that we'll change the default accept to a default deny. When this happens, all packets that were not explicitly accepted will be dropped (killed ☠️).

You can look for the current Policy displayed in the previous screenshot. Look at the first line and you'll see: chain INPUT (policy ACCEPT... The last step of this tutorial will be to change this to (policy DENY....

Anatomy of a rule

Let's take an incoming packet with the following attributes:

Protocole: TCP
Target port: 22

We receive a TCP packet on our VM. It's on our port 22. We can search what program listens on our port 22, if any, with the command:

$ netstat -tupln |grep 22
...
openssh * * 22 7

This shows that a process named OpenSSH listens on our port 22 with PID 7.

It's logical as port 22 is commonly used by SSH! Let's accept this packet explicitly and allow OpenSSH to deal with it.

To allow the packet we can use the iptables command line as follows:

iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT

💁Do not execute this command yet

What does all of this mean 😨 ?

iptables: It's the binary, I hope you understand that 😁
-A INPUT: Means "append" this rule to the INPUT chain. You don't have to specify a specific table because "filter" is selected by default (convenient!).
-p tcp: Means, filter packets on their protocol type. Can be tcp, udp, icmp, or all. If the incoming packet had been an UDP packet then this rule wouldn't match it and the packet would move on to the next rule.
--dport 22: Means, filter on packets that have a destination port set to 22. If our packet was meant to be addressed to another port (80, 53, etc) then this rule wouldn't match it.
-j ACCEPT: Means, what action to take when the rule matches a packet. Can be "accept", "drop" or jumping to another chain.

Preventing failures

🎉Good news! If you inadvertently lock yourself out by blocking SSH or else, you should know that by default all iptables are restored when the VM reboots! So if you get locked out, simply reboot using the hypervisor. All clouders and VM providers have a button somewhere to hard reboot a VM. No need to destroy it!

Also, you should save the current rules you have just in case :

iptables-save >> iptables_save.log

Adding a custom chain

We want to avoid putting our custom K8S rules in the INPUT chain. Instead, we'll create a new chain that will be dedicated to Kubernetes stuff and will be called from inside the INPUT chain.

Let's create a new chain in the filter table called KUBE-PROTECTION

iptables -N KUBE-PROTECTION

Now if we list our chains again

iptables -xvnL

⬆️Our new custom chain has been created

We can see the KUBE-PROTECTION chain but it's empty.

Adding rules to a custom chain

For example, let's say we want an "allow SSH" rule to be stored in the KUBE-PROTECTION chain. We would use the command like this:

iptables -A KUBE-PROTECTION -p tcp -m tcp --dport 22 -j ACCEPT

It's simply "-A "

Listing our rules:

iptables -xvnL

⬆️A rule has been added to our custom chain

Yahoo! The rule is correctly stored in the KUBE-PROTECTION chain.

It's fairly easy to read:

pkts: 0 => Is the number of packets that have matched this rule. For now, no packets have matched.
target: ACCEPT => Means that if the packet matches then it's accepted and can directly go through to the next iptable's table
prot: TCP => The protocol the packet must match to be evaluated by this rule
source: 0.0.0.0/0 => Means that the packet must be sent from a specific IP address so the rule can match it. "0.0.0.0/0" means all source IPs can match.
"dpt:22" => Means that the packet must be sent to our port 22 to match this rule.

Calling our custom chain

Having a custom chain to store custom rules is great. But for now, our incoming packet will not magically be redirected to our custom chain. As I mentioned earlier chains act like functions in programming. So we must call our custom chain from the INPUT chain (which, remember, is always the first chain to be called in the filter table)

iptables -A INPUT -j KUBE-PROTECTION

This might look a bit weird.

The "-A INPUT" means: Add a rule to the INPUT chain.
The "-J KUBE-PROTECTION" means: Let's take the action of jumping to a chain called KUBE-PROTECTION. This might feel strange because previously we saw that -j was meant to ACCEPT or REJECT a packet. Fortunately, this is the last action that can be specified with -j , there is no fourth option.

Now if we take a look at our filter table:

iptables -xvnL

⬆️We added an action rule in INPUT to jump to KUBE-PROTECTION

We now have 1 rule in our INPUT chain. Its "target" is "KUBE-PROTECTION". So any packet that reaches this line will jump to the specified chain.

We can also see that the number of matched packets ("pkts" circled in red) in our custom rule is now rising. As I'm using SSH, all my packets on port 22 are now matching this rule and being accepted!

In programming, functions automatically return to their caller when finished. It's the same here.

When all rules inside our KUBE-PROTECTION chain have been executed and the packet hasn't matched any rules then the execution goes back to the INPUT chain to the next rule.

Note that it could also reach another jump rule and continue to yet another custom chain…

Removing a rule

For many reasons, you may need to remove a rule. The simplest way is to list the rules with their number and remove the rule you want using the specific ID.

iptables -xvnL --line-numbers

Let's remove the "allow SSH" rule from the KUBE-PROTECTION chain. It's rule number 1.

iptables -D KUBE-PROTECTION 1

It's pretty simple: it's the chain name followed by the rule number to remove.

A few more useful iptables options

Before we dive into the specific Kubernetes use case let's see one last set of options:

iptables -A KUBE-PROTECTION
         -s 10.11.12.13,10.21.22.23
         -i eth0
         -p TCP
         -m multiport --dport 2379:2380
         -m comment --comment "This is a comment"
         -j ACCEPT

A few new selectors here:

-s <IP,IP,...>: Means, match only packets coming from one of these source IPs. If the packet is not sent from one of these IPs then the rule doesn't match and the packet carries on.
-i <network_interface>: Means, match only packets that are received on this specific network interface. Depending on your OS and version you will have one typical default interface name. Either "eth0", "ens5" or "enp". Newer systems tend to use "ens5" and older "eth0". Note that your VM will probably have other network interfaces but we won't care about them. Other processes deal with them.

To find your main interface you can use :

ip -a
# or
ifconfig

These will list your network interfaces and you should find a familiar name lost in there (eth, ens, enp)…

-m multiport: The "-m" is tricky because it can be used multiple times in one command to activate further options which is kind of dumb. In this case, it makes the next option possible.
--dport <portA:portB>: Means allow all ports in the range A to B. Port ranges are available due to the previous use of the "-m multiport" option.
-m comment: Means, active the comment option… I think that you see why this whole "-m" stuff is silly. Why not just allow comments on all rules… Brrr. Though I won't complain to the dead guys who developed this in 1999. At least it works!
--comment "<Some comment>" : Self-explanatory. You should comment on all your rules to describe what protocol or software they cover. For instance, for the "allow 22" rule we should rewrite the rule with

-m comment --comment "Allow SSH connectivity"

III. Building our Kubernetes protection rules

Strict minimum rules to have in the INPUT chain

We'll add all the Kubernetes stuff in a private chain and call it from the INPUT chain. However, you should check that your INPUT chain has the following rules evaluated before jumping to your custom chain:

iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
iptables -A INPUT -i eth0 -j ACCEPT -p icmp
iptables -A INPUT -j KUBE-PROTECTION

Before applying these rules check if you don't already have them using iptables -xvnL.

Line 1: (BORING STUFF, CONSIDERING MOVING ALONG) The -m contrack module allows us to use the following option --ctstate. It has two values: "related" and "established". The established field allows currently opened connections to stay alive. The related field on the other hand is used when a server returns an answer (to a request) on a different port than the one it was received on. For instance, you could have an FTP server that receives a packet on port 21 and answers on port 1234 (strange… I know!). If you don't allow related connections then the server won't be able to answer! This is quite frequent so this rule is a must-have.
Line 2: Accept packets on the loopback interface which basically means that you can talk to yourself. If you're a human then maybe consider a psychotherapist but for machines, it's all good!
Line 3: Accept SSH on port 22. Don't lose this one…
Line 4: Accept packets of type ICMP. ICMP is the protocol used by ping to allow anyone to ping your VM. You should remove it if you're directly connected to the internet. But if your VM is part of a private network then it's better to allow ping. It's good to know if the VM is alive or dead and ping does an amazing job of that.
Line 5: If a packet has not been accepted or rejected yet and attains this rule it jumps onto our custom chain KUBE-PROTECTION.

What are the known legit Kubernetes packets to allow?

Kubernetes provides information about ports on VMs that must be opened to allow it to work properly: here.

⬆️Needed ports for Kubernetes to work taken from the documentation

Based on the above screenshot, the allowed ports to listen on our 3 types of VMs depend on their role:

Master nodes:

6443 TCP => Main entry point for API-server
10250 TCP => Kubelet
8472 UDP (Canal/Flannel VXLAN overlay networking)
30000:32767 => Node ports services

Worker nodes:

10250 TCP => Kubelet
30000:32767 => Node ports services
53 TCP/UDP => CoreDNS

ETCDs:

2379:2380 TCP =>Main entry point for ETCD communication

Methodology to authorize Kubernetes-related requests and not break everything

All that is not Kubernetes-related network must be investigated!

I'm 99% sure that if you only open the above ports and switch to "default deny" then things will break…

That's why we must monitor incoming packets and understand what is sending them to us and why! For instance, your VMs could have monitoring software checking if the VM is healthy, and cutting off all accesses to these legitimate external services will go BOOM…

So we need a way to log these packets and decide if we write a rule to match them or not. If we don't explicitly reject them they will be rejected by the "default deny" when we activate it.

We can achieve this by adding a logging rule. Each packet that goes through this rule will be logged with all its details :

iptables -A KUBE-PROTECTION -j LOG --log-prefix "iptables: " --log-level 6

This rule will be added at the end of our chain and then log packets that none of our rules matched.

⬆️A log rule has been added at the end of our chain; Also note that I have added to INPUT the rules mentioned earlier

Now let's look at the logs:

$ journalctl -f |grep iptables
Jan 29 23:02:06 vps651163 kernel: iptables: IN=eth0 OUT= MAC=fa:17:3e:a6:79:bc:e6:5b:1f:4e:18:32:08:00 SRC=143.188.50.238 DST=192.65.42.44 LEN=52 TOS=0x00 PREC=0x00 TTL=42 ID=53443 DF PROTO=TCP SPT=35163 DPT=22 WINDOW=1191 RES=0x00 ACK URGP=0
Jan 29 23:02:07 vps651163 kernel: iptables: IN=eth0 OUT= MAC=fa:17:3e:a6:79:bc:e6:5b:1f:4e:18:32:08:00 SRC=137.114.37.1 DST=192.65.42.44 LEN=32 TOS=0x08 PREC=0x00 TTL=5 ID=1 DF PROTO=ICMP TYPE=8 CODE=0 ID=7479 SEQ=1
Jan 29 23:02:07 vps651163 kernel: iptables: IN=eth0 OUT= MAC=fa:17:3e:a6:79:bc:e6:5b:1f:4e:18:32:08:00 SRC=82.242.156.152 DST=192.65.42.44 LEN=84 TOS=0x00 PREC=0x00 TTL=243 ID=24788 DF PROTO=TCP SPT=49834 DPT=1234 WINDOW=1026 RES=0x00 ACK PSH URGP=0
Jan 29 23:02:07 vps651163 kernel: iptables: IN=eth0 OUT= MAC=fa:17:3e:a6:79:bc:e6:5b:1f:4e:18:32:08:00 SRC=76.242.156.152 DST=192.65.42.44 LEN=84 TOS=0x00 PREC=0x00 TTL=243 ID=24789 DF PROTO=TCP SPT=49834 DPT=6443 WINDOW=1026 RES=0x00 ACK PSH URGP=0
Jan 29 23:02:07 vps651163 kernel: iptables: IN=eth0 OUT= MAC=fa:17:3e:a6:79:bc:e6:5b:1f:4e:18:32:08:00 SRC=30.242.156.152 DST=192.65.42.44 LEN=84 TOS=0x00 PREC=0x00 TTL=243 ID=24790 DF PROTO=TCP SPT=49834 DPT=21 WINDOW=1026 RES=0x00 ACK PSH URGP=0
Jan 29 23:02:07 vps651163 kernel: iptables: IN=eth0 OUT= MAC=fa:17:3e:a6:79:bc:e6:5b:1f:4e:18:32:08:00 SRC=27.242.156.152 DST=192.65.42.44 LEN=84 TOS=0x00 PREC=0x00 TTL=243 ID=24791 DF PROTO=TCP SPT=49834 DPT=22 WINDOW=1026 RES=0x00 ACK PSH URGP=0

(scroll to the right)
There you'll see all the packets that didn't match any of our rules. If we had activated the "default deny" they all would have been dropped (killed 😵).

Anatomy of a packet log

Taking one random line from above and removing useless information for convenience:

IN=eth0 SRC=82.242.156.152 DST=192.65.42.44 PROTO=TCP SPT=49834 DPT=1234

What you should extract from each log line is:

SRC: That's the IP who contacted you.
DST: It's… You. So we don't care. Just being clear.
PROTO: The protocol used by the incoming packet: TCP/UDP/ICMP
DPT: The DestinationPort on your VM where the packet wants to arrive. For instance, if "DPT=22" then it's probably a packet for the SSH service.

What to do with these packet logs?

Don't ask me?? It's not like I can explicitly tell you if the TCP packet on port 1234 from 82.242.156.152 is legit!

BUT

I can tell you how you should proceed next…

For each log line check if a program is listening on your VM on this specific port. Looking for _1234 _port :

netstat -tupln | grep 1234
tcp 0 0 0.0.0.0:1234 0.0.0.0:*  LISTEN  14182/random-software

There you go. A program called "random-software" (pid 14182) is listening on this port. Is this legit? You should check with your boss but if something is already listening on the VM you should probably ACCEPT the packet.

If it's legit then add a new ACCEPT rule like this:

iptables -A KUBE-PROTECTION -p tcp -m tcp --dport 1234 -j ACCEPT -m comment --comment "random-software"

On the contrary, if no processes are listening on this port then you really should investigate the IP address from which it's coming. It will help you determine why it's contacting you again and again. If it doesn't seem legit then do nothing! When we go live and activate the "default deny" it will be blocked!

Allowing only certain sources

This part is for more advanced users, consider skipping ahead for now! The "-s" option of iptables allows packets to match rules only if they come from specific IP sources:

iptables ... -s 10.1.2.3,10.4.5.6,...

This is useful to restrict access to the Kubernetes-related ports to only the Kubernetes VMs themselves.

It's a bit tricky if you want to do some fine-grained work. But you can at least rewrite the policies using the IPs of all your masters, worker nodes, and ETCDs specifying all their IP addresses. This way only the cluster components can talk to each other and random external IPs won't.

However, this will need some level of automation on your side. Because when the cluster expands or shrinks (node-wise) then you must update all the iptables "-s" list with the new set of authorized IPs.

Ansible could help you resync rules for instance. 😉

A little upstart

To get started, here is the minimum list of rules based on the Kubernetes documentation and experimentations:

Master nodes:

iptables -A KUBE-PROTECTION -i eth0 -p tcp --dport 6443 -j ACCEPT -m comment --comment "API-server"
iptables -A KUBE-PROTECTION -i eth0 -p tcp --dport 10250 -j ACCEPT -m comment --comment "Kubelet"

iptables -A KUBE-PROTECTION -i eth0 -p tcp -m multiport --dport 30000:32767 -j ACCEPT -m comment --comment "NodePort"
iptables -A KUBE-PROTECTION -i eth0 -p tcp --dport 9200 -j ACCEPT -m comment --comment "Node Exporter"
# Common network virt protocol for pod to pod communication 
iptables -A KUBE-PROTECTION -i eth0 -p udp --dport 8472 -j ACCEPT -m comment --comment "VXLAN overlay"

# If using cilium
iptables -A KUBE-PROTECTION -i eth0 -p tcp -m multiport --dport 4240 -j ACCEPT -m comment --comment "Cilium"
# If using Flannel
iptables -A KUBE-PROTECTION -i eth0 -p tcp -m multiport --dport 8285 -j ACCEPT -m comment --comment "Flannel" 

# Logging packets that did not match any previous rules
iptables -A KUBE-PROTECTION -j LOG --log-prefix "iptables: " --log-level 6

Worker nodes:

iptables -A KUBE-PROTECTION -i eth0 -p tcp --dport 10250 -j ACCEPT -m comment --comment "Kubelet"
iptables -A KUBE-PROTECTION -i eth0 -p tcp -m multiport --dport 30000:32767 -j ACCEPT -m comment --comment "NodePort"
iptables -A KUBE-PROTECTION -i eth0 -p tcp --dport 53 -j ACCEPT -m comment --comment "CoreDNS"
iptables -A KUBE-PROTECTION -i eth0 -p udp --dport 53 -j ACCEPT -m comment --comment "CoreDNS"

# Logging packets that did not match any previous rules
iptables -A KUBE-PROTECTION -j LOG --log-prefix "iptables: " --log-level 6

ETCDs:

iptables -A KUBE-PROTECTION -i eth0 -p tcp -m multiport --dport 2379:2380 -j ACCEPT -m comment --comment "ETCD"

# Logging packets that did not match any previous rules
iptables -A KUBE-PROTECTION -j LOG --log-prefix "iptables: " --log-level 6

⚠️Remember to update "eth0" with your public interface.

Also, you can have fun with the "-s <IPs>" if you wish.

Finally, you must monitor the remaining packets and decide what to do with them. Accept? Reject? Do nothing (same as reject when we activate default deny)?

Changing the default ACCEPT to default DENY

It's time!

You have allowed all Kubernetes ports and other proprietary stuff that runs on the VMs. The rest must die!

Time to reject all that doesn't match our rules by activating the great denier !! (kinda scary)

iptables -P INPUT DROP

This means "When a packet reaches the end of the INPUT chain without having matched any rules then DROP it. It dies… Astalavista el packet ! Adios!

The default should shift from:

$ iptables -xvnL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
    pkts bytes target prot opt in out  source destination

(See "policy ACCEPT" ?)
To:

$ iptables -xvnL
Chain INPUT (policy DROP 0 packets, 0 bytes)
   pkts bytes target prot opt in out  source destination

(See "policy DROP" ?)

ℹ️ Remember that if all goes to sh*t you can just reboot the VM. You could also revert the default policy with iptables -P INPUT ACCEPT if needed.

I also recommend that you do this on one VM at a time. Start with a worker node. If all goes well then do all the workers. Then one ETCD, all ETCD, and finally one master, all masters. If the cluster is still alive then GG!

If not, then go back to your VMs and log the dropped packets with the logging command (cf above) to find what you should have left open!

IV. Keeping our iptables configuration after reboots

It's time to persist our configuration. Now this will mostly depend on your OS but I have a few suggestions.

Using a Linux package

sudo apt-get update
sudo apt-get install iptables-persistent

During the installation, you will be prompted to save the current IPv4 and IPv6 rules. Choose "Yes" to save the rules.

Usage:

The iptables-persistent package provides two commands: iptables-save and iptables-restore.

To save the current iptables rules:

sudo iptables-save > /etc/iptables/rules.v4
sudo ip6tables-save > /etc/iptables/rules.v6

To restore iptables rules at startup:

sudo systemctl enable netfilter-persistent

Now, on each system startup, the netfilter-persistent service will read the rules from /etc/iptables/rules.v4 for IPv4 and /etc/iptables/rules.v6 for IPv6 and apply them.

Alternatively, you can manually restore the rules:

sudo iptables-restore < /etc/iptables/rules.v4
sudo ip6tables-restore < /etc/iptables/rules.v6

Keep in mind that the exact procedure might vary slightly based on your Linux distribution, especially for non-Debian-based systems.

When the main interface goes UP

We have bound our rules to the public interface of the VM using "-i eth0" (or "ens5", etc). We can tell Linux to bring up our rules before the relevant interface is activated. This is, in my opinion, the safest way to go as there won't be any milliseconds where the VM is not firewalled during a reboot. Sometimes a millisecond is all it takes to get hacked.

Create a shell script /etc/firewall-start

#!/bin/bash
<Add your iptables rules here>

If you added all your custom rules in a custom chain then your script should perform the following iptables tasks:

A) Flush the chain (see iptables --flush)

B) Repopulate the content of your custom chain using the iptables rules mentioned earlier.

C) Check the existence of the rule that calls your custom chain from INPUT using the iptables -c option. If this returns true then your script doesn't need to add the jump rule as it already exists. If it returns false then add the jump instruction to the INPUT chain.

Edit /etc/network/interfaces

auto eth0
iface eth0 inet static
pre-up /etc/firewall-start # <= Add this line

This file will be read during startup and execute the script /etc/firewall-start which will populate your rules. Then the associated interface (_eth0 _here) will be brought up.

V. Conclusion

You made it alive. GG! 🎉

This may not be the most robust config that exists in the industry but you now have a working firewall on your cluster and that's a start! In the future, you may look at dealing with firewalls using Cilium and all the EBPF stuff that makes no sense. It will be great one day. Just not today. Today was iptables day! ^^

Installing PrivateGPT on WSL with GPU support

Emilien Lancelot — Sat, 20 Jan 2024 23:29:33 +0000

[ UPDATED 23/03/2024 ]

PrivateGPT is a production-ready AI project that allows you to ask questions about your documents using the power of Large Language Models (LLMs), even in scenarios without an Internet connection. 100% private, no data leaves your execution environment at any point.

Running it on Windows Subsystem for Linux (WSL) with GPU support can significantly enhance its performance. In this guide, I will walk you through the step-by-step process of installing PrivateGPT on WSL with GPU acceleration.

Installing this was a pain in the a** and took me 2 days to get it to work. Hope this can help you on your own journey… Good luck !

Prerequisites

Before we begin, make sure you have the latest version of Ubuntu WSL installed. You can choose from versions such as Ubuntu-22–04–3 LTS or Ubuntu-22–04–6 LTS available on the Windows Store.

Updating Ubuntu
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install build-essential

ℹ️ “upgrade” is very important as python stuff will explode later if you don’t

Cloning the PrivateGPT repo

git clone https://github.com/imartinez/privateGPT

Setting Up Python Environment

To manage Python versions, we’ll use pyenv. Follow the commands below to install it and set up the Python environment:

sudo apt-get install git gcc make openssl libssl-dev libbz2-dev libreadline-dev libsqlite3-dev zlib1g-dev libncursesw5-dev libgdbm-dev libc6-dev zlib1g-dev libsqlite3-dev tk-dev libssl-dev openssl libffi-dev
curl https://pyenv.run | bash
export PATH="/home/$(whoami)/.pyenv/bin:$PATH"

Add the following lines to your .bashrc file:

export PYENV_ROOT="$HOME/.pyenv"
[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

Reload your terminal

source ~/.bashrc

Install important missing pyenv stuff

sudo apt-get install lzma
sudo apt-get install liblzma-dev

Install Python 3.11 and set it as the global version:

pyenv install 3.11
pyenv global 3.11
pip install pip --upgrade
pyenv local 3.11

Poetry Installation

Install poetry to manage dependencies:

curl -sSL https://install.python-poetry.org | python3 -

Add the following line to your .bashrc:

export PATH="/home/<YOU USERNAME>/.local/bin:$PATH"

ℹ️ Replace by your WSL username ($ whoami)

Reload your configuration

source ~/.bashrc
poetry --version # should display something without errors

Installing PrivateGPT Dependencies

Navigate to the PrivateGPT directory and install dependencies:

cd privateGPT
poetry install --extras "ui embeddings-huggingface llms-llama-cpp vector-stores-qdrant"

In need of a free and open-source Multi-Agents framework built for running local LLMs?

▶️ Checkout Yacana

Nvidia Drivers Installation

Visit Nvidia’s official website to download and install the Nvidia drivers for WSL. Choose Linux > x86_64 > WSL-Ubuntu > 2.0 > deb (network)

Follow the instructions provided on the page.

Add the following lines to your .bashrc:

export PATH="/usr/local/cuda-12.4/bin:$PATH"
export LD_LIBRARY_PATH="/usr/local/cuda-12.4/lib64:$LD_LIBRARY_PATH"

ℹ️ Maybe check the content of “/usr/local” to be sure that you do have the “cuda-12.4” folder. Yours might have a different version.

Reload your configuration and check that all is working as expected

source ~/.bashrc
nvcc --version
nvidia-smi.exe

ℹ️ “nvidia-smi” isn’t available on WSL so just verify that the .exe one detects your hardware. Both commands should displayed gibberish but no apparent errors.

Building and Running PrivateGPT

Finally, install LLAMA CUDA libraries and Python bindings:

CMAKE_ARGS='-DLLAMA_CUBLAS=on' poetry run pip install --force-reinstall --no-cache-dir llama-cpp-python

Let private GPT download a local LLM for you (mixtral by default):

poetry run python scripts/setup

To run PrivateGPT, use the following command:

make run

This will initialize and boot PrivateGPT with GPU support on your WSL environment.

ℹ️ You should see “blas = 1” if GPU offload is working.

...............................................................................................
llama_new_context_with_model: n_ctx      = 3900
llama_new_context_with_model: freq_base  = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:      CUDA0 KV buffer size =   487.50 MiB
llama_new_context_with_model: KV self size  =  487.50 MiB, K (f16):  243.75 MiB, V (f16):  243.75 MiB
llama_new_context_with_model: graph splits (measure): 3
llama_new_context_with_model:      CUDA0 compute buffer size =   275.37 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =    15.62 MiB
AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 |
18:50:50.097 [INFO    ] private_gpt.components.embedding.embedding_component - Initializing the embedding model in mode=local

ℹ️ Go to 127.0.0.1:8001 in your browser

Uploaded the Orca paper and asking random stuff about it.

Conclusion

By following these steps, you have successfully installed PrivateGPT on WSL with GPU support. Enjoy the enhanced capabilities of PrivateGPT for your natural language processing tasks.

If something went wrong then open your window and throw your computer away. Then start again at step 1.

You can also remove the WSL with:

wsl.exe --list -v
wsl --unregister <name of the wsl to  remove>

If this article help you in any way consider giving it a like ! Thx

Troubleshooting

Having a crash when asking a question or doing make run ? Here are the issues I encountered and how I fixed them.

Cuda error

CUDA error: the provided PTX was compiled with an unsupported toolchain.
  current device: 0, in function ggml_cuda_op_flatten at /tmp/pip-install-3kkz0k8s/llama-cpp-python_a300768bdb3b475da1d2874192f22721/vendor/llama.cpp/ggml-cuda.cu:9119
  cudaGetLastError()
GGML_ASSERT: /tmp/pip-install-3kkz0k8s/llama-cpp-python_a300768bdb3b475da1d2874192f22721/vendor/llama.cpp/ggml-cuda.cu:271: !"CUDA error"
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
        - Avoid using `tokenizers` before the fork if possible
        - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
make: *** [Makefile:36: run] Aborted

This one comes from downloading the latest CUDA stuff and your drivers are not up to date. So open the Nvidia "Geforce experience" app from Windows and upgrade to the latest version and then reboot.

CPU only

If privateGPT still sets BLAS to 0 and runs on CPU only, try to close all WSL2 instances. Then reopen one and try again.

If it's still on CPU only then try rebooting your computer. This is not a joke… Unfortunatly.

A note on using LM Studio as backend

I tried to use the server of LMStudio as fake OpenAI backend. It does work but not very well. Need to do more tests on that and I’ll update here.

For now what I did is start the LMStudio server on the port 8002 and unchecked “Apply Prompt Formatting”.

On PrivateGPT I edited “settings-vllm.yaml” and updated “openai > api_base” to “http://localhost:8002/v1" and the model to “dolphin-2.7-mixtral-8x7b.Q5_K_M.gguf” which is the one I use in LMStudio. It’s displayed in LMStudio if your wondering.

Other authors you might like

▶️ Checkout Yacana

Goodbye Sealed Secrets, hello SOPS

Emilien Lancelot — Sat, 30 Jul 2022 20:12:00 +0000

This tutorial will provide you with all the steps and commands to setup SOPS in your shell, Kubernetes, Helm and Visual Studio Code.

We assume that you are on Windows and using WSL as Linux environment. We will setup Sops in both.

Want to dive right in ? Click here to jump directly to the installation section.

What is SOPS

Sops is a binary able to encrypt configuration files. But rather than encrypting the whole file Sops understands format (JSON, YAML, INI, etc) and will only encrypt the values of each line (in a key/value pair).

The aim is to store encrypted configurations containing sensitive information into your versionning system (Git, SVN, etc) and then be able to work with these encrypted files effortlessly.

Sops provide the following advantages :

Encrypting values but not keys
- Preserves file structure even when encrypted
Safely saving encrypted files to Git
Editing encrypted files seemlessly
- Sops will open the unencrypted version of the file in your favorite text editor
- When saving, the file will be automatically reencrypted with its new content
Encrypting specific keys and not the whole file
- Perfect for long configuration files that have few pieces of sensitive information inside.
Working with all configuration files and not only Kubernetes secrets
- Can encrypt your Helm’s values.yaml files to safely store sensitive pieces of data inside it (passwords, etc)
No backend in Kubernetes needed
- The only requirement is to install Sops locally
- Managing encrypted files will be done locally
Plugins are available for all major tools to support working with Sops

Sops encryption example

Let’s say we have this classic Kubernetes secret. The secret value is not encrypted but only encoded to base64.



apiVersion: v1
data:
  secret_value: U29wcyBydWxlcw== 📖
kind: Secret
metadata:
  creationTimestamp: null
  name: my-secret
  namespace: default

After being given to Sops this is what you get:



apiVersion: ENC[AES256_GCM,data:PFk=,iv:,[...]tag:G9mg==,type:str]🔒
data:
    secret_value: ENC[AES256_GCM,data:Ex2KhoAlCG4w==,[...]🔒
kind: ENC[AES256_GCM,data:eW1JrnrV,[...]🔒
metadata:
    creationTimestamp: null
    name: ENC[AES256_GCM,data:fkWXVtQS/C62,iv:GCTfamtu4ixP4w=[...]🔒
    namespace: ENC[AES256_GCM,data:J96kuhjMUA==,iv:QZMnvDGqX[...]🔒
sops:
[...]

As you can see, all the values have been encrypted. But the keys are still in plain text.

Sops can also encrypt specific values based on keys that you give it.
Here, let’s say we want to ONLY encrypt the “secret_value” key :



apiVersion: v1📖
data:
    secret_value: ENC[AES256_GCM,data:9VBm+32QSCOaU+9exohSg==[...]🔒
kind: Secret📖
metadata:
    creationTimestamp: null
    name: my-secret📖
    namespace: default📖
sops:
[...]

Only the “secret_value” value has been encrypted. The rest of the file remains untouched.

Why we stopped using Sealed Secrets

What does it do ?

We have been using SealedSecrets for a year but it’s time for a change. This (cool) technology allows you to encrypt a Kubernetes secret on disk and have it commited on GIT. This way, even if a hacker gets its hands on your repository, he won’t have access to the content of your secrets.

Why it’s not perfect ?

SealedSecret is great for a lot of usecase. But in our opinion it does come short here and there:

Must install a backend in the cluster
Certificate rotation might get complicated
Default encryption mode doesn’t allow to move secret from one namespace to another
On the Kubernetes cluster the generated secret is still in plain text But the most problematic part is…

We have sensitive data stored in plain text inside helm charts, for instance passwords inside the “values.yaml” of helm charts. This is not covered at all by sealed secrets.

Installation of Sops

Install the Sops binary on WSL

Download the binary for linux : https://github.com/mozilla/sops/releases



$ wget <URL_OF_THE_BIN_FROM_GITHUB>

$ sudo mv sops-<VERSION> /usr/local/bin/sops

Install Age

Sops doesn’t do the encryption itself. It leverages other encryption technologies like PGP, AGE, GCP KMS, Azure key vault, Hashicorp Vault, etc

As PGP is kind of deprecated, we’ll install the new kid on the block : Age

Age is a tool to encrypt files. It doesn’t care about the format of the file itself. To encrypt only the values or parts of a file we need to rely on Sops. Age is only a tool that Sops will use.

Let’s install Age…



$ AGE_VERSION=$(curl -s "https://api.github.com/repos/FiloSottile/age/releases/latest" | grep -Po '"tag_name": "(v.*)"' |grep -Po '[0-9].*[0-9]')

$ curl -Lo age.tar.gz "https://github.com/FiloSottile/age/releases/latest/download/age-v${AGE_VERSION}-linux-amd64.tar.gz"

$ tar xf age.tar.gz

$ sudo mv age/age /usr/local/bin

$ sudo mv age/age-keygen /usr/local/bin

Generate public and private keys

Now that Age is installed, you must create a public and a private key.



$ age-keygen -o key.txt

$ cat key.txt
# created: 2022-07-30T17:02:43+02:00
# public key: age1rdje4gwnm2cc6uu6lvggzjj8gktu4cw5ng8yyjhrluwvqq387cls58dsy4
AGE-SECRET-KEY-18LMDYJRYZMRM52CYQA9MZW79M85CPR0HJHCASXF5ADT03KJ290HS88599E

$ mkdir ~/.sops

$ mv ./key.txt ~/.sops

▶️The private and public keys created by Age are stored inside the file key.txt. We moved this file inside a .sops directory inside the home folder.

To tell Sops where everything is, add these ENV variables to your shell configuration (~/.bashrc or ~/.zshrc)



export SOPS_AGE_KEY_FILE=$HOME/.sops/key.txt
export EDITOR=nano # Replace by your favorite editor

ℹ️ Dont forget to source your shell profile before continuing or at least open a new shell.

Testing Sops and Age are working

Let’s encrypt a file…



# Creating a test file
$ echo "my-key: my-value" > config.yaml

# Encrypting the file
sops --encrypt --age $(cat ~/.sops/key.txt |grep -oP "public key: \K(.*)") ./config.yaml

▶️ we used the --age parameter to specify what public key to use. Here, we rely on bash to replace during runtime the public key present in key.txt.

Now to save the file to the disk a simple right redirection will do the trick :



$ sops --encrypt --age $(cat ~/.sops/key.txt |grep -oP "public key: \K(.*)") ./config.yaml > config_enc.yaml

$ cat config_enc.yaml
my-key: ENC[AES256_GCM,data:Jw1oRQcV8=,tag:Tzd/oUQ[...]==,type:str]
sops:
[...]

Edit the file using Sops



$ sops config_enc.yaml
<Your favorite editor should have opened>
<The file in front of you should be unencrypted>
<When saving, modifications you made will be encrypted>

On the fly decryption and re-encryption is a fabulous feature. You don’t even know the file is encrypted. It’s all automatic !

Configure the tools

Add aliases to your interpretor

This should work effortlessly on bash and zsh.

As the syntax to encrypt a file is bit tedious (due to the fact that you must pass the Age public key to Sops) we have prepared an alias that can be added to your .bashrc or .zshrc



function cypher {
    filename=$(basename -- "$1")
    extension="${filename##*.}"
    filename="${filename%.*}"
    sops --encrypt --age $(cat ~/.sops/key.txt |grep -oP "public key: \K(.*)") $2 $3 $1 > "$filename.enc.$extension"
}

Remember to always source your profile before trying new aliases.

Usage:



# Creating test files
$ echo "some_key: some_value" > test_alias.yaml

# Encrypting
$ cypher test_alias.yaml

$ ls
test_alias.enc.yaml

You could also update the alias to edit the file in place and not generate a new one. This should look like this :



function cypher_inplace {
    sops --encrypt --in-place --age $(cat ~/.sops/key.txt |grep -oP "public key: \K(.*)") $2 $3 $1
}

To encrypt only parts of a file you must explicit the keys using a regex. You won’t need to be a regex master to achieve this. Check out this example:



$ echo "key1: value1" >  test_regex.yaml
$ echo "key2: value2" >> test_regex.yaml
$ echo "key3: value3" >> test_regex.yaml

$ cat test_regex.yaml
key1: value1
key2: value2
key3: value3

$ cypher test_regex.yaml  --encrypted-regex='^(key1|key3)$'

$ cat test_regex.enc.yaml
key1: ENC[AES256_GCM,[...]tag:qHy9z22CIxcb9ykpPAWikQ==,type:str]
key2: value2
key3: ENC[AES256_GCM,data:3Faqn,[...]tag:1VB3/wrLekmg==,type:str]

As you can see, only the values for key1 and key3 have been encrypted. key2 has been left untouched.

You can add any keys you want separated by | . For instance --encrypted-regex='^(<a_first_key>|<a_second_key>|<a_third_key>)$

Installing Sops for Visual Studio Code

As we have seen, Sops allows to edit encrypted file seemlessly. Visual Studio Code can do the same, provided it has a Sops plugin.

Download the Sops binary for Windows (or whatever OS your VS code runs on)

https://github.com/mozilla/sops/releases Add the binary into your OS path
On Windows that would be c:\Windows\system32
Rename it to sops.exe On VScode, download the Sops plugin @signageos/vscode-sops.
You shouldn’t have to, and yet, to configure the plugin in VScode go to file > preferences > settings > plugins > Sops
Copy the keys that were setup on linux (WSL) to your Windows partition: ```

$ mkdir /mnt/c/Users//AppData/Roaming/sops/age -p

$ cp ~/.sops/key.txt /mnt/c/Users//AppData/Roaming/sops/age/keys.txt

▶️ The path `AppData/Roaming/sops/age` is where VSCode expects your keys to be. It can be configured, in the plugin settings if needed.

### Installing Sops for Helm

A great Sops advantage is that you can leave plain text sensitive data inside the `values.yaml` of your helm chart and simply let Sops encrypt the file.

❗ Therefore the file cannot be used with helm directly as it won’t natively be able to read encrypted files…
▶️️ This is why you should install this Helm plugin. It allows to use all the classic helm commands but when encountering an encrypted file, it will automatically decrypt it and continue the installation process.

$ helm plugin install https://github.com/jkroepke/helm-secrets --version v3.14.0

To use this plugin, you simply have to add the word `secrets` after the helm binary.
For instance when installing a helm chart with an encrypted values.yaml :

$ helm secrets install minecraft-server itzg/minecraft

Installing Sops for Kubectl
There isn’t any good solution to mix kubectl and sops for now. But piping the output of Sops to kubectl apply is quite simple.

$ sops -d .yaml | kubectl apply -f -


Thank you for following this tutorial. If it helped you setup your Sops installation please consider giving a 👏.

Written by [Emilien Lancelot](www.linkedin.com/in/emilien-lancelot-62943389)

[Stack Overflow](https://stackoverflow.com/users/5512455/doctor)

In need of Function As A Service on Kubernetes ? Say no more ! [Checkout Gitfaas now !](https://github.com/rememberSoftwares/gitfaas) Free for all, opensource for everyone.