DEV Community: Narasimha Prasanna HN

AIGr.id: Internet of Intelligence - We need your feedback!

Narasimha Prasanna HN — Fri, 25 Apr 2025 12:33:10 +0000

Link to the source code
Hackernews announcement

AIGrid represents a fundamental shift from siloed, monolithic AI to an open, decentralized, and networked AI paradigm.

Instead of building big, closed AI systems, OpenOS connects smaller, independent AI clusters into a Global AI Network.

🌐 AIGr.id is a decentralized network of interconnected AI components that coordinate to share data, perform tasks, and compose into higher-level collective intelligence.

🧩 Why AIGrid ?

Built as Global public infrastructure for AI - not owned or controlled by any single entity.
Works like Commons - shared, contributed to, and improved by a global community.
Sovereign - communities to own, control, and govern their AI without external dependence.
Open Protocol - Public set of standards that define how objects and actors interact across the network. Promoting Interoperability, coordination, transparency and extensibility.

⚙️ AIGr.id is Powered by AIOS (OpenOS.AI) — A decentralized AI Operating System for distributed AI networks.

OpenOS.AI is 100% open source and community-driven.

🚧 Current Status

⚠️ Beta Alert

This project is still in beta and not ready for production use.

Testnet is expected in First week of May 2025.

Meanwhile welcome your feedback!

Click here to know more about the upcoming activities

🔗 Links:

✨ What Can You Do with OpenOS?

Connect Kubernetes clusters into a global compute network
Deploy your AI models (like LLMs or vision models) as reusable Blocks
Deploy multiple blocks on same GPU to save resources
Define workflows using vDAGs (virtual Directed Acyclic Graphs)
Share and re-use models, data, blocks and compute infrastructure
Use Python policies to control how the network behaves
Extend your Blocks with third-party tools via init containers
Collect and use metrics to make smart decisions and observe the network

🧰 Key Features

Feature	What It Means
🌍 Global Clustering	Connect clusters into a unified network
⚙️ Smart Scheduling	Run AI tasks where resources are available
🛠️ Python Policies	Use Python scripts to control the system
🧱 Modular Blocks	Reusable building blocks for AI
🧠 Split LLMs	Run parts of a model across machines
🧪 GPU Sharing	Run multiple jobs on the same GPU
🔗 Distributed Graphs	Define workflows across blocks and clusters
📦 Plug in Tools	Bring your own frameworks, models, or services
📡 Send Tasks Easily	Submit tasks through gRPC APIs
🔍 Observe Everything	Track system performance with metrics

For the detailed breakdown of features visit this link

📌 Usecases

🌐 Internet of Intelligence
🧠 Collective Intelligence
🏛️ National Sovereign AI Grid
🏢 Enterprise Sovereign AI Grid
🧑‍🤝‍🧑 Community Federated AI Grid

🚀 Quickstart

Follow the Quickstart Guide to:

📚 Learn More

Section	Link
📄 Concept Overview	Concepts
🧭 How It Works	Architecture
🛠️ Setup Instructions	Installation Guide
🧪 Tutorials	Quickstart
🗂️ User Guides	User Flows

🎯 Our Objectives

At OpenOS, we’re building more than just a platform — we’re designing the foundation for a plural, sovereign AI future. Our mission is to create AI infrastructure that is:

Open — accessible, inspectable, and modifiable by anyone
Decentralized — not controlled by any single company, cloud, or country
Composable — built from reusable, modular components called Blocks
Governable — enforceable policies built in as first-class citizens
Interoperable — works with your own models, data, and systems
Collaborative — made by and for a global community of contributors

We want to make it easy to:

Run large AI models (like LLMs and Vision AI) across many machines
Share AI, compute and data infra across organizations
Define custom behaviors through simple Python policies
Enable new forms of AI collaboration — cross-team, cross-cloud, and cross-border
Build networks that anyone can join, contribute to, and benefit from

Upcoming activities:

📊 OpenOS vs Ray/Anyscale comparison 🗓️ ETA May, 2025
🚀 OpenOS vs NVIDIA Dynamo Inference Server 🗓️ ETA: May 2025
🧪 Benchmark results (LLMs, Vision models, and more) 🗓️ ETA May, 2025
🔐 Platform Security upgrades (RBAC, decentralized ID) 🗓️ TBA
💾 Model/Asset Security 🗓️ TBA

🤝 Join Us!

OpenOS is community-driven — anyone can contribute.

We’re looking for:

Designers and Developers
Engineers
Content Creators
Policy, governance, and ethics researchers
Builders of all kinds

Get Involved

💬 Join our Discord
📧 Email us: community@opencyberspace.org

Together, let’s build the future of open, shared, and sovereign AI.

Check out my new programming language called - Boson, written in Rust!

Narasimha Prasanna HN — Fri, 15 Jul 2022 17:47:01 +0000

I recently built and open sourced my programming language called "boson", it is An interpreted, dynamically-typed, multi-threaded, general purpose hobby programming language written in Rust.

Features:

Multiple Data Types: char, int, float, string, array, hashtable, bytes and buffer.
Airthmetic, Logical operations
Variables and Constants
Control and Looping structures
Functions and Lambda expressions
Threads and Multi-threading
Shell operator to run shell commands within the language statements
Some basic built-in functions
Iterators (psuedo iterators)
Byte code generation, serialization and loading

Link to GitHub repository

Let's build a simple asynchronous Web3 client for Ethereum blockchain in python

Narasimha Prasanna HN — Mon, 25 Apr 2022 05:48:02 +0000

Ethereum has a very well defined set of methods that can be used to interact with the blockchain from external systems (probably from our current Web 2.0 systems). Ethereum exposes these functionalities over JSON-RPC 2.0 protocol. In brief, JSON-RPC is a light-weight data interchange protocol built on top of application layer protocols like HTTP and Web Sockets. Using JSON-RPC we specify the method we want to invoke and the list of parameters to be passed, the method is then executed by the Ethereum's JSON-RPC server to which we have connected and returns the output as JSON which can be consumed by the caller. When compared to gRPC and other known RPC protocols JSON-RPC is relatively simple and uses plain text JSON which is more readable and easily understandable. We can see the list of methods provided by Ethereum blockchain via JSON-RPC protocol here.

Here is a sample JSON-RPC payload to get the information of a block on the ethereum blockchain by it's number:

{
    "jsonrpc": "2.0",
    "method": "eth_getBlockByNumber",
    "params": ["0xa0f46a", true],
    "id": 1
}

We are calling the method eth_getBlockByNumber by passing the block number 10548330 (or 0xa0f46a in hex) and the boolean value which specifies whether to output full transaction output or just the transaction hashes. Similar to eth_getBlockByNumber there are many methods with their own parameter requirements.

We can make HTTP POST request to the JSON-RPC node by submitting the above JSON payload. Here is an example of how it can be done with python:

import requests
import time

data = {
    "jsonrpc": "2.0",
    "method": "eth_getBlockByNumber",
    "params": ["0xa0f46a", False],
    "id": 1             # id is just a number for keeping track of the request on client-side
}

st = time.time()
response = requests.post("https://rinkeby.infura.io/v3/22b23b601d364f999c0a7cf6deb7bad4", json=data)
et = time.time()

print('time taken: ', et - st, 'seconds')
print('output', response.json())

Output:

time taken:  0.8727474212646484 seconds
output {'jsonrpc': '2.0', 'id': 1, 'result': {'baseFeePerGas': '0xb', 'difficulty': '0x2', 'extraData': '0xd683010a11846765746886676f312e3138856c696e75780000000000000000005484b299653ddb09be6dfcaf019b2d387564cd4eedc3c7a670837fca65feaab251a0a95cb1e14fbb6b7e00680d0318eb03522b7ae5ba0d6623991439b8a257a001', 'gasLimit': '0x1c9c380', 'gasUsed': '0x5c35eb', 'hash': '0x7b5bc82fae1cb1a695e4bea1da405b8b3e953d30ce63167571bb0eba538ec028', 'logsBloom': '0x0828404000000b0000044801a110a92050208002624440000c03384090200008805042185008200410000c0000001a1231408804080cbc0004000a1011363040360000000080206a41a000080022602108060204946410228604805486100000022041002200c003000412101800481080c0002000000601840100d1000802001000a30c06c100100248b00000008040c50000010240010880000044406030380200000008020408488440000580200a3504006408c000188808214e0000120048200d0228044001000200810020404408000680810800140b00110a0480e20122343c0aa290400020ac000081050891001963b0482223400040580980002403', 'miner': '0x0000000000000000000000000000000000000000', 'mixHash': '0x0000000000000000000000000000000000000000000000000000000000000000', 'nonce': '0x0000000000000000', 'number': '0xa0f46a', 'parentHash': '0xc70e1a85c2d7e865ffe71ae1e8c78539ad2d9268b469abfa06554c2c8ac71205', 'receiptsRoot': '0xdec4ee3b890662d88f6635d5ffa05c0aba25731701224735192d076d6fad019c', 'sha3Uncles': '0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347', 'size': '0x61c5', 'stateRoot': '0x9af1e2dee823147a30eff0e66c4498addbc90550aacbd65f184b1fcf13a3a162', 'timestamp': '0x62628dd8', 'totalDifficulty': '0x10a83f5', 'transactions': ['0x8fc61b71557e445c832d0c7ac0d9a19dac13b4cf92a40dd59fa27b1faf3a5781', '0x1598f0db451fd6503bcba735dbf91b785a7aeee4fba37cc83d75508ce242c354', '0xe07bcd936979f10378af2610dc5209656662754c05a501404bd757564f9b7ca8', '0xb579c46b357a344e339bbe695cbeb7d8e80cbf3a48e359c3ac68ec937c2b41bd', '0x2020771e9c08ccae2770393c0c652d5161ae6d276bf5b2509f7e38d5d008a182', '0xca805dfe9fc1e2a13cfbf0cfc6fb3be29e12d60e29deb68140293559a4388384', '0x44b67a281d17279f0cd005b7101bfbbba3300463b5a5d3123eca32e06f0b5810', '0xd990f40743486fba6ce0d187516ad49c6cd92d7520ce80447606aea69a86c841', '0x11849bac6d3d20090d8a708cd59aaaef3412a63c4c3f1b26dacef5cc999c5038', '0xfe60a91ac75fe31a8df0acc425c931c8ecaf2692519bb365bf8501def85f5dda', '0x4f4715aa0f544b8c1a2d6229dbe683d2814081d8a39609cfcb12451d3c87bb7d', '0x97e439cb02f1e8bf4f7db89e96407e67bbbc75253bf82ebb2aa3e10304376c30', '0x129021ec232712f422089a4fa2bfb7f0c5a58e19ea1bfa8e0c37ed380942e589', '0xca429b804e925e4e28a0132a4ac42b7bd19583193cbaf5da7d8ce956cae4ef5c', '0x12fc6f6277452657d730b5518016ef84281cf5961d3ed4d1a38439a6df64dca9', '0x7e733c1e4f9ce0e0f5a3d7d455ef870588844687f980386f788c378940eaa911', '0x50c86aa5756a3d8bc0282668a9e58378b1297f0be4c7a77aa26ef38aa69698fc', '0xd49611a95ba2eed4426a3942e38d3ff043eba86e171542be04b3a3a550d318d3', '0x0a92aa7a60f1d3b97d86f9665cece2bf85248ac7d5e732547cf33ee17e387f15', '0xa91e155c129b4d8449635aee1c1cbbbde7a7f8108cc45c05aa030fc34b7b16e8', '0xf7a900e2a50400b4bbcb4f6c51256cf9125347c171b107d4a86ff7d9c852b59f', '0x090add0f1d441c97b54f40d8ec54575102f66307fa03e330d44cee6bb6a1b69c', '0xad8869d19e73ee2f57e4f0d3697a58bf902ad03cea3c1c7b7cf40c9e621ffaeb', '0x022d10b70c903220475bab28d412153a47514abed92288bfbe98224dc8e7128c', '0x367cec4698f401f166864b485d21c9aedfd26f8d28d512cd15bcd7ddb4f7e86d', '0x6ba8af557b1931d5d0e095f4ca04ccc31e9a96c03596b59264c7d894814fafef', '0x1de3719a13d7141ac9db10668b29921df6dce2d3b8ee5c6307b9d19fa9b67e8a', '0x13928fe6d5af70bff7773ecf2c11528a7abcba6b8898bdd72bb68bc6d9bed414', '0x7784245cf801dfcb7c2cbd92abcee9b78e0e4fc02b617d8a2884a9f9ecff878b', '0x6a926356ea4d1198605be6cddab626b9fc5fb3f8934bcb7f0785b42fcdba1866', '0x4046f17200f5c2a316f042d5393bd555f2d05d97615122089a34972f424e676b', '0x81a4036a5e4723d93cd058b63768545444e6b29abee1cb2d9b87b05ee2310927', '0x30a1ec3718a279aec9dc10a1ff3214a2c193e89797152f7e8ab92e6728d02580', '0xb18d695cba61e4d021505bdf001c565458bd25e0f291be34311ac03dc00334e8', '0xca4d12a4b7c51ef30be26fb7489be9c6cfed25603d41f2503b38a98e342ffbfb', '0x86ca6b8b86fc54d017b116e69f188baf3d2c60fbdbb4e4617380a436a1fce767', '0x117b48440bbbdd89edf5776df2e5fed599e4ef69e5bd1405a9cac3b8264faede'], 'transactionsRoot': '0xed1e6d5dd38173c6f6fe0ec25c056a71d3e0ca9c39d1c891b63af8993526c5f5', 'uncles': []}}

This is the most simple way to get started with making JSON-RPC calls. We directly made raw RPC calls on the blockchain, instead of that we can also use full-fledged client libraries avaibale in many languages. For example web3.py is a python implementation of high-level ethereum JSON-RPC calls.

From the above output we came to know that an average web3 call via JSON-RPC takes 900ms (this depends on many factors and is different for different RPC methods) to get data of one block. Imagine we need to query hundreds of such blocks, in that case we will be wasting lot of time (approximately 90s) in our case. To mitigate this issue, we can either go for concurrent JSON-RPC calls using asynchronous networking capabilities or follow JSON-RPC batch specification.

In the following sections we will create a simple client package that can be used to make concurrent and batch JSON-RPC calls.

Preparing for concurrent and batch calls:

In the above example we used requests module to make JSON-RPC calls over HTTP, this is fine but the blocking nature of synchronous HTTP requests becomes a bottleneck when we want to scale for thousands of RPC calls. So instead of requests module, we will use aiohttp which is built around python's asyncio capabilities. We will first create a class that abstracts away all aiohttp stuff and provides a cleaner API to use as a python module.

class EthAioAPI:

    def __init__(self):
        pass

    async def __aenter__(self):
        self.session = aiohttp.ClientSession()
        return self

    async def __aexit__(self, *err):
        await self.session.close()
        self.session = None

This class has __aenter__ and __aexit__ methods which will be called upon entry and exit of asyncio scope respectively, we will use these methods to manage our session object which will be created and closed automatically as we enter and leave the scope.

We want to pass in the URL to which we want to connect in the constructor. Our API should accept N tasks and execute them together as batch or concurrently. To accommodate these features, we modify the above code as follows:

class EthAioAPI:

    def __init__(self, url: str, max_tasks=100):
        self.url = url
        self.current_tasks = []
        self.current_id = 0
        self.max_tasks = max_tasks

    async def __aenter__(self):
        self.session = aiohttp.ClientSession()
        return self

    async def __aexit__(self, *err):
        await self.session.close()
        self.session = None

    def push_task(self, task: dict):

        if self.current_id >= self.max_tasks:
            raise Exception("maximum tasks exceeded")

        payload = {
            "jsonrpc": "2.0",
            "method": task["method"],
            "params": task["params"],
            "id": self.current_id
        }

        self.current_tasks.append(payload)
        self.current_id += 1

    def set_max_tasks(self, n: int):
        self.max_tasks = n

We added few members in the constructor, the url stores the URL string of the gateway RPC node to which we want to connect, current_tasks holds the list of tasks submitted via push_task method and current_id keeps track of a number that is just incremented every time to assign a new ID for the task. max_tasks represents how many tasks we can execute together, as you can see a check is made in push_task method before allowing the task to be added to the list, we can use set_max_tasks anytime to change this limit. Now that we have necessary structures to hold our tasks, we can go ahead and build functions to execute these tasks together - we can do this in two ways, i.e either concurrently or as a batch.

Making concurrent (asynchronous) requests:

aiohttp provides us capabilities to execute tasks concurrently, using the asynchronous system calls provided by the operating system, in other words, unlike requests module, here each call is executed in a non-blocking way without blocking our main thread, thus we can schedule N requests at a time to run in background and await for results. To achieve this, we create an asynchronous function called tasklet which makes a single JSON-RPC call without blocking. To run N concurrent requests we call tasklet N times and they are executed concurrently. Here is our tasklet function:

async def tasklet(id: int, url: str, session: aiohttp.ClientSession, payload: dict) -> dict:
        try:
            response = await session.post(
                url, json=payload,
                headers={"Content-Type": "application/json"}
            )
            json_resp = await response.json()
            json_resp['success'] = True
            return json_resp
        except Exception as e:
            return {"success": False, "exception": e, "id": id}

The tasklet function makes a web3 call, if there is a failure returns the failure information or it returns the response dictionary by adding a new entry "success": True to the response. Finally we create a class method exec_tasks_async which calls a tasklet for each task submitted using push_task method.

async def exec_tasks_async(self):
        fns = []
        for id, task_payload in enumerate(self.current_tasks):
            task_fn = EthAioAPI.tasklet(
                id, self.url, self.session, task_payload)
            fns.append(task_fn)

        outputs = await asyncio.gather(*fns, return_exceptions=True)

        self.current_tasks.clear()
        self.current_id = 0
        return outputs

This method pushes tasklet function for each task to the event loop and calls them together using asyncio.gather() and gathers the outputs, the outputs are returned in the same order in which the tasks are submitted. Once we receive the outputs we reset our task list so new tasks can be submitted next time.

This method allows us to call N JSON-RPCs at a time without waiting for individual calls to complete. However, in this approach we are using the resources on the client side to open N concurrent sockets and also we end up making lot of Web3 calls which might sometimes get rate-limited.

Making batch request:

As per JSON-RPC spec, we can send a list of RPC calls at a time and expect the results for all the calls from the server at once. This is left to the implementors of the JSON-RPC server, they can choose to implement this functionality or choose to ignore it, however few Ethereum JSON-RPC implementations supports batched calls. We will now add exec_tasks_batch class method to add Batch calls support in our client:

async def exec_tasks_batch(self):
        try:
            response = await self.session.post(
                self.url, json=self.current_tasks, headers={
                    "Content-Type": "application/json"}
            )

            json_resp = await response.json()
            self.current_tasks.clear()
            self.current_id = 0

            return json_resp

        except Exception as e:
            raise e

As you can see, this method just submits the entire task list as it is and the output we get is a list of responses for each call in the list.

In this method, we will be using the resources of the RPC server to execute N tasks at a time for us, in client side we need only one socket and we open only one connection. (Here we are depending on the server. So make sure the RPC server you are connecting to supports this functionality).

Final code:

import asyncio
import aiohttp


class EthAioAPI:

    def __init__(self, url: str, max_tasks=100):
        self.url = url
        self.current_tasks = []
        self.current_id = 0
        self.max_tasks = max_tasks

    async def __aenter__(self):
        self.session = aiohttp.ClientSession()
        return self

    async def __aexit__(self, *err):
        await self.session.close()
        self.session = None

    async def tasklet(
        id: int,
        url: str,
        session: aiohttp.ClientSession,
        payload: dict
    ) -> dict:

        try:
            response = await session.post(
                url, json=payload,
                headers={"Content-Type": "application/json"}
            )

            json_resp = await response.json()
            json_resp['success'] = True

            return json_resp

        except Exception as e:
            return {"success": False, "exception": e, "id": id}

    def push_task(self, task: dict):

        if self.current_id >= self.max_tasks:
            raise Exception("maximum tasks exceeded")

        payload = {
            "jsonrpc": "2.0",
            "method": task["method"],
            "params": task["params"],
            "id": self.current_id
        }

        self.current_tasks.append(payload)
        self.current_id += 1

    async def exec_tasks_batch(self):
        try:
            response = await self.session.post(
                self.url, json=self.current_tasks, headers={
                    "Content-Type": "application/json"}
            )

            json_resp = await response.json()
            self.current_tasks.clear()
            self.current_id = 0

            return json_resp

        except Exception as e:
            raise e

    async def exec_tasks_async(self):
        fns = []
        for id, task_payload in enumerate(self.current_tasks):
            task_fn = EthAioAPI.tasklet(
                id, self.url, self.session, task_payload)
            fns.append(task_fn)

        outputs = await asyncio.gather(*fns, return_exceptions=True)

        self.current_tasks.clear()
        self.current_id = 0
        return outputs

    def set_max_tasks(self, n: int):
        self.max_tasks = n

We can create a folder called aio_eth and copy this code as __init__.py which allows us to import aio_eth and use it as a module.

Let's test!

We can use the class we built in the previous sections to test and verify the functionalities, First let's execute a sample JSON-RPC call N times concurrently and check how it works:

import asyncio
# our module
import aio_eth
import time

URL = "https://rinkeby.infura.io/v3/b6fe23ef7add48d18d33c9bf41d5ad0c"

async def query_blocks():
    async with aio_eth.EthAioAPI(URL, max_tasks=100) as api:
        for i in range(10553978, 10553978 + 70):
            api.push_task({
                "method": "eth_getBlockByNumber",
                "params": [
                    hex(i), True
                ]
            })

        st = time.time()
        _ = await api.exec_tasks_async()
        et = time.time()
        print('time taken: ', et - st, ' seconds')

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    result = loop.run_until_complete(query_blocks())

Output:

time taken:  1.5487761497497559  seconds

So it takes us around 1.5 seconds to execute 70 RPC calls asynchronously. Now let's try batch JSON-RPC call:

import asyncio
import aio_eth
import time

URL = "https://rinkeby.infura.io/v3/b6fe23ef7add48d18d33c9bf41d5ad0c"

async def query_blocks():
    async with aio_eth.EthAioAPI(URL, max_tasks=100) as api:
        for i in range(10553978, 10553978 + 70):
            api.push_task({
                "method": "eth_getBlockByNumber",
                "params": [
                    hex(i), True
                ]
            })

        st = time.time()
        _ = await api.exec_tasks_batch()
        et = time.time()
        print('time taken: ', et - st, ' seconds')

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    result = loop.run_until_complete(query_blocks())

Output:

time taken:  3.698002576828003  seconds

To achieve same result using Batch JSON-RPC call it takes us 3.7 seconds, which is twice as that of concurrent JSON-RPC calls. However, this varies across different methods and amount of data the server returns over the network. To achieve the same in sequential way, it would take us around 65-70 seconds, thus using either concurrency or batch mode we can save lot of time and hence we can scale our client to make lot of JSON-RPC calls in less time.

Getting aio_eth package from PyPi

I have published the above code as a PyPi package so people can download and use it. The PyPi package can be found here. We can also use pip to install this package:

pip install aio-eth

The package is open source and can be found on Github.

Let's build a simple system calls monitoring dashboard using bpftrace and streamlit

Narasimha Prasanna HN — Sat, 16 Apr 2022 09:22:09 +0000

If you are from Computer Science background probably you might have heard of "system calls" (also referred as "syscalls" to keep it short), if you haven't then you can read this interesting beginner friendly article on system calls. In general, system calls are the glue between Operating System kernel and User applications. System Calls are required for everything, without system calls the user application cannot do anything, because user applications need kernel support for memory management activities, reading and writing from/to disk, display, network interface, audio devices etc. In general, the user application becomes virtually useless if it cannot make system calls. Let us write and compile a simple C program that reads a file from the disk and outputs it, once we compile the program, we will use strace to see how many system calls our program makes to get the job done. (I will be using Ubuntu 21.04 for all the experiments in this article).

So this is our C program, it is simple and straightforward, we are just opening the file called text.txt (assume that it is present) using fopen(), reading the first line using fscanf and printing it to the terminal using printf.

#include <stdio.h>

int main(int argc, char **argv) {
    char buffer[1024];
    FILE *fptr = fopen("text.txt", "r");
    if (fptr == NULL) {
            printf("failed to open text.txt");
    }
    fscanf(fptr, "%[^\n]", buffer);
    printf("Read data: %s\n", buffer);
}

Let's compile this:

gcc read_file.c -o read_file

We should be able to run the binary read_file, now let's trace what all system calls this program made. We will use strace to count the number of system calls made by this program when executed. Let's run:

strace -C ./read_file

This should print a table showing some basic statistics of the system calls made during the program execution.

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 49.67    0.000301         301         1           execve
 14.69    0.000089          11         8           mmap
  6.44    0.000039          13         3           openat
  4.46    0.000027           9         3           mprotect
  4.46    0.000027           6         4           newfstatat
  3.47    0.000021           5         4           pread64
  3.14    0.000019          19         1           munmap
  2.81    0.000017           8         2           close
  2.64    0.000016           5         3           brk
  2.15    0.000013           6         2           read
  2.15    0.000013          13         1         1 access
  1.65    0.000010          10         1           write
  1.65    0.000010           5         2         1 arch_prctl
  0.66    0.000004           4         1           lseek
------ ----------- ----------- --------- --------- ----------------
100.00    0.000606          16        36         2 total

As expected, we have openat, read and write system calls present in the table, but why are there so many other system calls? And also, openat is called thrice but we are opening the file only once - Well, these are expected, because C compiler is going to add some wrapper functions and bootstrap code, some of the core libraries are also dynamically opened and loaded, thus we have lot of other system calls which are unknown to us.

strace

strace (as we saw in the above section) provides lot of other capabilities as well, for example we can also attach it to a PID and trace all the system calls made by that process (We can use strace -p <PID>). We can also configure strace to suite many use-cases by using one or more options provided by it, just run strace -h to list available options. But we have a problem with strace - i.e the overhead of tracing. To understand the problems of strace we need to look at ptrace - the underlying system call used by strace.

About ptrace() system call:

strace and most of the tracing tools like GDB make use of ptrace system call provided by Linux Kernel. Any program can use ptrace system call with various options to get itself attached to another process and set breakpoints to examine the internal state of that process. In our case, strace starts as the parent process and calls ptrace initially passing PTRACE_ATTACH and the PID of the child process that needs to be traced - in our case it is read_file built from read_file.c. The child process also calls ptrace passing PTRACE_TRACEME flag. The kernel then makes some checks and privilege verifications before allowing strace to get attached to the child process, if passed the kernel initializes some internal data structures required for tracing and notifications. Next, strace calls ptrace again, but this time it passes PTRACE_SYSCALL flag, which makes the kernel to write TIF_SYSCALL_TRACE flag into the child processes's internal thread state, which is like an indication that tracing is enabled for that process. After this, the child process is allowed to execute and strace waits for the child to make system calls. If child makes any system call, the TIF_SYSCALL_TRACE will be seen by the kernel's system call entry level function, which will force the child process to trap (halt) by sending SIGTRAP signal. Now that the child process is halted, kernel collects and passes the architecture specific and architecture independent state data of the halted child process to the parent, i.e to our strace process. strace can now parse this information and use it to display the tracing information. After this, child process is allowed to execute the system call, once system call is complete, another trap is generated by passing SIGTERM to the child and parent is notified about the completion of system call again by passing all the architecture specific and architecture independent state data, then the child process is allowed to continue. This cycle repeats for all the system calls made by the child process. So the tracer i.e the parent (strace) is notified twice per system call - one at system call entry and another one at exit.

Now that we know about ptrace system call and the tool strace built on top of it, we can look at the disadvantages of this approach:

ptrace cannot be used for system wide tracing - this is because we need to attach tracing to all the available processes on the system which is not recommended. In general ptrace is process specific.
This method involves lot of interactions between kernel and user-space because ptrace passes the entire information to the tracer tool via a signal, this information can be huge and it needs to be copied into a buffer in the user-space.
Child process needs to be interrupted upon system call - ptrace sets TIF_SYSCALL_TRACE in the thread state information of the child process which makes the kernel to pause the process's execution upon entry and exit of the system call, this can be a bottleneck for a process that heavily depends on system calls. (Example: Disk or Network intensive applications)

All these drawbacks doesn't mean ptrace is bad, tools written using ptrace can be used for low level debugging and tracing, but these tools are not suitable for continuous system-wide tracing.

eBPF and bpftrace

Recently, starting from Linux kernel 4.x, the community introduced a new functionality called Extended Berkeley Packet Filters (eBPF), eBPF is a small sandboxed virtual machine that executes BPF byte-code (BPF is a simple filtering and data crunching language, before eBPF, BPF was used to filter and analyze network traffic by directly attaching it to data-link layer, read more about BPF here.) completely in the kernel space, eBPF scripts can be attached to many event triggers within the kernel (like interrupts, system calls, breakpoints, function calls etc), kernel maintains the mapping of these event triggers and attached eBPF scripts, then it automatically executes these scripts whenever any of such events occur. This is a very powerful kernel feature, because it allows developers to extend the functionality of the kernel in a event driven way without writing and loading complex kernel modules. It is also possible to dynamically attach/remove these scripts and receive the outputs back in the user-space. Let's see how this can be helpful for monitoring and tracing:

Completely Event Driven: We don't have to anymore stop the execution of the child process (like we did when using ptrace), these eBPF scripts are executed automatically when the event occurs.
Data Crunching in the kernel: If our analysis involves lots of data, we can filter and aggregate them in the kernel itself without transferring it to the user-space - this is because eBPF VM runs in kernel space.
Can monitor system-wide events: eBPF is not restricted to one process or thread, it is system-wide because it runs in the kernel.

This solves some of the issues we had with ptrace based implementations. Recently iovisor project came up with a tool called bpftrace which can be used as an alternative to strace with more number of additional features. bpftrace uses eBPF for core tracing and provides user-space tools to harvest the tracing data. To understand more about bpftrace read this reference guide. bpftrace is not restricted just to system calls tracing, we can also use it to trace Disk I/O operations, network operations, CPU utilization etc. Since bpftrace supports eBPF at it's core, we can use BPF scripts to write our own tracing scripts easily. If you are interested, have a look at some of the cool tracing scripts built using BPF, all these scripts can be readily used with bpftrace.

Let's experiment!

To try out bpftrace, make sure you are using the latest kernel, this is the kernel I am using right now (uname -r) on my Ubuntu 21.04 Linux machine.

5.11.0-49-generic

You can get the latest binary of bpftrace from their GitHub releases page. Once downloaded, check the version. This is the version I am using (just run ./bpftrace --version)

bpftrace v0.14.1

Now, let's run a simple BPF script with bpftrace that emits the count of system calls made by each process running on the system every 5 seconds. We need the output in JSON format so other applications can consume it, we can tell bpftrace to emit JSON output by passing -f json flag. Note that bpftrace needs to be executed as sudo. Here is the command we will execute:

sudo ./bin/bpftrace -f json -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); } interval:s:5 { print(@); clear(@); }'

This script should make bpftrace emit the system call count of all the processes every 5 seconds. Here is the output:

{"type": "attached_probes", "data": {"probes": 2}}
{"type": "map", "data": {"@": {"dockerd": 1, "packagekitd": 1, "epmd": 1, "ibus-engine-sim": 2, "gosecrets": 2, "terminator thre": 2, "ibus-extension-": 2, "pipewire-media-": 3, "GUsbEventThread": 3, "rtkit-daemon": 4, "gvfs-afc-volume": 5, "runsvdir": 6, "Chrome_ChildIOT": 6, "Netlink Monitor": 6, "uml_switch": 6, "docker": 7, "gsd-sharing": 8, "thermald": 8, "MemoryPoller": 9, "containerd": 9, "mc:worker_00": 10, "mc:worker_02": 10, "sh": 10, "Cache2 I/O": 10, "apache2": 15, "dbus-daemon": 16, "ibus-daemon": 17, "mc:worker_03": 18, "avahi-daemon": 20, "systemd": 21, "ThreadPoolForeg": 24, "civetweb-master": 25, "memsup": 30, "memcached": 30, "JS Watchdog": 32, "Socket Thread": 34, "ThreadPoolServi": 35, "NetworkManager": 37, "systemd-resolve": 39, "DOM Worker": 40, "gmain": 44, "pulseaudio": 47, "TaskCon~ller #0": 49, "0_poller": 50, "TaskCon~ller #3": 51, "IPDL Background": 62, "inet_gethost": 65, "threaded-ml": 66, "gnome-terminal-": 75, "goport": 82, "GeckoMain": 84, "TaskCon~ller #2": 93, "4_scheduler": 96, "mc:worker_01": 99, "8_dirty_io_sche": 103, "LS Thread": 132, "gdbus": 146, "JS Helper": 148, "IPC I/O Parent": 151, "IPC I/O Child": 155, "dnsmasq": 170, "df": 202, "bpftrace": 207, "aux": 228, "4_dirty_cpu_sch": 313, "code": 325, "2_dirty_io_sche": 413, "Timer": 422, "gnome-shell": 427, "TaskCon~ller #1": 466, "goxdcr": 661, "3_scheduler": 792, "2_scheduler": 956, "NonIoPool0": 1057, "NonIoPool1": 1063, "godu": 1116, "Isolated Web Co": 1119, "prometheus": 1266, "cbq-engine": 1306, "1_scheduler": 1674, "projector": 2138, "SchedulerPool0": 2278, "alsa-sink-ALC89": 3142, "3_dirty_io_sche": 3208, "indexer": 20237}}}
{"type": "map", "data": {"@": {"wpa_supplicant": 1, "snap-store": 1, "epmd": 1, "CacheThread_Blo": 1, "mc:executor": 2, "ibus-engine-sim": 2, "ibus-extension-": 2, "saslauthd-port": 2, "GUsbEventThread": 2, "thermald": 4, "Compositor": 4, "pool-/usr/libex": 4, "GpuWatchdog": 5, "gvfs-afc-volume": 5, "runsvdir": 6, "Chrome_ChildIOT": 6, "docker": 8, "dockerd": 8, "ibus-daemon": 9, "MemoryPoller": 9, "uml_switch": 9, "jemalloc_bg_thd": 10, "mc:worker_00": 10, "containerd": 14, "apache2": 15, "mc:worker_03": 16, "MediaTrackGrph": 17, "civetweb-worker": 24, "inet_gethost": 26, "JS Watchdog": 30, "memcached": 31, "mc:worker_02": 40, "irqbalance": 40, "civetweb-master": 40, "gmain": 40, "0_poller": 45, "pulseaudio": 64, "systemd-resolve": 69, "GeckoMain": 69, "gnome-terminal-": 73, "gdbus": 74, "TaskCon~ller #1": 83, "threaded-ml": 88, "IPDL Background": 89, "mc:worker_01": 100, "6_dirty_io_sche": 103, "TaskCon~ller #3": 122, "aux": 138, "sigar_port": 148, "bpftrace": 193, "4_dirty_cpu_sch": 210, "IPC I/O Child": 213, "IPC I/O Parent": 215, "TaskCon~ller #2": 222, "goport": 250, "TaskCon~ller #0": 275, "4_scheduler": 313, "gnome-shell": 362, "code": 500, "Timer": 530, "goxdcr": 692, "8_dirty_io_sche": 716, "3_scheduler": 797, "2_dirty_io_sche": 875, "2_scheduler": 956, "NonIoPool0": 1031, "NonIoPool1": 1056, "prometheus": 1144, "3_dirty_io_sche": 1307, "cbq-engine": 1328, "1_scheduler": 1787, "projector": 2034, "Isolated Web Co": 2052, "godu": 2228, "SchedulerPool0": 2293, "alsa-sink-ALC89": 3144, "indexer": 19889}}}

We ran bpftrace for 10 seconds and thus we have two output logs in JSON format (at first we receive attach event).

Let's build a simple dashboard in python to display the live data

Now that we are receiving the system call counts every 5 seconds, let's use it to build a live dashboard. There are hundreds of tools and libraries available for building dashboards these days, I decided to go with streamlit - this python framework will allow us to spin up a dashboard with less code (I wanted to keep it as simple as possible). To get started with streamlit, make sure you have python 3.0+ (usually this will be installed by default on many Linux distributions). We will install streamlit using pip.

pip install streamlit==1.8.1

We will create a subprocess that runs bpftrace and collects the streamed content on stdout line by line, we can use yield for this task. We iterate over the outputs as and when they are emitted and update our streamlit table view. Here is the python code which does that, name it dashboard.py:

import subprocess
import os
import shlex
import json
import streamlit as st
import pandas as pd

SCRIPT = "tracepoint:raw_syscalls:sys_enter { @[comm] = count(); } interval:s:5 { print(@); clear(@); }"

def exec(cmd):
    popen = subprocess.Popen(cmd, stdout=subprocess.PIPE, universal_newlines=True)
    for stdout_line in iter(popen.stdout.readline, ""):
        yield stdout_line 
    popen.stdout.close()
    return_code = popen.wait()
    if return_code:
        raise subprocess.CalledProcessError(return_code, cmd)

def execute_and_listen_for_updates():

    st.title("Real-time syscalls counter")
    table_placeholder = st.empty()
    total_placeholder = st.empty()

    try:
        command = shlex.split("./bin/bpftrace -f json -e '{}'".format(SCRIPT))
        for entry in exec(command):
            entry = json.loads(entry)
            if entry["type"] == "map":
                data_dict = entry["data"]["@"]
                proc_names, counts = [], []
                total = 0
                for k, v in data_dict.items():
                    proc_names.append(k)
                    counts.append(v) 
                    total += v

                df = pd.DataFrame(
                    {"counts": counts},
                    index=proc_names
                )

                table_placeholder.table(df)
                total_placeholder.text("total system calls: " + str(total))

    except Exception as e:
        print('internal error:', e)
        os._exit(0)


if __name__ == "__main__":
    execute_and_listen_for_updates()

Thanks to streamlit because we could create a dashboard in less than 55 lines of code. Now let's run the streamlit app:

sudo streamlit run dashboard.py

This should spin up the webserver on port 8501 by default

You can now view your Streamlit app in your browser.

  Local URL: http://localhost:8501
  Network URL: http://192.168.0.106:8501

We can open the dashboard at http://localhost:8501 or http://<your-lan-device-ip>:8501 to see the live system call table which updates every 5 seconds. Here is a screenshot of the dashboard from my computer:

<img src="https://raw.githubusercontent.com/Narasimha1997/blog/gh-pages/assets/syscall-dashboard/dashboard.png" alt="drawing">

If you are interested to explore more, you can explore many other streamlit charts. The code I wrote above is not so optimal and many things can be improved, I just wrote that to demonstrate how bpftrace can be integrated with other applications. There is a tool called pixie which provides observability for kubernetes clusters using eBPF underneath, if you are interested you can also have a look at it as well.

How I built my own blog without much coding

Narasimha Prasanna HN — Tue, 29 Mar 2022 13:02:37 +0000

Two days back, I started planning to build my own blogging site. Since it was a calm weekend, I had enough time to explore various ways I can try to build my own blogging site. Most of the initial solutions that came to my mind involved building a full fledged blogging application on my own that involved many fancy features like Database, user registration, comments, likes, views count, interactive content etc. However, soon I decided not to go about it because it would be an overkill for what I am intending to do. My requirements to be precise (at a high level) were as follows:

Create a blog without much coding and it must be done in few hours, so I can enjoy my weekend.
Should be easy to add new posts every now and then - as easy as just creating a new file for every post.
Pagination - this was an important requirement because I wanted the viewers to see few posts at a time in chronological order without bombarding their UI with all the available posts in a single list (this would also increase the overall load time as the blog grows)
Should support markdown syntax - because it has good expressability while maintaining simplicity.
Easy to deploy and publish - in other words I wanted something like a CI/CD mechanism that is deeply integrated with platforms like GitHub, because I wanted to use Github-Pages for serving my blog.

Going further in this post, I will be explaining how each of these requirements was satisfied. After exploration and quick googling I found this tool called jekyll, to my surprise, it more of less supported all my requirements (with some additions).

Jekyll to the rescue:

Jekyll is a Ruby package that allows us to write content as plain text (of course using Markdown - as per requirement 4) and transform it into a static website without having to worry much on building something from scratch (as per requirement 1). It also allows for customization, we can add our own styles, header, footer etc. To my surprise, GitHub provides capabilities to build github-pages with Jekyll, they even have a well established workflow that listens for commits, automatically trigger the build process and publishes the site with new changes (as per requirement 5). We also have many plugins built for Jekyll to extend its core functionality - thank god we also have a pagination plugin (as per requirement 3).

I decided to write this post to help others get started easily without writing much code.

1. Getting Started - Create a GitHub Repository and enable gh-pages:

This is fairly easy, if you have used GithHub before, most probably this will be like a cake-walk for you.

Follow this tutorial to create a new repository.
Follow this tutorial to enable gh-pages feature for the repository you created.

In my case, I wanted all the codebase related to my blog to be under gh-pages branch and not under main or master, so I selected gh-pages as the source branch. GitHub also provides some pre-configured jekyll themes for you to choose, I selected hacker theme, because I am a hacker fanboy - who grew up watching Matrix and Mr.Robot.

Once done, clone the repository to make modifications locally and test it out, In my case it was:

# clone the repository
git clone git@github.com:<your-username>/<your-repo-name>.git
# don't forget to check gh-pages branch
git checkout gh-pages

2. Installing Ruby, Gem and Jekyll for local development and testing:

To test your blog locally you might need to install Ruby and other tools, this will be useful during the initial stages because you will be making lot of changes to the codebase. Run these commands to install Ruby (I use ubuntu, if you are on a different Linux distribution based on Red-Hat or other operating system - you can refer to this page.)

On Ubuntu 20.04+:

# start with an update (just to stay updated)
sudo apt update
# install ruby (gem will be installed along Ruby), We get tools like gcc, g++ and make via build-essential
sudo apt install ruby-full build-essential zlib1g-dev

To make sure you are all set, just check ruby and gen versions.

ruby -v
# on my system, it shows: ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux-gnu] (can be different on your machine based on architecture and OS you are using)

gem -v
# 3.2.5 (on my machine)

gem or RubyGems is a package manager for Ruby, just like how we have npm, pip and cargo for Node, Python and Rust. Jekyll must be downloaded as a gem package, so we use gem command to do that. But for building the website locally we need lot of other tools, github-pages gem provides these tools for us, jekyll is also packaged along with github-pages. Therefore you need to install only github-pages gem.

# use sudo if you are getting permission error
gem install github-pages

3. Configure your blog

Once jekyll and other tools are installed, you can set-up your blog. The easiest way is to clone my repository and checkout the gh-pages branch. Most of the source code you see in my repository is borrowed from tocttou/hacker-blog. Once cloned, copy the contents of my repository to your repository (under gh-pages branch), Run these commands:

# clone my repo
git clone git@github.com:Narasimha1997/blog.git
# change directory to by repo you cloned just now
cd blog
# checkout gh-pages branch
git checkout gh-pages
# remove all my existing posts
rm -r _posts/*.md
# copy all the contents to your repo directory
cp -r . /path/to/your/repo

Now go back to your project directory and edit the _config.yml file according to your needs. The current _config.yml looks like this:

# title and description of the site (will be used in <title> tag)
title: Narasimha Prasanna HN
description: Software Developer - Python, JavaScript, Go, Rust
# use hacker theme
theme: jekyll-theme-hacker
# this is the base URL (use http://localhost:4000/blog/ to access locally)
baseurl: /blog
plugins:
  # use paginator plugin
  - jekyll-paginate
defaults:
  -
    scope:
      path: ""
      type: "posts"
    values:
      layout: "post"
source: .
destination: ./_site
permalink: /:title
# display 3 posts in a page
paginate: 3
paginate_path: /page/:num/
# this will be displayed as the banner of the blog's home page
banner: "root@prasanna-desktop:~#"
# your linkedin profile
linkedin: https://in.linkedin.com/in/narasimha-prasanna-hn-17aa89146
# your Github profile
github: https://github.com/Narasimha1997
# your portfolio
portfolio: http://prasannahn.ml/

The comments in this file will guide you to understand the meaning of each parameter. Once modified, you should be able to serve your blog locally. Run:

jekyll serve

Then you should be able to view the site at http://localhost:4000/blog/. Jekyll supports live-reloading, so you can view your changes reflected on the site without running jekyll serve command again.

4. Publish your blog to Github:

Once you are satisfied with the configuration, stage your changes, make local commit and push it to the remote branch (i.e gh-pages). This can be done by executing following commands:

git add .
git commit -m "<some nice message>"
git push origin gh-pages

Now go to the repository on Github, you will see that a workflow has been triggered, this workflow will perform
almost similar steps you did locally and deploys the website. Once the workflow is complete you can check your blog live at: https://<your-username>.github.io/<your-repo-name> for me it is https://Narasimha1997.github.io/blog, which you can view here.

Originally published on my blog.

The common misconception about TypeScript

Narasimha Prasanna HN — Sun, 21 Mar 2021 11:47:33 +0000

Recently my friend started learning TypeScript, yesterday he came to me with a doubt, he ran into a problem and he couldn't figure it out for days. (He is a newbie but a curious learner). His doubt made me realize a common misconception some people have about TypeScript. So I am writing this post to explain what leads to the misconception and how it can be rectified. Please note: This post is for newbies and aspiring JavaScript developers, an experienced developer would feel this is an obvious thing.

A brief overview:

My friend was actually trying to build a front-end using React and TypeScript. I may not reveal the exact details of what he was trying to achieve but I try to give similar example. His setup had a web-server written using Express.js and it had some APIs. Some part of the front-end code made a GET request to the API and receives the response in JSON format, then it would manipulate the content to display the result on the web-page. I will try to simulate such environment by writing two script files.

The web-server code, in the simulation, the web-server will have a dummy end-point and returns a dummy JSON object upon request.
The front-end code, which is just a script in my case, it makes HTTP GET request and fetches the object, it them performs a simple operation on that object and console logs the result, written in TypeScript, which I will compile to JavaScript using the official type-script compiler.

The server code: (server.js)

const express = require('express')

app = express()

app.get("/dummy", (req, resp) => {
    return resp.status(200).json({
        dummyValue : 121
    })
})

app.listen(6000, () => {
    console.log("Running server")
})

This code looks simple, it should be clear that the server returns an integer field called dummyValue with some random value.

The client: (client.ts)

import axios, { AxiosResponse } from 'axios'


interface DummyResponse {
    dummyValue: number;
}

const generateResult = (response: DummyResponse): number => {
    return response.dummyValue + 1
}

const makeRequest = async (url: string) => {
    try {
        const response: AxiosResponse<DummyResponse> = await axios.get(url)

        if (response.status !== 200) {
            throw `Got response ${response.status}`
        }

        if (!response.data) {
            throw "No data in the response"
        }

        const respJson: DummyResponse = response.data
        const result: number = generateResult(respJson)

        console.log(`Result : ${result}`)

    } catch (err) {
        console.log(`Failed to get response err string = ${err}`)
    }
}

makeRequest('http://localhost:6000/dummy')

The client code is written in TypeScript, the script uses axios to make an HTTP GET request, the script clearly defines interfaces and types wherever necessary. generateResult function takes the response object and increments dummyValue by 1. Then the value is simply returned. You can also have this package.json if you wish to reproduce:

{
  "name": "client",
  "version": "1.0.0",
  "description": "",
  "main": "server.js",
  "scripts": {
    "build-client": "tsc client.ts",
    "test-client": "node client.js",
    "start-server": "node server.js"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@types/node": "^14.14.35",
    "axios": "^0.21.1",
    "express": "^4.17.1",
    "typescript": "^4.2.3"
  }
}

I have defined three script commands here. build-client will build the JavaScript file client.js from client.ts using tsc. test-client will run the generated client.js file using the local node.js environment. start-server will start the web-server written using express.js.

To compile and get started, you can copy these three files locally, and run the following commands:

npm i
npm run build-client

The problem:

As of now, let's run the server and test the client script.
To run the server:

npm run start-server

Next, in another terminal you can run the client:

npm run test-client

The client produces the following output as expected:

Result : 122

This is fine, the client just did what it was expected to do, it made a request to http://localhost:6000/dummy and got the result {'dummyValue' : 121}, it then added 1 to the dummyValue and so the result is 122.

Now we will change the server side code a bit, but we will not touch the client side, we reuse the same compiled JavaScript client.js. Let's change the server code like this:

const express = require('express')

app = express()

app.get("/dummy", (req, resp) => {
    return resp.status(200).json({
        dummyValue : "121"  //dummyValue has a string value now.
    })
})

app.listen(6000, () => {
    console.log("Running server")
})

We did a very simple change, we just made dummyValue to contain a string value instead of a number type. We start the server again and run the same client code again, but we get the following output:

Result : 1211

We got an output, without any failures, but is this correct?? Of course not!! This is a very big difference, instead of getting 122 we got 1211, imagine how this wrong result can mess up further computations, or imagine how damaging this would be if this is some blood-preassure monitoring system or something related to healthcare!! LoL. The developer who built the front-end would not be aware of this after deployment unless he checks the app periodically.

My friend couldn't figure this out because the code still worked fine and it did not return any errors or warnings, but he came to me when he noticed the issue. (Remember, he is new to JavaScript)

Keeping this in mind, let's dive into the concepts.

Why people use TypeScript??

First, you have to understand why we need TypeScript when coding with JavaScript is enough for most of the use-cases. People use TypeScript for compile-time type-safety. As per the definition from Microsoft, TypeScript is a superset of JavaScript and provides many built-in features that are absent in vanilla JavaScript. Compile-Time type-checking is one of the major features. In general, TypeScript is an extension to JavaScript where every symbol has a fixed/static type. JavaScript doesn't care about the types, the variables/constants can take any value and they can be changed as well (for variables) at any point. So JavaScript is an untyped language. People can find untyped languages like JavaScript easier to use, but the real issue arise when codebase grows larger, you might end up in a point where you no longer can keep track of the variables you used and the type of each variable, because you have to keep track of type-safety by yourself to maintain integrity of results and to avoid unnecessary bugs.

TypeScript solves this problem, it binds a static type to each of the symbol you use and keeps track of the assignments by itself, since TypeScript does this for you, you don't have to worry about doing it yourself. Thus, TypeScript makes larger codebases easier to maintain and shareable across developers and teams. But wait! There is a catch.

Compile Time v/s Runtime Type-checking:

Even though TypeScript is a superset of JavaScript, it cannot run on it's own. It is just a JavaScript generator. In the end, the TypeScript code gets compiled to plain JavaScript. The JavaScript generated from TypeScript can be executed on any JavaScript implementation. During compilation, TypeScript compiler checks for type-mismatches and report such errors. For example, a number can be added to only another number, you cannot add a string to a number. Since such issues are reported at compile-time, developer can make sure there are no bugs due to mixing of types in production, which is a good thing!

This doesn't mean such issues will never occur again. Because the type-safety is evaluated only once i.e during the compilation and never again. The JavaScript that gets generated from TypeScript doesn't embed any type related information as JavaScript implicitly does not care about types. In other words, at runtime your application is still not type-safe.

Issues in handling Dynamic Data (no fixed schema/types)

Since TypeScript gaurentees only Compile-Time safety, no errors at compile-time doesn't mean your application will never crash. At compile-time there is no dynamism, that means, you assumed one fixed data-type and worked on it. Imagine this data comes from an external source (from an external API/service), then the data is dynamic and its structure or type can change at anytime, but the application you wrote and deployed using TypeScript will not take this into account, because at runtime, your typescript application exists as a plain untyped JavaScript. In most of the cases, the type-conversion is automatic and works on the principles defined by JavaScript implementation, for example a number is type-casted to string when added with another string, this happens silently and without any notification or exception. This is the most common type of bugs in JavaScript, as many websites deal with unpredictable and dynamic data from external/third-party APIs.

In the example I considered, the client code statically defined the type of the API response through DummyResponse interface, the interface assumed the dummyValue key to be a number type, because of this reason the function generateOutput was able to add 1 to the dummyValue without any compile-time errors, since both the values of addition were of the same type. In case two, however, the type of dummyValue changed to a string on the server-side, but the client was not aware of this change, even though this was against the principles of TypeScript, the error was ignored because it was the Runtime JavaScript that saw the dynamic output and performed the operation without considering the type of dummyValue.

This is not the case in any strongly-typed language because these languages will at least throw a runtime exception or an error. (Languages like Go and Java)

Is this really a problem of TypeScript?

No, it is not, because TypeScript never promised run-time type-checks implicitly. People often misunderstand this and assume TypeScript provides both run-time and compile-time type safety. The notion of a Type goes away once the script is compiled. If you are familiar with Python, you can compare TypeScript to Python's Type-System, these tools exist to help developers to get rid of bugs and headache during development, but many people assume it can handle both cases of type checking. This happens because of the knowledge gap, a developer who is not aware of Types or Type checking might fail to understand this limitation of TypeScript and ignore to do explicit type-checks on dynamic data.

How to avoid this problem?

The solution to this problem is straightforward, do explict type-checks, I can modify generateOutput function to incorporate explicit type-check as follows:

const generateResult = (response: DummyResponse): number => {
    try {
        if (typeof response.dummyValue !== "number") {
            throw `Improper type of dummyValue, expected number, got ${typeof response.dummyValue}`
        }
        return response.dummyValue + 1

    } catch (err) {
        console.log(`Failed to generate result, error = ${err}`)
        return NaN
    }
}

The function performs type-check and throws an exception if the condition is not satisfied. There are literally hundreds of ways to perform explicit type-checking. If you don't want to write code yourself for type-checking or you deal with a complex data with many nested objects, then you can consider using some popular validation library from npm. Here I try list a few of them:

These libraries can perform validation on complex objects using simple schema definitions. You can also look at how Mongoose ODM does schema validation against MongoDB data and follow a similar schema structure.

Avoiding explicit Type-checks at front-end:

There is no suitable way to get rid of type validation completely, because JavaScript doesn't perform type-checks implicitly, but you can avoid it to some extent by changing your application architecture, here are few tips:

Implement dynamic schema validation system and let the front-end fetch infer the schema from the backend and change it's validation flow accordingly, by this way you can avoid changing schema at multiple places. Check [https://json-ld.org/] for a similar analogy.
Do not consume data from external/third-party APIs directly, build a backend service that acts like a proxy, implement validation function in the backend, this backend can also filter-out some unnecessary fields that are not required by the front-end. By this way you can keep the front-end clean and handle all the complexities at the backend, this is also a good security practice. Check Dependency inversion principle
Consider using GraphQL as it performs validation internally.
You can also consider Protobuf + gRPC instead of HTTP/s + REST.

Thanks for reading!

Create Kubernetes Jobs in Golang using K8s client-go API

Narasimha Prasanna HN — Sat, 13 Feb 2021 08:45:51 +0000

Few months back, I was building a system that involved launching and monitoring jobs on Kubernetes programmatically, basically I was writing a controller kind of a service that would take requests from an API and schedule jobs on the Kubernetes cluster. I was using Go and there is only one fully fledged client API in go called client-go. In this post I am just documenting the process of creating k8s jobs using client-go, so that anyone working on the same problem would find this helpful.

Prerequisites:

A working Kubernetes setup, if you don't have a multi-node cluster, you can setup a local kubernetes environment using minikube or microk8s.
Basic knowledge of Go is necessary.
Must have setup a working Go modules directory.

Installing client-go package:

The client-go library is an official kubernetes client SDK by K8s community, you can use this library to programmatically manipulate your kubernetes cluster. Basically all Kubernetes controllers and components are built using this library. The kubernetes client tool kubectl is also built using client-go. client-go has following packages, which you must be aware of, before installing it:

kubernetes : This package offers you APIs using which you can connect to your Kubernetes service. It provides different kinds of connection APIs. It also provides you the access to clientset interface, which can be used to specify and manipulate K8s objects.
discovery : This package offers you APIs to discover K8s APIs.
dynamic : client-go provides you built-in types and constructs to specify any object, this strongly typed nature helps developers to identify type-errors at compile-type, but at the same time, the type system is rigid and offers less flexibility, on the other hand, dynamic API can be used to gain more flexibility because it allows developers to specify the objects as a nested map (can be de-serialized from JSON, YAML easily), this nested map can be changed easily as the need changes without having to re-compile the codebase, since the map is created dynamically and type-checking happens at runtime.
plugin/pkg/client/auth : This module offers you authentication plugins.
transport : This package is used to set up auth and start a connection.
tools/cache : This package is useful for writing controllers.

To install client-go as a module, you need to have the latest Go version which supports module system. See this tutorial if you want to know more about Go module system. Let's create a simple go module called "github.com/k8sjobs"

mkdir k8sjobs
cd k8sjobs && go mod init github.com/k8sjobs

If everything is ok, you must see go.mod file created. Let's check.

cat k8sjobs/go.mod

Output:

module github.com/k8sjobs

go 1.15

So, we have the module ready, now let's install client-go. Client-go has many versions, we will install one of the latest stable versions of client-go. From the module root, i.e from k8sjobs directory, run :

go get k8s.io/client-go@v0.20.2

You many have to wait for installation to get complete. We can verify the installation by checking go.mod file.

module github.com/k8sjobs

go 1.15

require k8s.io/client-go v0.20.2 // indirect

The go.mod has client-go as one of it's indirect dependencies which is fine as of now because we don't have any source file which imports k8s.io/client-go yet.

What we will build?

We will build a simple command line tool that takes job name, container name and entrypoint as arguments and creates a job on the kubernetes cluster. We will be using golang's built-in package - flag to do this. We will create a file called main.go inside our module's root, which does everything that's required. So, let's start coding.

import (
    "flag"
    "fmt"
)
func main() {
    jobName := flag.String("jobname", "test-job", "The name of the job")
    containerImage := flag.String("image", "ubuntu:latest", "Name of the container image")
    entryCommand := flag.String("command", "ls", "The command to run inside the container")

    flag.Parse()

    fmt.Printf("Args : %s %s %s\n", *jobName, *containerImage, *entryCommand)
}

So now we have a basic main function setup, this will simply accept arguments jobName, containerName and imageName and prints them. Now let's begin using client-go library.

1. Connect to the cluster:

The first step is to connect to the cluster. Connection to the cluster requires K8s cluster config file, since we are connecting to an external remote cluster. Since it is assumed that you have a cluster set-up, you must be having the config file ready. It is assumed that the cluster config file is located at $HOME/.kube/config. We will write a function called connectToK8s.

Our imports till now:

import (
    "flag"
    "log"
    "os"
    "path/filepath"

    clientcmd "k8s.io/client-go/1.5/tools/clientcmd"
    "k8s.io/client-go/kubernetes"
)

The function connectToK8s connects to the cluster and then creates a clientset and returns it.

func connectToK8s() *kubernetes.Clientset {
    home, exists := os.LookupEnv("HOME")
    if !exists {
        home = "/root"
    }

    configPath := filepath.Join(home, ".kube", "config")

    config, err := clientcmd.BuildConfigFromFlags("", configPath)
    if err != nil {
        log.Panicln("failed to create K8s config")
    }

    clientset, err := kubernetes.NewForConfig(config)
    if err != nil {
        log.Panicln("Failed to create K8s clientset")
    }

    return clientset
}

Now we call the function inside main:

clientset := connectToK8s()

Now we have the clientset, we can create a job on the cluster. To do this, we will write a function called launchK8sJob which will be of the form:

func launchK8sJob(clientset *kubernetes.Clientset, jobName *string, image *string, cmd *string)

But before we begin, let's recall how the K8s job specification looks like:

apiVersion: batch/v1
kind: Job
metadata:
  name: ls-job
spec:
  template:
    spec:
      containers:
      - name: ls-job
        image: ubuntu:latest
        command: ["ls", "-aRil"]
      restartPolicy: Never
  backoffLimit: 4

This is the sample specification taken from here. The K8s specification uses batch/v1 API to create Jobs, since everything in Kubernetes has a pre-defined template, we have a template for Jobs creation as well, which is shown above. The kubernetes clientset type provides Batchv1 construct which we can use to manipulate jobs. Let's create a job using Batchv1 construct.

Here are the imports till now:

import (
    "context"
    "flag"
    "log"
    "os"
    "path/filepath"
    "strings"

    batchv1 "k8s.io/api/batch/v1"
    v1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    kubernetes "k8s.io/client-go/kubernetes"
    clientcmd "k8s.io/client-go/tools/clientcmd"
)

Now the function launchK8sJob:

func launchK8sJob(clientset *kubernetes.Clientset, jobName *string, image *string, cmd *string) {
    jobs := clientset.BatchV1().Jobs("default")
    var backOffLimit int32 = 0

    jobSpec := &batchv1.Job{
        ObjectMeta: metav1.ObjectMeta{
            Name:      *jobName,
            Namespace: "default",
        },
        Spec: batchv1.JobSpec{
            Template: v1.PodTemplateSpec{
                Spec: v1.PodSpec{
                    Containers: []v1.Container{
                        {
                            Name:    *jobName,
                            Image:   *image,
                            Command: strings.Split(*cmd, " "),
                        },
                    },
                    RestartPolicy: v1.RestartPolicyNever,
                },
            },
            BackoffLimit: &backOffLimit,
        },
    }

    _, err := jobs.Create(context.TODO(), jobSpec, metav1.CreateOptions{})
    if err != nil {
        log.Fatalln("Failed to create K8s job.")
    }

    //print job details
    log.Println("Created K8s job successfully")
}

That's it! We just need to call this function from main. Here is the entire source file:

package main

import (
    "context"
    "flag"
    "log"
    "os"
    "path/filepath"
    "strings"

    batchv1 "k8s.io/api/batch/v1"
    v1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    kubernetes "k8s.io/client-go/kubernetes"
    clientcmd "k8s.io/client-go/tools/clientcmd"
)

func connectToK8s() *kubernetes.Clientset {
    home, exists := os.LookupEnv("HOME")
    if !exists {
        home = "/root"
    }

    configPath := filepath.Join(home, ".kube", "config")

    config, err := clientcmd.BuildConfigFromFlags("", configPath)
    if err != nil {
        log.Fatalln("failed to create K8s config")
    }

    clientset, err := kubernetes.NewForConfig(config)
    if err != nil {
        log.Fatalln("Failed to create K8s clientset")
    }

    return clientset
}

func launchK8sJob(clientset *kubernetes.Clientset, jobName *string, image *string, cmd *string) {
    jobs := clientset.BatchV1().Jobs("default")
    var backOffLimit int32 = 0

    jobSpec := &batchv1.Job{
        ObjectMeta: metav1.ObjectMeta{
            Name:      *jobName,
            Namespace: "default",
        },
        Spec: batchv1.JobSpec{
            Template: v1.PodTemplateSpec{
                Spec: v1.PodSpec{
                    Containers: []v1.Container{
                        {
                            Name:    *jobName,
                            Image:   *image,
                            Command: strings.Split(*cmd, " "),
                        },
                    },
                    RestartPolicy: v1.RestartPolicyNever,
                },
            },
            BackoffLimit: &backOffLimit,
        },
    }

    _, err := jobs.Create(context.TODO(), jobSpec, metav1.CreateOptions{})
    if err != nil {
        log.Fatalln("Failed to create K8s job.")
    }

    //print job details
    log.Println("Created K8s job successfully")
}

func main() {
    jobName := flag.String("jobname", "test-job", "The name of the job")
    containerImage := flag.String("image", "ubuntu:latest", "Name of the container image")
    entryCommand := flag.String("command", "ls", "The command to run inside the container")

    flag.Parse()

    clientset := connectToK8s()
    launchK8sJob(clientset, jobName, containerImage, entryCommand)
}

Note that, we are setting RestartPolicy as Never and BackOffLimit to zero, it means, if the job fails, it will never be restarted by the job controller because of BackOffLimit and the Pod created by the job will die gracefully because of RestartPolicy being set to Never.

Let's build the module and run it. Before building, we set GOBIN to current working directory so that we will get the binary k8sjobs in the current working directory.

export GOBIN=$PWD
go install

If compilation is successful, you should see k8sjobs binary generated in the current working directory. Let's run the binary by providing the arguments.

./k8sjobs --jobname=test --image=alpine:latest --command="ls -aRil"

If the cluster is configured properly, the command must exit successfully and we should see the output something like this.

2021/02/13 13:39:49 Created K8s job successfully

Now let's check the job using kubectl. (I am using microk8s, so I use kubectl binary that comes with microk8s toolchain. You can just use kubectl if you have installed it separately, or just alias microk8s kubectl to kubectl).

microk8s kubectl get jobs

Output:

NAME   COMPLETIONS   DURATION   AGE
test   1/1           73s        13m

The output shows that the job has been created and is completed successfully. Now let's describe the job to check if it's the same job we created.

microk8s kubectl describe job test

Output:

Name:           test
Namespace:      default
Selector:       controller-uid=48b7b61d-4ead-4033-91cd-035f539c27bf
Labels:         controller-uid=48b7b61d-4ead-4033-91cd-035f539c27bf
                job-name=test
Annotations:    <none>
Parallelism:    1
Completions:    1
Start Time:     Sat, 13 Feb 2021 13:39:49 +0530
Completed At:   Sat, 13 Feb 2021 13:41:02 +0530
Duration:       73s
Pods Statuses:  0 Running / 1 Succeeded / 0 Failed
Pod Template:
  Labels:  controller-uid=48b7b61d-4ead-4033-91cd-035f539c27bf
           job-name=test
  Containers:
   test:
    Image:      alpine:latest
    Port:       <none>
    Host Port:  <none>
    Command:
      ls
      -aRil
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Events:
  Type    Reason            Age   From            Message
  ----    ------            ----  ----            -------
  Normal  SuccessfulCreate  15m   job-controller  Created pod: test-l9zwk
  Normal  Completed         14m   job-controller  Job completed

This is the description of the job we created. You can verify that the container image name and the command is same as what we passed as arguments.

So that's it. This is just a basic example of how you can use client-go library to create K8s jobs. If you have free time, explore the library because there are whole lot of functionalities the library offers.

Building a secure/sandboxed environment for executing untrusted code

Narasimha Prasanna HN — Sat, 16 Jan 2021 08:58:04 +0000

What is a Sandbox?

Let's try to understand the meaning of the term sandbox first, before learning how to build one. Sandbox is a like a container that isolates the environment in which the software is run. In other words, a sandbox provides a secure environment which restricts the software inside the sandbox from accessing the resources of the host, the resources can be a file-system, network, some set of kernel system-calls etc. The application of sandbox is obvious, you can execute the code which you don't trust without worrying about security much. Here are some examples that will help you understanding sandbox better:

Sandboxing is built into modern browsers, this restricts malicious websites from stealing your sensitive data or damaging the client machine, because the code runs inside a sandbox and the sandbox restricts the code from calling any host-level functions.
Most of the online coding tutorials that allow remote code execution are powered by sandboxing tools, these tools provide a separate isolated environment for you, thus restricting you from accessing the server resources or files belonging to other users.

In this tutorial, we will be building a simple sandbox solution. Note that this is not the perfect one (Many companies have worked years together on security), but still it manages to give you an idea of building your own sandbox.

What will be the consequences if you don't use a sandbox?

Imagine a scenario where your browser did not provide any security/isolation, in such case anyone good at JavaScript/C++ can somehow manage to access your file-system from the browser itself without even notifying you, this will allow attackers to steal your private content and use them for blackmailing you. Imagine an online code-editing tool without a sandbox, you as the user can write a code that deletes the files on their server, rendering the server useless, you can also use server's resources to mine bitcoins or attack others, you can also delete the files of other users using the tool. These are very critical consequences and any developer who thinks about building a browser, code-editing tool or any software that allows remote-code-execution should consider building a sandbox and securing his/her system from attackers.

What sandbox must provide:

The sandbox should hide the host file-system from the untrusted code which runs inside the sandbox.
It should block the code inside the sandbox from making system-calls directly on the host-kernel which can be dangerous.

Prerequisites of this tutorial:

Basic knowledge of containers and docker.
Basic knowledge of creating and deploying software as containers.
Basic knowledge of Linux and System-Calls.

Solution-1 : Using Containers with Docker

If you are into software development you must be aware of containers. Containers provide an isolated environment where the software can run with all its dependencies. Containers provide their own file-system, so the app that runs inside the container cannot access the host's original file-system. So let's build a simple program to demonstrate this. I will be using C, you can use any language of your choice.

This program just lists all the files from the root / and exits. (list_files.c)

#include <stdlib.h>

int main(int argc, char **argv) {
   system("ls /");
}

This is a simple program and should work without any issues if you are any linux machine. Let's compile this, we will compile this a static binary so that we don't need libc support in our container. But wait, we will first run on the host directly.

gcc list_files.c -static -static-libgcc -static-libstdc++ -o list_files

If compilation is successful, this should produce a binary by name list_files, we will make sure this is static executable and does not depend on libc.

ldd list_files

This will output something similar if it is a static binary.

not a dynamic executable

Now, let's run this, note that we are running directly on the host.

./list_files

This will output all the files under root /

bin    dev   initrd.img      lib64   mnt   root  snap      sys  var
boot   etc   initrd.img.old  lost+found  opt   run   srv       tmp  vmlinuz
cdrom  home  lib         media   proc  sbin  swapfile  usr  vmlinuz.old

Since we ran the code directly on the host machine, we are able to see the contents of the root file-system. The code has exclusive access to our file-system and other resources. So it is not recommended to run untrusted code directly. Now let's containerize it and see how we can provide basic restrictions. We will use busybox a minimal container image. We don't have to install anything specific, remember why we built static executable. This is how our Dockerfile looks.

FROM busybox

COPY ./list_files /list_files
WORKDIR /
ENTRYPOINT ["/list_files"]

So, that's it, we will copy the list_files binary and run it on the start of container. Let's build and run.

docker build . -t sandbox_test
docker run --rm -ti sandbox_test

Now, it should produce the following output:

bin         etc         list_files  root        tmp         var
dev         home        proc        sys         usr

You can see that the / has changed. It is the file-system of busybox our application is seeing, not our host file-system. Containers also provide process-isolation, network isolation etc so our application is somehow isolated. Hurray! We built a simple sandbox (This is not the final solution, read till the end)

Let's refine our work further, till now we manually packaged every application we built as a container image, which is lot of work. So, let's build a generic sandbox container that can run any binary inside the container environment without packing them explicitly. To build this, we follow the steps as below:

We create a small C program that starts inside the container.
The program reads from stdin and writes the contents of stdin to a file inside the container.
It then executes the file it created.
Returns the output back to the host as a string.(writes to stdout)

Let's build this! I am using C, you can use GO, Rust or any language that runs on native metal (rather than interpreted like Java/Python etc). Create a file called sandbox.c and let's write a function called write_stdin_to_file which reads from stdin and writes to a file called binary.

Includes and some definitions:

#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <fcntl.h>
#include <stdbool.h>

#define BUFFER_SIZE 4096
#define OUTPUT_BUFFER 1024
typedef unsigned char uchar;

write_stdin_to_file function:

void write_stdin_to_file(int * size) {

    uchar buffer[BUFFER_SIZE];
    int read_bytes = 0, itrs = 0;
    * size = 0;

    int fd = open("./binary", O_RDWR | O_CREAT, 0777);
    FILE * fp = fdopen(fd, "wb");

    if (fp == NULL) {
        fprintf(stdout, "Failed to open the file for writing\n");
        exit(-1);
    }

    while (true) {
        read_bytes = read(0, buffer, BUFFER_SIZE);
        if (read_bytes < 0) {
            fprintf(stdout, "Failed to read binary data, exiting");
            fclose(fp);
            exit(-1);
        }

        if (read_bytes == 0) {
            //EOF
            break;
        }

        //write data to the file
        *size = *size + read_bytes;
        fwrite(buffer, sizeof(uchar), read_bytes, fp);
    } 

    //wrote the file, close it.
    fclose(fp);
}

And now the main function, main calls write_stdin_to_file and then executes it.

int main(int argc, char **argv) {
    int size = 0, fread_bytes = 0;

    char output_buffer[OUTPUT_BUFFER];
    write_stdin_to_file(&size);

    if (size == 0) {
        fprintf(stdout, "Empty binary file, discarding\n");
        exit(0);
    }

    FILE * process_fd = popen("./binary", "r");
    if (process_fd == NULL) {
        fprintf(stdout, "Failed to execute the binary\n");
        exit(-1);
    }

    fprintf(stdout, "Executing binary inside the sandbox\n");

    //read the data as buffers and stream it to stdout
    while (true) {
        fread_bytes = fread(output_buffer, sizeof(uchar), sizeof(uchar) * OUTPUT_BUFFER, process_fd);

        if (fread_bytes == 0) {
            //EOF
            fclose(process_fd);
            exit(0);
        }

        if (fread_bytes < 0) {
            //Error 
            fprintf(stdout, "Failed to read the output");
            exit(-1);
        }

        output_buffer[fread_bytes] = '\0';

        fprintf(stdout, "%s", output_buffer);
    }

    return 0;
}

Let's compile this as static executable:

gcc src/sandbox.c -static -static-libgcc -static-libstdc++ -o sandbox

Now, lets containerize it using docker.

FROM busybox

COPY ./sandbox /sandbox
WORKDIR /
ENTRYPOINT ["/sandbox"]

Let's build it:

docker build . -t sandbox:latest

After the build is complete, we can run it. But before running, keep this in mind, the entrypoint is sandbox binary, this binary listens of stdin, so we need to pass the binary we need to execute as stdin so that the sandbox binary can execute it inside the container environment.

cat list_files | docker run --rm -i sandbox

This is a simple command, we are using cat to read the binary from host system, after reading, it pipes the output to sandbox program which runs inside the docker. The sandbox executes it inside the container environment and then emits the output as stdout, since stdout is the terminal, the output will be printed on the screen, we can see that below:

Executing binary inside the sandbox
bin
binary
dev
etc
home
proc
root
sandbox
sys
tmp
usr
var

That's it! We have created a container image called sandbox which we can use it to run any binary inside the container environment. But is this the end?? Of course, not!! This approach is still unprotected. Let's see why and how we can address this issue.

Problems with using containers alone as sandbox

We somehow managed to provide network and file-system isolation, but still we are not safe. That's because of the nature of containers. Containers do not provide kernel-level isolation, in other words, eventhough the containers are isolated, they still use host-kernel for their functionality, i.e any system call made by the application inside the container will execute some host-kernel function which is not secure, that means, any clever programmer can write an application that combines multiple system-calls and escape the container isolation to get into the host environment, then he can do anything. But we can solve this as well. Using Userspace kernels.

Userspace Kernel with gVisor:

gVisor by Google is a userspace application kernel written in Go. Userspace kernel is a software that runs completely in user-mode and has less privilege (since it runs in user-mode). It also acts as a kernel emulation layer, that means, it can act as a fake kernel and can receive and process system-calls, thus hiding the host kernel. gVisor is compatible with OCI and provides a OCI runtime called runsc that can be used by container management tool like docker as the underlying runtime. (docker uses runc as the default runtime).

You can install gVisor by following the guide here. Once installed, make sure you have registered runsc as one of the possible runtimes for docker. Check the file /etc/docker/daemon.json, it should contain an entry like this:

{
    "runtimes": {
        "runsc": {
            "path": "/usr/bin/runsc"
        }
    }
}

You can then restart the docker daemon to ensure the changes are applied.

sudo systemctl restart docker

That's it! We can now run our binaries inside a sandbox with complete isolation. Here is how we can include gVisor in our execution command.

cat list_files | docker run --runtime=runsc --rm -i sandbox

You can see --runtime=runsc added, which means we are telling the docker to use gVisor (runsc) instead of runc.

Why I wrote this?

Recently I was building a clone of Go playground as a weekend project, a tool for running go programs online. Folks who developed Go playground were aware of security and they used NaCL sandbox initially. You can read the blog on this here. After the deprecation of NaCL, the developers shifted to this approach to provide security. While I was scratching my head over security for my hobby project, I got a chance to read original playground's source code and found a solution very similar to this. I thought of adding this to my project and also write a blog on this. To learn more, you can follow these repositories:

You can also consider using MircoVM projects as an alternative to gVisor, like :

If you do, please let me know in the comments.
Thanks for spending your precious time in reading this post. Do let me know your opinions and better options in comments.

Communication between Microservices in a Kubernetes cluster

Narasimha Prasanna HN — Tue, 15 Dec 2020 06:28:21 +0000

Kubernetes is a popular, open source container orchestrator which takes care of creating, running and managing your app composed of microservices across multiple nodes. Kubernetes is an ideal choice for deploying and managing microservices these days. It is natural that we want these microservices to talk to each other, Kubernetes provides multiple ways to achieve this. I decided to curate them here so anyone can quickly find a reference if they are working with Kubernetes.

To begin with, we will create a simple setup that will help us realise different examples better. This is not a production grade set-up or any real-world scenario, this is just a simulation of two pods where one pod communicates with another, the first pod is an HTTP web-server and the second is a simple curl client, which makes a request to the web-server and terminates. We will be creating a Job for the client, because Jobs are the best way to deploy terminating instances on K8s.

To test this yourself, make sure you have a working K8s cluster, atleast a minikube

Let's deploy the web-server:

We will be using the web-server image provided by katacoda an interactive K8s learning platform. I will be using the same deployment file provided in one of the playgrounds. (web-server.yaml)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: webapp1
  template:
    metadata:
      labels:
        app: webapp1
    spec:
      containers:
      - name: webapp1
        image: katacoda/docker-http-server:latest
        ports:
        - containerPort: 80

If you are familiar with Kubernetes, you can easily guess what this yaml says. It simply tells K8s to create a deployment which creates a pod, the pod runs the container image katacoda/docker-http-server:latest, it runs on port 80 inside the pod, so any request made to the pod at the port 80 should be received by this web-server. Let's deploy this with kubectl.

kubectl create -f web-server.yaml

If the cluster is properly setup, the deployment must be created and the pod should be running by now. Let's check.

kubectl get deployments

The output:

NAME      READY   UP-TO-DATE   AVAILABLE   AGE
webapp1   1/1     1            1           15m

Now, let's see the pod. (I am using -o wide to see more information about the pod)

kubectl get pods -o wide

Output:

NAME                       READY   STATUS    RESTARTS   AGE   IP           NODE               NOMINATED NODE   READINESS GATES
webapp1-6b54fb89d9-ct7fk   1/1     Running   0          17m   10.46.0.30   ip-172-31-56-227   <none>           <none>

Yes! We have the pod running for the deployment we created, Kubernetes has assigned an internal IP to the pod, which is 10.46.0.30. We can use this IP anywhere inside our cluster to talk to the service. So, open the terminal inside the cluster (if minikube, run terminal directly, if you are using VMs, ssh into one of the VMs which is part of the cluster).
and make a GET to port 80 using curl. Make sure you replace the IP with the given IP in your cluster.

curl http://10.46.0.30

We see the response as shown below:

<h1>This request was processed by host: webapp1-6b54fb89d9-ct7fk</h1>

This is the response returned by the web server. This means our set-up is correct and the web server is running.

Now it is time to setup another pod which makes a request to the web-server pod. To do this we use byrnedo/alpine-curl image and simply call curl command inside the pod by specifying the same IP. We will be creating a Job for this, since this is an one time activity. Let's create an YAML for this Job. The job simply makes a curl request to the IP we specified. 10.46.0.30 is the IP of the server pod we created before. (-s is just to avoid printing unnecessary status and progress bar) (client-job.yaml)

apiVersion: batch/v1
kind: Job
metadata:
  name: client-job
spec:
  template:
    spec:
      containers:
      - name: client
        image: byrnedo/alpine-curl
        command: ["curl", "-s",  "http://10.46.0.30"]
      restartPolicy: Never
  backoffLimit: 4

Now we will deploy the job on K8s and see the result.

kubectl create -f client-job.yaml

Let's see the job

kubectl get jobs

Output:

NAME         COMPLETIONS   DURATION   AGE
client-job   1/1           2s         18m

The job is created and has terminated successfully. Now let's see the logs. Here client-job-z6nql is the pod created by the job client-job which we created in the previous step.

kubectl logs client-job-z6nql

So this will output the curl result.

<h1>This request was processed by host: webapp1-6b54fb89d9-ct7fk</h1>

So that's it, our set-up is complete, now we explore various ways the communication between pods can be achieved, in fact this is one of the ways to communicate but it is very unreliable, we will see why.

1. Using Pod IPs directly

What we did till now was to communicate with web-server using it's internal IP directly. Whenever we create a pod in Kubernetes, it automatically assigns an internal IP to it. The IP will be picked up from CIDR range and will be assigned to the Pod. This IP will be available throughout the cluster and using this IP any pod can address our web-server. This is the simplest way to achieve communication, but it has some serious drawbacks.

The Pod IPs can change - In case the cluster got restarted, the Pod IPs can change sometimes, this might break your client or the requesting service.
You need to know the IP in-prior - Many K8s deployments are dynamic in nature, they are set-up and installed by CD tools, this makes it impossible to know the IP of the Pod in prior, because the Pod can get any IP when it is created.

2. Creating and using Services

Since Pods are non-permanent and dynamic in nature as discussed above, addressing them permanently becomes a problem. To mitigate issue Kubernetes came up with the concept of Services.

Service is an networking abstraction for a group of pods. In other words, a service maps a pod or a group of pods using a single name which never changes. Since the service assigns a constant name to a group of pods, we don't have to worry about Pod's IP anymore, this abstracts away the changing IP problem of pods. Secondly, since we create and assign service names, they can be used as a constant address for communication, K8s internal DNS takes care of mapping service name to Pod IPs.

In order to bring this into our set-up, we just have to create a Service resource for the web-server we created. Let's create the service definition with YAML. (web-app-service.yaml)

apiVersion: v1
kind: Service
metadata:
  name: web-app-service
spec:
  selector:
    app: webapp1
  ports:
    - protocol: TCP
      port: 80

The YAML file looks clean, selector is an important aspect to take care of. The selector is the one which tells where the service should map. We are targeting webapp1 deployment by using app selector label. Let's deploy this service now.

kubectl create -f web-app-service.yaml

Now, let's see whether the service is created.

kubectl get svc

Output:

NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
kubernetes        ClusterIP   10.96.0.1       <none>        443/TCP   83d
web-app-service   ClusterIP   10.111.195.22   <none>        80/TCP    6m17s

Yes, our service is running. The service got a ClusterIP, cluster IPs are static and are assigned only during the creation of service. Like Pod IPs, ClusterIP is available throughout the cluster for use, unlike Pod IP, cluster IP never changes, so now atleast we have a static destination for addressing permanently. But wait, it still didn't address another issue, how can we know this cluster IP prior, one way is to assign our own IP address (hardcoded), but it doesn't make any sense, it is the functionality of Kubernetes to assign IPs. Here are different ways we can mitigate this issue.

Using Environment variables

It can be tedious to know service cluster IP before or manually assign an IP address. But, Kubernetes has a solution for this problem. Whenever a Pod is created, kubernetes injects some environment variables into the pod's environment, these environment variables can be used by containers in the pod to interact with the cluster. Fortunately, whenever you create a service, the address of the service will be injected as an environment variable to all the Pods that run within the same namespace. If you exec into any of the pod and run env command, you will see all the variables that are exported by K8s.

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=client-job-bbwd6
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_SERVICE_PORT=443
WEB_APP_SERVICE_PORT_80_TCP_PORT=80
KUBERNETES_PORT_443_TCP_PORT=443
WEB_APP_SERVICE_PORT=tcp://10.111.195.22:80
WEB_APP_SERVICE_PORT_80_TCP_PROTO=tcp
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP_PROTO=tcp
WEB_APP_SERVICE_SERVICE_HOST=10.111.195.22
WEB_APP_SERVICE_SERVICE_PORT=80
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
WEB_APP_SERVICE_PORT_80_TCP=tcp://10.111.195.22:80
WEB_APP_SERVICE_PORT_80_TCP_ADDR=10.111.195.22
HOME=/root

In the list we can see WEB_APP_SERVICE_SERVICE_HOST and WEB_APP_SERVICE_SERVICE_PORT, these are the host and port variables of the service web-app-service we created in one of the previous step. Any pod which runs in a namespace gets the ClusterIP and port details of all the services created within the same namespace. The kubernetes convention of these environment variables is as follows:

{{SERVICE_NAME}}_SERVICE_HOST   # For ClusterIP
{{SERVICE_NAME}}_SERVICE_PORT   # For port

All - in the service name are replaced by an underscore (_) , since Linux doesn't support - in variable names. Let's create a job to test this quickly:

apiVersion: batch/v1
kind: Job
metadata:
  name: client-job
spec:
  template:
    spec:
      containers:
      - name: client
        image: byrnedo/alpine-curl
        command: ["/bin/sh", "-c", "curl -s http://${WEB_APP_SERVICE_SERVICE_HOST}:${WEB_APP_SERVICE_SERVICE_PORT}"]
      restartPolicy: Never
  backoffLimit: 4

Instead of using Pod IPs or ClusterIP directly, we are using environment variables to dynamically infer the service IP and service port. Let's deploy this job. (client-job-env.yaml)

kubectl create client-job-env.yaml

The job should run without any errors if the service is mapped properly and we should see the response from web-server. Let's check. (client-job-s7446 is the pod created by the job)

kubectl logs client-job-s7446

Output:

<h1>This request was processed by host: webapp1-6b54fb89d9-ct7fk</h1>

Yes! The service is working properly as expected and we are able to address the service as desired.

Using Service names (Requires cluster DNS)

Another easier way is to use service name directly, if the port is already known. This is one of the simplest ways of addressing, but it requires cluster DNS to be set-up and working properly, most of the kubernetes deployment tools like kubeadm or minikube comes with core-dns installed. Also for core-dns to function correctly, you might require a CNI plugin like flannel, cilium, weavenet etc.

For example, if we create a service by name web-app-service, the URL http://web-app-service should be routed to the web-server pod properly. (on port 80 by default), any URL http://web-app-service:xxxx should be routed to the web-server pod at xxxx port. Kubernetes DNS takes care of name resolution. Let's redeploy the job by making this modification (client-job-dns-1.yaml)

apiVersion: batch/v1
kind: Job
metadata:
  name: client-job
spec:
  template:
    spec:
      containers:
      - name: client
        image: byrnedo/alpine-curl
        command: ["curl", "-s", "http://web-app-service"]
      restartPolicy: Never
  backoffLimit: 4

As you can see, we replaced the IP with name of service directly. This should work if cluster DNS is working properly. Let's check logs. (client-job-mj5vr is the pod created by the job)

kubectl logs client-job-mj5vr

Output:

<h1>This request was processed by host: webapp1-6b54fb89d9-ct7fk</h1>

3. Communicating between services across namespaces

Till now all our deployments and jobs were in a single namespace. If the web-app and the client job are in different namespaces, we cannot communicate using environment variables, as Kubernetes doesn't inject variables from other namesapces. We cannot use just service names like web-app-service as they are valid only within the namespace. So, how do we communicate across namespaces? Let's see.

Using fully-qualified DNS names

Kubernetes has an answer for this problem as well. If we have cluster-aware DNS service like CoreDNS running, we can use fully qualified DNS names. starting from cluster.local Assume that our web-server is running in namespace test-namespace and has a service web-app-service defined. We can address this using an URL shown below:

web-app-service.test-namespace.svc.cluster.local

Sounds tricky?? Here is the breakdown of the URL

.cluster.local : This is the root of our cluster DNS, every resource must be accessed from root.
.svc : This tells we are accessing a service resource.
test-namespace : This is the namespace where our web-app-service is defined.
web-app-service: This is our service name.

We can use URLs like http://web-app-service.test-namespace.svc.cluster.local:[xxxx] (xxxx is the Port, you can optionally ignore this if the service is mapping default http port 80)

So the general format for addressing a service in another namespace is to use a fully qualified DNS name like the one shown above. It is always suitable to use URLs like this as they are universal and can be addressable anywhere throughout the cluster. Again here is the general format of the URL:

 {{service_name}}.{{namespace}}.svc.cluster.local

So that's it! we have seen various possible ways to address and communicate between micorservices running on a Kubernetes cluster.

Thanks for spending your time reading this post. Please let me know your views and opinions in the comments section.

Crafting a concurrent queue in Golang

Narasimha Prasanna HN — Sun, 06 Dec 2020 09:46:09 +0000

Queue is one of the most important data structure. Using queues, we can model many computational tasks easily and efficiently. Queues are used everywhere, in Operating systems, they are used to schedule processes, in routers, queues are used as buffers to store packets before processing them. Even cloud native applications composed of miroservices and workers use queues in between them for communication. Queues are extensively used for asynchronous processing.

What is a Job queue?

If you have ever used node.js, you would be familiar with job queues already. The so called Event Loop in node is nothing but a job queue which schedules asynchronous operations to any on the free uv-threadpool workers. This type of pattern is common in many systems out there. So here is how queue is used for asynchronous processing, consider the scenario below:

There will be a separate thread or a group of threads waiting for some kind of input.
The main thread produces these inputs ( or tasks )

In this case, there has to be a medium which must act as a glue between input main thread and worker threads running in background. The medium should be designed in such a way that the input which is fed first should get a chance to be processed first, then the next. The input fed into this medium can be consumed by any of the free worker threads. Queue is the best data-structure for this scenario, because of FIFO (First-in, First-out) property, this property ensures ordering, if we make this data-structure somehow shareable between multiple threads we can achieve what we wanted to.

This is how a job queue looks like, the main thread places input on to the queue and continues its operation without waiting for the output. Since the workers wait for input, any one of the worker pops the input from the queue and and processes it. Any of the thread which is free can pop the input. Since the thread which is doing the work will have no time to pop the next input from the queue, only the free thread will get a chance to pop from the queue and becomes busy. If no thread is free, the inputs will still be there in the queue following the right order.

What are the advantages?

Imagine a scenario where main thread did all the work, it would work much slower because the main thread cannot process next job until it finishes the current one. A single thread cannot run on multiple cores at once, that is the main reason it runs sequentially. But on a machine with multiple cores, this is not a good approach, because all other cores sit ideally without doing any work. Since we used thread pool here, the free threads in our pool runs on separate cores and we are exploiting parallelism, if we have a 8 core machine, we can now process 8 inputs at once in the same time we took to process a single input. The main thread can now concentrate on fetching new inputs rather than processing them, because putting an input on the queue doesn't take much time.

Concurrent queue

Every advantage comes with a disadvantage as well. In the case we discussed above, we cannot use a normal queue, because it is not thread-safe. Since we have multiple threads operating on the same queue, synchronization problems will be obviously there, which we need to take care of. A concurrent-queue is a special queue designed to handle such issues. It allows multiple threads to operate on the same queue without any synchronization problems. In this blog we will be designing one such concurrent queue that can be used by multiple goroutines. The queue we design will have following properties:

Can hold any type of data, i.e the queue is a generic implementation (interface type in golang)
Expandable in size, it grows and shrinks as required. (i.e no pre-allocated memory)
Thread-safe by nature, no synchronization issues.

Let's create a normal queue first

Synchronization issues has nothing to do with how data is stored, we only have to control how data is accessed. This allows us to implement a normal queue first and use it as a storage backend for the concurrent queue.

Here is how we are going to design the queue:

We will be using a doubly-linked list.
The queue will have push and pop functions.
We insert data at the head of the list and pop from the tail. This is how push and pop will be implemented.
We make the queue to grow only until maxSize. So we keep track of current size every time an enqueue or dequeue occurs and take decisions based on the current size. Here is an implementation of the queue using doubly linked list.



//Node storage of queue data
type Node struct {
    data interface{}
    prev *Node
    next *Node
}

//QueueBackend Backend storage of the queue, a double linked list
type QueueBackend struct {
    //Pointers to root and end
    head *Node
    tail *Node

    //keep track of current size
    size    uint32
    maxSize uint32
}

func (queue *QueueBackend) createNode(data interface{}) *Node {
    node := Node{}
    node.data = data
    node.next = nil
    node.prev = nil

    return &node
}

func (queue *QueueBackend) put(data interface{}) error {
    if queue.size >= queue.maxSize {
        err := errors.New("Queue full")
        return err
    }

    if queue.size == 0 {
        //new root node
        node := queue.createNode(data)
        queue.head = node
        queue.tail = node

        queue.size++

        return nil
    }

    //queue non-empty append to head
    currentHead := queue.head
    newHead := queue.createNode(data)
    newHead.next = currentHead
    currentHead.prev = newHead

    queue.head = currentHead
    queue.size++
    return nil
}

func (queue *QueueBackend) pop() (interface{}, error) {
    if queue.size == 0 {
        err := errors.New("Queue empty")
        return nil, err
    }

    currentEnd := queue.tail
    newEnd := currentEnd.prev

    if newEnd != nil {
        newEnd.next = nil
    }

    queue.size--
    if queue.size == 0 {
        queue.head = nil
        queue.tail = nil
    }

    return currentEnd.data, nil
}

func (queue *QueueBackend) isEmpty() bool {
    return queue.size == 0
}

func (queue *QueueBackend) isFull() bool {
    return queue.size >= queue.maxSize
}

We have created a QueueBackend, this is just a storage layer on top of which synchronization primitives are implemented. The code above is simple and anyone with basic data structures knowledge can understand it. Next we are going to implement synchronization primitives.

Adding concurrency support

We are going to create a ConcurrentQueue type which uses QueueBackend type internally. The ConcurrentQueue type is defined as shown below:



//ConcurrentQueue concurrent queue
type ConcurrentQueue struct {
    //mutex lock
    lock *sync.Mutex

    //empty and full locks
    notEmpty *sync.Cond
    notFull  *sync.Cond

    //queue storage backend
    backend *QueueBackend
}

Here, the backend is a pointer variable which holds the address of the queue storage layer we created before. Now let us see what other members actually mean.

lock : This is a member pointer of type sync.Mutex, a mutex is a synchronization primitive, mutex simply means a lock, lock can be either locked or unclocked. If the mutex is locked, other threads will wait until the lock is unlocked. This ensures only one thread can access the queue at a time. Here is how a thread will operate on the queue: a. The thread first locks the mutex, this way it gains exclusive control over the queue by avoiding other threads from accessing the queue at the same time. b. It then calls euqueue or dequeue over the queue and once that is done, it unlocks the mutex, allowing other waiting threads to get access to the queue.

This is more than enough to provide a basic synchronization support for the queue, but we actually need more. Once a thread gains control over the queue, it pops the data if present, if no data is present, it has to wait. One way to implement this wait operation is to use a while loop and simply keep checking the queue for data every time. This is not so cool because it runs in an infinite loop taking all the CPU cycles (the core usage goes to 100%). We need some mechanism that allows us to suspend the thread until data becomes available, we can do so by using conditional locks. We have defined two such locks, notEmpty and notFull. notEmpty means that, the queue is not empty and has some data, so anyone can pop. notFull means that, the queue is not full and the main thread can still push the data into it. If queue is not in notEmpty state, the consumer (worker threads) will wait. Similarly, if notFull condition is not met, the producer (main thread) will wait for queue to become empty. Now let us see how euqueue is implemented:



func (c *ConcurrentQueue) enqueue(data interface{}) error {
    c.lock.Lock()

    for c.backend.isFull() {
        //wait for empty
        c.notFull.Wait()
    }

    //insert
    err := c.backend.put(data)

    //signal notEmpty
    c.notEmpty.Signal()

    c.lock.Unlock()

    return err
}

The equeue function checks if anyone has already locked the queue, if not, it locks the queue to gain control, next it checks if the queue is full, if yes, it waits until it is not full. (in the line c.notFull.Wait()), After waiting, a new input is pushed into the queue, calling push over the queue backend. Now the queue is not empty, because there is a new input, c.notEmpty.Signal() is called, this notifies the threads waiting over notEmpty that a new entry has been inserted and the queue is no more empty, finally the lock is released, allowing other threads to operate on the queue.

Now, let us see how dequeue is implemented at the consumer's side:




func (c *ConcurrentQueue) dequeue() (interface{}, error) {
    c.lock.Lock()

    for c.backend.isEmpty() {
        c.notEmpty.Wait()
    }

    data, err := c.backend.pop()

    //signal notFull
    c.notFull.Signal()

    c.lock.Unlock()

    return data, err
}

The consumer tries to access the queue for reading, if it is locked, it waits for the thread which is already accessing the queue to release the lock. Next, it checks if the queue is empty, if yes, it waits over notEmpty mutex. (in the line c.notEmpty.Wait()). Once the queue is not empty (remember c.notEmpty.Signal() in enqueue), it pops the entry and notifies the producer to push new input, since there is a space in the queue now. This is done by calling c.notFull.Signal()).

That is it, we have implemented basic enqueue and dequeue methods with synchronization support. We also implement a miscellaneous method called getSize which simply returns the size, even this is done with lock because it might return the wrong value if another thread push/pops the data at the same time.



func (c *ConcurrentQueue) getSize() uint32 {
    c.lock.Lock()
    size := c.backend.size
    c.lock.Unlock()

    return size
}

To wrap this entire thing, we create a method called NewConcurrentQueue which simply creates an instance of ConcurrentQueue type and initializes it with proper values. Here we also initialize all the necessary mutexes. Look at it's code below:



//NewConcurrentQueue Creates a new queue
func NewConcurrentQueue(maxSize uint32) *ConcurrentQueue {
    queue := ConcurrentQueue{}

    //init mutexes
    queue.lock = &sync.Mutex{}
    queue.notFull = sync.NewCond(queue.lock)
    queue.notEmpty = sync.NewCond(queue.lock)

    //init backend
    queue.backend = &QueueBackend{}
    queue.backend.size = 0
    queue.backend.head = nil
    queue.backend.tail = nil

    queue.backend.maxSize = maxSize
    return &queue
}

I was able to create a simple HTTP work pool using this queue implementation as a job queue. You can checkout the complete source code here : httppool

Thanks for reading this!

Selecting the best open-source library/framework for your next project.

Narasimha Prasanna HN — Sun, 01 Nov 2020 15:48:21 +0000

Everyday we all use Open source projects directly or indirectly to do some task (as a user) or build our own projects on top of it (as a developer). If you are a developer, you have to be bit careful while choosing an open source library/framework to build your project because the future of your project is going to be dependent on that framework. Choosing the right framework can boost up the project's progress, at the same time choosing a bad framework can slow down your progress and maybe you have to spend extra time to debug it or sometimes re-write the entire project using another framework or from scratch on your own.

Choosing the right framework depends on both your intuition and experience (The term experience here means how much you have experimented on similar projects before, not number of professional years). I am not an experienced developer, but I've had bad experiences with the past decisions I made while choosing a framework (Later I had to re-write my project), as time progressed through experimentation I was able to pick a good quality framework most of the time. A good framework here means Quality of the project, not the use-case. (Obviously we will not choose a framework which doesn't suit our use-case). So here I have made an attempt to list few points that makes an Open source project usable or good, maybe if you are an author planning to create your next open source framework, then this will act like a checklist for you. This post is for newbies who are into programming. If you are not a newbie, all the points below may seem obvious

1. Project roadmap

Many open source projects are abandoned after the initial commit, the developer never cares about adding new features, so obviously there will not be any roadmap for such projects. Be careful while using such projects, because the developer who created such project may comeback one day and change the entire code-base. (This has happened once for me, I had to pull back the initial commit). In some other cases, the repo itself can be removed, leaving no backups. If you look at successful open source projects/frameworks (like tensorflow, react, node, flutter etc), they all have a well defined roadmap. Roadmap also gives you a look into the future of the framework.

2. Versioning, Nightly builds and Latest stable releases:

Versioning is very important for any project, it helps the developers who build on the project to use a specific snapshot of the code-base. Many popular open source projects use Semantic versioning standard to version the code-base snapshots.
There are two types of releases (majorly) 1. Nightly builds and Latest Stable builds. If you want the most stable, well-tested version of the framework, you can choose latest stable release, but if you want to use the latest cutting-edge feature (and you can't wait for next-stable), you can always choose latest nightly build (These builds are almost available every day). If you are using a nightly build for production, be careful, because nightly builds can break at any point (the latest features may not be tested for edge-cases). Also if you are planning to pull the library/framework directly from GitHub, never pull the master branch directly, the fresh master branch may contain features which are recently merged or not tested yet (you might be in trouble if you do this, sometimes!!), go to releases and check for the list of version tags, tags are like snapshots of the code-base which is stable, you can pull that snapshot to make sure you are safe.

3. Dependencies and code-packaging

Majority of open-source projects are dependent on other projects (Example : Tensorflow depends on many C/C++ libraries like eigen, abseil, protobuf, cuda etc), if you are using such framework, indirectly you are using its dependencies as well. If any dependency breaks, the framework breaks, so your project breaks as well!!. Dependency management is an important thing which if often ignored in most of the projects. Many modern programming languages handle this problem by shipping a package manager along with the language compiler and runtime. (Example : pip for python and npm for node). These package managers can automatically download and install right-dependency for you, so need not worry much. Things become complex if the project is not released as a package, because you have to somehow integrate the code-base yourself, this is where versioning plays an important role. Whenever you feel like using an open source framework check if the dependencies are versioned properly. (If they are not versioned, the package manager always downloads the latest version which can cause trouble).
For example, recently I was working on some audio processing project using librosa - python, the latest version of librosa requires numba which in turn is dependent on llvm-11.0, but the latest version of llvm which can be installed using apt is llvm-8.0, I had to build llvm-11.0 from source, which cased some other problems. (These issues are common :p)

4. Community support and popularity

This is the most important aspect if you are using a complex framework like React or Tensorflow. We all run into trouble at some point and we need to find solutions quickly to proceed further, at that point the obvious thing we all do is to google the problem (or just copy paste the error message :p). If the library is popular we obviously get a stackoverflow link which solves our problem, if the library is not so popular, the obvious thing is the issues and community forum, where we find common problems other developers are facing. Most successful projects have a community forum where you can discuss your problem and get the solution from other developers. (React has many such slack channels both official and unofficial ones, Even node.js has many such forums)
So if you are really using the framework for something important, make sure you have a proper community support, otherwise you many end up digging the source code and breaking your mind at last!!

5. Documentation and Unit-Tests

Of course, a good project must have these things for sure. Documentation will help you to understand what all features are available in the framework, writing documentation can be boring, but these days there are many tools out there which can make things easier for you, these tools can automatically export a documentation in HTML format from the code comments (Example: look at python's docstring standard). It is not recommended to use a framework which does not have a proper documentation, unless you are capable of reading and understanding directly from the source code.
A framework can have many features, but are you sure all these features work as they intend to work?? This is where unit-tests comes into picture, If you plan to use a project which has unit tests, you can clone the repo, build it and run the unit-tests, if all the unit-tests are successful, you can use the framework with confidence. Unit-tests sometime serve as an alternative to code-documentation, if there is no documentation, you can use unit-tests itself as documentation as they show you how all the functions are meant to be invoked and what all input types they accept to produce proper output.

6. Build-scripts and Docker images

Not all the frameworks are available as executables, sometimes you have to build/compile them on your own (If you are a C/C++ developer, you have to deal with this everyday). This is where build scripts and built-automation tools are really helpful. Many good C/C++ projects I have seen so far atleast provide a Makefile which automates the build-process with right include files and install location. Some complex projects like Tensorflow make use of advanced build-systems like bazel and cmake. Build systems are here to make our life easier, imagine how time consuming it can be to figure out how to compile a project with thousands of source files on your own. These build-scripts can come handy when you make changes to the code-base (many build-systems can cache old objects and compile only the changed files to reduce the build time). It is even better if you find out a framework which also provides a Dockerfile, Dockerfile is used to build a docker image, which automatically packs everything required to build on top of the framework. Docker images run as containers when executed, you can think of container as an isolated environment where you can run the code without breaking anything directly on the host machine. (Technically, a container allows your code to run in a isolated namespace and also changes the root-filesystem path so that container no longer has access to the host file-system).

Thanks for reading my post. Please share your opinions in the comments section, you are always welcome to suggest/add more points :)

Understanding C/C++ Build system by building a simple project

Narasimha Prasanna HN — Fri, 02 Oct 2020 15:53:48 +0000

C/C++ is the mother of many popular programming languages out there today, all the fancy programming languages we use today like Python, JavaScript are built using C/C++. For example, the standard python interpreter CPython is built using C and the most popular JavaScript implementation V8 is built using C/C++, C/C++ also powers most of the underlying libraries used by Node.js, In other words, C/C++ powers most of the open source software ever written by humans. One of the main reasons we prefer high level languages like Python is because of the robust package management tools they provide, we don't have to worry about managing dependencies anymore pip automatically manages it for us. Same case holds true for JavaScript as well. These languages also have robust build systems which allow us to build and ship the software more easily.

C/C++ also has few popular build systems like cmake and bazel which manages dependencies automatically, but in this post, we will be compiling a C/C++ project without making use of these tools in order to understand how things work internally.

We will be building a simple system logger that logs total free RAM memory of the system every 5 seconds.

First things first! Let's create a project structure:

Project structure has to be easily understandable and should isolate different functionalities as much as possible to avoid confusion. No one will ever stop us from using our own project structure, but most of the open source projects built with C/C++ use this structure :

project_root
  - include
  - src
      - module-1
      - module-2
      - module-n
      - main.c/main.cc (depends on the project)
  - Makefile
  - README
  - LICENSE
  - misc files

Let's have a look at what each and every file/directory means:

include - This is the place where all our header files live.
src - The directory that contains all our source code. We can have multiple sub-directories/modules inside src. Also we can have a main function file inside src.
Makefile : Makefiles are used by make command, we will be using make to build our project.

For our project we will have the following structure:

memlogger
   - bin  - will explain the need for this
   - include
       - free_memory_api.h
       - file_writer.h
   - src
       - free_memory_api
          - free_memory_api.c
       - file_writer
          - file_writer.c
       - main.c
   - Makefile

Let's Code

We can start writing code once we are done with the project structure setup. I will not be explaining the code in depth to avoid writing very long post, but we will stress more on the concepts.

What are header files?

Header files are blueprints of our actual C code. For every C module we write, it is a good practice to export the header-file. These header files are used by the compiler to understand what all functions are exported by a module. Once compilation is done, header files are not used anywhere. The actual use of header files comes into picture when our project/module is used as a module in some other project, other programmers can simply include our header file to use the function declarations we exported.

Let us create free_memory_api.h as per the structure :

#ifndef __FREE_MEMORY_API
#define __FREE_MEMORY_API
//this is our API function which returns free memory in bytes
unsigned long long get_free_system_memory();
#endif

Let us create file_writer.h which declares file writer API

#ifndef __FILE_WRITER_API
#define __FILE_WRITER_API

#include <stdio.h>
//opens the log file for writing
FILE * open_log_file(char * path);
// we will use this function to write contents to the log file
void write_log_to_file(FILE * file, unsigned long long free_memory);
//closes the log file
void close_log_file(FILE * file);
#endif

Let's define these APIs:

We declared what all APIs we need, but we did not write the underlying code for those APIs. We will be writing the code for file logging and getting free memory from the system. Before writing the code, we have to import the blueprint we declared before.
file_writer.c

#include "file_writer.h"

// Open the log-file in append mode and return it
FILE * open_log_file(char * file_path) {
    FILE * fp = fopen(file_path, "a");
    return fp;
}

// Close the file 
void close_log_file(FILE * fp) {
    if(fp) {
        fclose(fp);
    }
}

//write log entry into the file
void write_log_to_file(FILE * fp, unsigned long long free_memory) {
    if(fp) {
        fprintf(fp, "free_memory=%llu\n", free_memory);
    }
}

Now let us define the free memory api i.e free_memory_api.c:

#include <sys/sysinfo.h>
#include "free_memory_api.h"

unsigned long long get_free_system_memory() {
    struct sysinfo info;
    if (sysinfo(&info) < 0) {
         return 0;
    }
    return info.freeram;;
}

And finally, main.c

#include "file_writer.h"
#include "free_memory_api.h"

#include <unistd.h>

int main(int argc, char **argv) {
    if (argc < 2) {
        printf("Provide log file name\n");
        return 0;
    } 

    unsigned long long free_memory = 0;
    while(1) {
        free_memory = get_free_system_memory();
        FILE * log = open_log_file(argv[1]);
        write_log_to_file(log, free_memory);
        close_log_file(log);
        sleep(5);
    }
}

Let's start building the project

Now that we have written the code, it is time to compile the project. Now we have multiple modules in our project. These modules can be linked together to build a standalone executable, or we can build individual modules alone as shared libraries and link them together at runtime.

Building a static-monolithic executable

In this section, we build a single binary that can be shipped, there are many ways we can build a C/C++ project. In this post we will only build a standalone executable which is the most easiest way of building a C project.
We make use of make a Linux command-line utility that can automate any task, it is a series of shell commands which can be grouped and tagged under a name to perform a specific task. We can write multiple such tasks conveniently using a Makefile. Let's see our Makefile now.

COMPILER=gcc

file_writer:
    @$(COMPILER) -c src/file_writer/*.c -Iinclude/ -o bin/file_writer.o
    @echo "Built file_writer.o"

free_memory_api:
    @$(COMPILER) -c src/free_memory_api/*.c -Iinclude/ -o bin/free_memory_api.o 
    @echo "Built free_memory_api.o"

project:
    $(COMPILER) -c src/main.c -Iinclude/ -o bin/main.o
    @$(COMPILER) bin/free_memory_api.o bin/file_writer.o bin/main.o -o memlogger
    @echo "Finished building memlogger"

Even though we can build the entire project in a single command, I have divided this into three phases.

file_writer : This make rule will generate file_writer.o object file under ./bin.
free_memory_api : This rule generates free_memory_api.o under ./bin.
project : This builds the entire project,it generates main.o and links main.o with other two object files to create a standalone executable called memlogger.

Let's execute these commands with make:
Step-1. file_writer.o

   make file_writer

Step-2. free_memory_api.o

   make free_memory_api

Step-3. Final binary:

   make project

Executing the binary:

We can execute the binary by running it like a normal linux executable:

memlogger logs.txt

After 15 seconds, we see 3 entries in the log file:

free_memory=8322523136
free_memory=8330776576
free_memory=8335728640

Which is approximately 8GB free out of 16GB RAM and it is correct. hurray! we created a small system logger.

What happened under the hood??

It is important to understand the build process we used here. To understand that, we need to know the concept of object files.

What we did in the Makefile??

We defined three rules, each rule builds a module, the third rule goes ahead by one step and links all the three modules.
We used gcc compiler, (g++ for C++ projects). Options used:

-c : This tells the compiler to only compile and don't perform linking, since we are explicitly linking the object files in Step-3.
-I : Since we defined our own headers, we have to provide it to the compiler during compile time, by default compiler searches for these headers in standard locations, we can also tell the compiler to include our custom location for resolving the headers using -I.
-o : The output file name.

What are object files?

An object file is a linux ELF - Executable and linkable format binary produced by the compiler. ELF is designed by POSIX standards and all Linux distributions can understand what an object file is. What this object file contains in layman terms is a mapping table and symbol definitions.

mapping-table or Symbol table : The mapping table contains a set of symbols defined by the object file and an offset in text segment where actual code for the symbols are defined.
Symbol definitions : This section contains machine code of all our functions. So, to get the machine code of a function/symbol we do these two steps: First, we lookup the symbol table of the object file to get its offset in the text segment. Second, we go to the text segment and obtain its code.

These are just layman terms, ELF has standard definitions for mapping-table and Segment definitions which are confusing for a beginner.

Let's see the object file contents of file_writer.o, we use a tool called readelf which is default in all the linux systems.

readelf -h bin/file_writer.o

Output:

ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              REL (Relocatable file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          0 (bytes into file)
  Start of section headers:          1168 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           0 (bytes)
  Number of program headers:         0
  Size of section headers:           64 (bytes)
  Number of section headers:         13
  Section header string table index: 12

These are ELF headers. Now let's see the Symbol table (or mapping table in our terms)

readelf --syms bin/file_writer.o

Output:

Symbol table '.symtab' contains 16 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND 
     1: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS file_writer.c
     2: 0000000000000000     0 SECTION LOCAL  DEFAULT    1 
     3: 0000000000000000     0 SECTION LOCAL  DEFAULT    3 
     4: 0000000000000000     0 SECTION LOCAL  DEFAULT    4 
     5: 0000000000000000     0 SECTION LOCAL  DEFAULT    5 
     6: 0000000000000000     0 SECTION LOCAL  DEFAULT    7 
     7: 0000000000000000     0 SECTION LOCAL  DEFAULT    8 
     8: 0000000000000000     0 SECTION LOCAL  DEFAULT    6 
     9: 0000000000000000    41 FUNC    GLOBAL DEFAULT    1 open_log_file
    10: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND _GLOBAL_OFFSET_TABLE_
    11: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND fopen
    12: 0000000000000029    34 FUNC    GLOBAL DEFAULT    1 close_log_file
    13: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND fclose
    14: 000000000000004b    54 FUNC    GLOBAL DEFAULT    1 write_log_to_file
    15: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND fprintf

As we can see in the table, we have entries for open_log_file, close_log_file, write_log_to_file which are the API functions we defined. Hurray! Our object file is correct. Also, if you observe carefully, we see the presence of fclose, fopen and fprintf and they are prefixed with UND which means these symbol addresses are not known yet, but C/C++ runtime resolves them during linking which is in step-3, it either links to these functions statically or dynamically during the runtime, we will see the concept of shared libraries in the next part.

Similarly, we can run the same command for free_memory_api.o to see it's symbol table. We get the output as :

Symbol table '.symtab' contains 14 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND 
     1: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS free_memory_api.c
     2: 0000000000000000     0 SECTION LOCAL  DEFAULT    1 
     3: 0000000000000000     0 SECTION LOCAL  DEFAULT    3 
     4: 0000000000000000     0 SECTION LOCAL  DEFAULT    4 
     5: 0000000000000000     0 SECTION LOCAL  DEFAULT    5 
     6: 0000000000000000     0 SECTION LOCAL  DEFAULT    7 
     7: 0000000000000000     0 SECTION LOCAL  DEFAULT    8 
     8: 0000000000000000     0 SECTION LOCAL  DEFAULT    6 
     9: 0000000000000000    89 FUNC    GLOBAL DEFAULT    1 get_free_system_memory
    10: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND _GLOBAL_OFFSET_TABLE_
    11: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND sysinfo
    12: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND printf
    13: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND __stack_chk_fail

We can see get_free_system_memory as a FUNC and sysinfo which is undefined.

What we did in the final step?

In the first two steps, we compiled the modules and generated the object files, but they can't be executed because they don't have the main function definition which is the entry-point of any C/C++ program. We have two commands in Makefile under project rule (step-3), the first command only compiles the main.c file into main.o, let's try to run it, it should run because it has main function.

./main.o

Output:

bash: ./bin/main.o: cannot execute binary file: Exec format error

We cannot run it because it is not an executable file, it is an object file, the final step is still remaining which links all the three object files and generate the final executable binary.
Before that we will try to link only main.o and discard the remaining two modules, let's see what happens:

gcc bin/main.o -o memlogger

Output:

bin/main.o: In function `main':
main.c:(.text+0x36): undefined reference to `get_free_system_memory'
main.c:(.text+0x4d): undefined reference to `open_log_file'
main.c:(.text+0x64): undefined reference to `write_log_to_file'
main.c:(.text+0x70): undefined reference to `close_log_file'
collect2: error: ld returned 1 exit status

This is exactly what was supposed to happen, the executable file needs the following functions but don't know where they are. So we need to link it with remaining two object files.

gcc bin/free_memory_api.o bin/file_writer.o bin/main.o -o memlogger

Now the compiler looks into the Symbol tables of file_writer.o and free_memory_api.o to resolve the functions which were undefined in our previous command. Since the Symbol tables of these two object files defines those symbols/functions the linking is successful and the final executable is generated.

Let's see the mapping table or Symbol table of our memlogger binary:

readelf --syms memlogger

Output:

    26: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS crtstuff.c
    27: 0000000000000780     0 FUNC    LOCAL  DEFAULT   14 deregister_tm_clones
    28: 00000000000007c0     0 FUNC    LOCAL  DEFAULT   14 register_tm_clones
    29: 0000000000000810     0 FUNC    LOCAL  DEFAULT   14 __do_global_dtors_aux
    30: 0000000000201010     1 OBJECT  LOCAL  DEFAULT   24 completed.7698
    31: 0000000000200d88     0 OBJECT  LOCAL  DEFAULT   20 __do_global_dtors_aux_fin
    32: 0000000000000850     0 FUNC    LOCAL  DEFAULT   14 frame_dummy
    33: 0000000000200d80     0 OBJECT  LOCAL  DEFAULT   19 __frame_dummy_init_array_
    34: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS free_memory_api.c
    35: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS file_writer.c
    36: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS main.c
    37: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS crtstuff.c
    38: 0000000000000c5c     0 OBJECT  LOCAL  DEFAULT   18 __FRAME_END__
    39: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS 
    40: 0000000000200d88     0 NOTYPE  LOCAL  DEFAULT   19 __init_array_end
    41: 0000000000200d90     0 OBJECT  LOCAL  DEFAULT   21 _DYNAMIC
    42: 0000000000200d80     0 NOTYPE  LOCAL  DEFAULT   19 __init_array_start
    43: 0000000000000a78     0 NOTYPE  LOCAL  DEFAULT   17 __GNU_EH_FRAME_HDR
    44: 0000000000200f80     0 OBJECT  LOCAL  DEFAULT   22 _GLOBAL_OFFSET_TABLE_
    45: 0000000000000a30     2 FUNC    GLOBAL DEFAULT   14 __libc_csu_fini
    46: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_deregisterTMCloneTab
    47: 0000000000201000     0 NOTYPE  WEAK   DEFAULT   23 data_start
    48: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND puts@@GLIBC_2.2.5
    49: 0000000000201010     0 NOTYPE  GLOBAL DEFAULT   23 _edata
    50: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fclose@@GLIBC_2.2.5
    51: 0000000000000a34     0 FUNC    GLOBAL DEFAULT   15 _fini
    52: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __stack_chk_fail@@GLIBC_2
    53: 00000000000008dc    34 FUNC    GLOBAL DEFAULT   14 close_log_file
    54: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND printf@@GLIBC_2.2.5
    55: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __libc_start_main@@GLIBC_
    56: 0000000000201000     0 NOTYPE  GLOBAL DEFAULT   23 __data_start
    57: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fprintf@@GLIBC_2.2.5
    58: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __gmon_start__
    59: 0000000000201008     0 OBJECT  GLOBAL HIDDEN    23 __dso_handle
    60: 0000000000000a40     4 OBJECT  GLOBAL DEFAULT   16 _IO_stdin_used
    61: 000000000000085a    89 FUNC    GLOBAL DEFAULT   14 get_free_system_memory
    62: 00000000000009c0   101 FUNC    GLOBAL DEFAULT   14 __libc_csu_init
    63: 0000000000201018     0 NOTYPE  GLOBAL DEFAULT   24 _end
    64: 0000000000000750    43 FUNC    GLOBAL DEFAULT   14 _start
    65: 0000000000201010     0 NOTYPE  GLOBAL DEFAULT   24 __bss_start
    66: 0000000000000934   130 FUNC    GLOBAL DEFAULT   14 main
    67: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fopen@@GLIBC_2.2.5
    68: 00000000000008fe    54 FUNC    GLOBAL DEFAULT   14 write_log_to_file
    69: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND sysinfo@@GLIBC_2.2.5
    70: 0000000000201010     0 OBJECT  GLOBAL HIDDEN    23 __TMC_END__
    71: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_registerTMCloneTable
    72: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND sleep@@GLIBC_2.2.5
    73: 00000000000008b3    41 FUNC    GLOBAL DEFAULT   14 open_log_file
    74: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND __cxa_finalize@@GLIBC_2.2
    75: 0000000000000698     0 FUNC    GLOBAL DEFAULT   11 _init

Since the Symbol table is very big, I have pasted entries starting from 26. As you can see our final executable has all our definitions from all the three modules and also there are no UND symbols, these symbols are now replaced with something like fopen@@GLIBC_2.2.5. This means, the code for these functions are not copied into our binary, instead they have to be resolved during the runtime, it is the responsibility of Linux loader ld.so to link these symbols dynamically during runtime.

So this is it! We are done with the first part of the Post. If you are reading this line. I would like to really thank you for reading the entire Post, keep the compliments even if you skipped everything and came here directly.

Thank you :) Have a good time.